Howto: Automated AWS spot instance provisioning with persisting of data

The end result after completing all the steps in this howto

Prerequisites

Bringing up the environment (Virtual Private Cloud)

  1. From your terminal, clone my scripts repository and cd into it
git clone git@github.com:radekosmulski/aws-setup.git && cd aws-setup
./create-env.sh

Configuring our workstation in the cloud

./request-spot-instance.sh p2.xlarge 0.25
if [ -d “$HOME/aws_scripts” ] ; then
PATH=”$HOME/aws_scripts:$PATH”
fi
authorize-current-ip && spot-instance-connect
wget https://raw.githubusercontent.com/radekosmulski/courses/master/setup/install-gpu.sh && bash install-gpu.sh
sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
exec bash && ./start-jupyter-notebook
cat ~/aws_scripts/spot-instance-connect
https://<IP addr>:8888

Creating the persistent volume

sudo fdisk /dev/xvdf
n
p
<enter>
<enter>
w
sudo mkfs -t ext4 /dev/xvdf1
./mount-workspace
cd workspace
sudo chmod go+rw .
./spot-instance-remove.sh

Grande Finale

Workflow steps

./request-main-compute-instance.sh p2.xlarge 0.25
main-compute-instance-connect
./mount-workspace
./start-jupyter-notebook
./main-compute-instance-remove.sh
  1. Why do I need to authorize and deauthorize my IP address?
    This configures ingress rules in the security group of our VPC so that only computers with a given public IP address can access your notebook. This might be a nuisance to some and can be disabled from AWS console. Will provide more details if necessary.
  2. There are two additional steps that can make this setup even more convenient — I only configured this for myself several weeks after the initial version but I am enjoying those changes very much. This is what the process of booting up the instance looks like with additional configuration steps I mention below:
Starting the instance up after completing the additional configuration steps
  1. Configure /etc/fstab following the instructions described here starting with step #7.
  2. Add the following line to your /etc/rc.local
    su ubuntu -c ‘bash /home/ubuntu/start-jupyter-notebook
  3. Redo steps 7 through 9 of the above howto, then follow with step 26.

--

--

--

I ❤️ ML / DL ideas — I tweet about them / write about them / implement them. Recommender Systems at NVIDIA

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Test inference of yolov5 on a aws ec2 instance

Inheritance and Polymorphism

Recording video for object detection with Python and OpenCV

Getting to Know C# Attributes

HackThisSite Realistic Mission 1

How to Use Asp.Net Technologies to Develop web applications?

PHP 7.2 Coming to Halt: How to prepare for this?

Go-Svelte Journey : Golang Backend with Hexagonal Architecture (Part-2)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Radek Osmulski

Radek Osmulski

I ❤️ ML / DL ideas — I tweet about them / write about them / implement them. Recommender Systems at NVIDIA

More from Medium

Machine Learning services in AWS

How I created my own fake medical data source file using faker Library in Databricks

SageMaker Networking

Sneaky look at AWS SageMaker!