Howto: Automated AWS spot instance provisioning with persisting of data

The end result after completing all the steps in this howto

Prerequisites

Bringing up the environment (Virtual Private Cloud)

  1. From your terminal, clone my scripts repository and cd into it
git clone git@github.com:radekosmulski/aws-setup.git && cd aws-setup
./create-env.sh

Configuring our workstation in the cloud

./request-spot-instance.sh p2.xlarge 0.25
if [ -d “$HOME/aws_scripts” ] ; then
PATH=”$HOME/aws_scripts:$PATH”
fi
authorize-current-ip && spot-instance-connect
wget https://raw.githubusercontent.com/radekosmulski/courses/master/setup/install-gpu.sh && bash install-gpu.sh
sudo shred -u /etc/ssh/*_key /etc/ssh/*_key.pub
exec bash && ./start-jupyter-notebook
cat ~/aws_scripts/spot-instance-connect
https://<IP addr>:8888

Creating the persistent volume

sudo fdisk /dev/xvdf
n
p
<enter>
<enter>
w
sudo mkfs -t ext4 /dev/xvdf1
./mount-workspace
cd workspace
sudo chmod go+rw .
./spot-instance-remove.sh

Grande Finale

Workflow steps

./request-main-compute-instance.sh p2.xlarge 0.25
main-compute-instance-connect
./mount-workspace
./start-jupyter-notebook
./main-compute-instance-remove.sh
  1. Why do I need to authorize and deauthorize my IP address?
    This configures ingress rules in the security group of our VPC so that only computers with a given public IP address can access your notebook. This might be a nuisance to some and can be disabled from AWS console. Will provide more details if necessary.
  2. There are two additional steps that can make this setup even more convenient — I only configured this for myself several weeks after the initial version but I am enjoying those changes very much. This is what the process of booting up the instance looks like with additional configuration steps I mention below:
Starting the instance up after completing the additional configuration steps
  1. Configure /etc/fstab following the instructions described here starting with step #7.
  2. Add the following line to your /etc/rc.local
    su ubuntu -c ‘bash /home/ubuntu/start-jupyter-notebook
  3. Redo steps 7 through 9 of the above howto, then follow with step 26.

--

--

--

I ❤️ ML / DL ideas — I tweet about them / write about them / implement them. Self-taught RoR developer by trade.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Nexus Repository with Mule 4

Leetcode — Spiral Matrix

Sometimes the system will forcefully reboot, and that should be a big issue.

API versioning — A better approach

Django Custom Widget With 3 Examples

Android Paging Library tutorial with Retrofit using MVVM Architecture.

What will happen if in our IT project a Stakeholder takes a role of a Product Owner?

Snapd Must Die!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Radek Osmulski

Radek Osmulski

I ❤️ ML / DL ideas — I tweet about them / write about them / implement them. Self-taught RoR developer by trade.

More from Medium

Beginner’s guide to Amazon Web Services in 2022, Part 1: Why should you use AWS?

Getting Started Machine Learning with AWS

Get to Know About Amazon Kinesis Data Streams!

Install RStudio on AWS!

AWS, EC2, RStudio, Data Science, Statistics, Data Engineering, Server administration