How the heck did I do that ?!

For my first useful experiment with Amazon Elastic Cloud Computing (EC2 for short), I choose remote backup. Simply put, we’ll write a simple script, that fires up an Amazon EC2 instance, attaches a storage device and sync my  valuable data to the Amazon EC2 device. Oh, and lets not forget to shutdown the EC2 instance to save us some money.

The EC2 toolkit is used for controlling EC2 services, which  is entirely controlled
from the command-line.  I choose to do all scripting in bash, my favorite for writing simple scripts which mainly interface with console software.

I’m using the same EC2 set-up as last time, make sure to download the EC2 API tools  (here) and at least set EC2_CERT, EC2_PRIVATE_KEY,  EC2_HOME and EC2_URL variables. This should be sufficient to communicate with Amazon’s EC2 webservice.

Configure and go!

Once you have that up and running (try ‘ec2-describe-images’), change the configuration inside the backup script a bit:

# Amazon EC2 settings, set your own config params!
EC2_IMG=ami-26042c52
EC2_ZONE=eu-west-1a
EC2_KEY=ec2b-keypair
EC2_VOLUME=vol-90a346f9
EC2_STORAGE_DEVICE=sdb

The EC2_IMG refers to the Amazon machine image (AMI) that is used for the remote  ’server’. The script will automatically start an instance if none is running. Set the EC2_ZONE to your preferred EC2 zone (eu-west-1a or eu-west-1b for EU users). You’ll need to have a persistent block device present in te same  zone! Create a block device or look up an existing once (using ec2-describe-volume) and set EC2_VOLUME to the instance name.

The EC2_STORAGE_DEVICE refers to a device name inside the EC2 instance, to which the EC2 volume will to attached.

Let’s change to behavior of the backup script a bit. Currently it’s set up to start a machine instance if none is running, set CONFIG_RUN_INSTANCE=0, if you manually want to start an instance.  The script always checks for running instances first, so there will never be more than one instance running at once. The next setting is CONFIG_TERM_INSTANCE, this controls if instances are shutdown when the backup has finished. Lets set it to ‘0′ for now, so we can try out different set-ups without constantly using new instances.

Don’t worry about the other settings for now (RSH_KEY, RS_EXCLUDE_LIST etc.) these are not used for any of the EC2 commands.

Example run

Okay, using the script for the first time:

jay@ns:~$ ./ec2-backup.sh
Checking EBS volume .
Finding running instance .. none found
Starting new instance .
Attaching EBS volume .

The authenticity of host 'ec2-79-125-60-240.eu-west-1.compute.amazonaws.com (79.125.60.240)' can't be established.
RSA key fingerprint is 14:fc:71:8e:90:6c:ac:52:1a:0d:13:2f:4a:45:b5:6f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-79-125-60-240.eu-west-1.compute.amazonaws.com,79.125.60.240' (RSA) to the list of known hosts.
Setting up LVM Volume Groups  Reading all physical volumes.  This may take a while...
  Found volume group "backup" using metadata type lvm2
  2 logical volume(s) in volume group "backup" now active
.
Stopping automounter: done.
Starting automounter: done.

Setting up SSH tunnel

Backing up /etc

Closing SSH tunnel

Yey, success! Had to manually accept the new SSH key there, thats something to be fixed for unattended usage. Most stuff going on here is from the do_backup function, which is supposed to execute any backup script you like. There are some useful parameters here:

• $INSTANCE_ADDRESS, contains the EC2 machine instance (dynamic) host address
• $INSTANCE_ID, Amazon EC2 instance identifier
• $INSTANCE_STATE, machine state (should be ‘running’)

Customizing

Implement the do_backup function any way you like. I’ve choosen to use it for setting up an SSH tunnel to the EC2 instance with a rsync daemon running inside the tunnel. This makes it much easier to maintain the EC2 AMI, because there is hardly any configuration required. It’s also a secure way of setting up the rsync communication, we’re re-using the EC2 key pair and there are no ‘trusted’ hosts from the EC2 machine.  This is how it’s done:

# Setup SSH tunnel
echo "Setting up SSH tunnel"
ssh -f -N -l root -i $RSH_KEY -L 127.0.0.1:2020:127.0.0.1:873 root@$INSTANCE_ADDRESS
SSH_PID=$!

The SSH tunnel is set-up using the same key that was used for starting to machine instance ($RSH_KEY). The ssh command binds local port 2020 to remote port 873, which is the default port for rsyncd. Once this is done, we can backup files by connecting to the local side f the SSH tunnel:

sudo rsync $RSYNC_OPT /etc      rsync://localhost:2020/server-ns
sudo rsync $RSYNC_OPT /home/jay      rsync://localhost:2020/server-ns

As an example, i’ve made a backup of the servers /etc directory and my homedir. Ofcourse, decide for yourself what you want to backup and place it here.  Another nice effect of using a local tcp port is that your not bothered by kernel user restrictions. I run rsync as root because access to all files is needed, while the backup script and the SSH tunnel are dont need to run in privileged mode.

Interested ?

Feel free to download and use my EC2 backup script.  If you have useful feedback or modifications, please leave a comment.

There’s plenty of room for improvement, I’ll post update versions here as I go..

Download version 0.12:  ec2-backup.sh

This entry was posted on Wednesday, January 28th, 2009 at 21:34 and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.