AWS auto AMI-Updater

The purpose of this blog post is to acknowledge a use case of AWS autoscaling. During auto-scaling in AWS, we often pass user-data and that user-data keeps growing with time due to which an instance spin time is increased significantly. For that purpose, we need to create latest AMI time to time with all the user-data packages pre-installed so that whenever a new instance comes up during autoscaling, it takes less time during spinning.

Whenever an activity of updating a new AMI is performed, that often takes place from a running production instance, that is not a very good practice as production environment should never be exposed to changes unless its very urgent to do so. So, currently our AMI updation process includes a lot of manual efforts to perform following tasks,

1) Identify currently used Launch Configuration in the auto-scaling group

2) Identify current AMI ID inside launch configuration, from which instance are rolled currently.

3) Spin a new instance from above found AMI ID, make all the changes (those were previously made using user-data).

4) Create a new AMI and terminate the instance.

5) Update the launch configuration with the new AMI.

6) Update the new launch configuration in the autoscaling group

So, a lot of manual work is needed to update an AMI, however these manual efforts can be minimized and the task can be automated with the use of Packer. Installation of Packer can be referred here

“Packer is an open source software for creating machine images for multiple platforms from a single source configuration. It runs on every major operating system and can create machine images for multiple platforms in parallel.”

We have written a script to perform all the above task automatically. The script uses four subscripts.

1) variables.config – The file that holds all the variable that are used in the parent script.

2) PACKER_JSON – A JSON file that will be used by packer to build and provision new AMI
3) changes_to_be_applied.sh – A shell script for updating all the changes in AMI
4) LC_SKELETON – A JSON script to create new launch configuration

Below script is the parent script (say parent_script.sh) under which all the above scripts are invoked. The command to execute parent_script.sh is listed below,

[js]bash parent_script.sh variables.config[/js]

[shell]
#!/bin/bash
set -e

DATE=`date +"%Y%m%d%H%M"`
new_lc_name="createdby-ami-updater-$DATE"

function findLaunchConfig(){
_lc=$(aws autoscaling describe-auto-scaling-groups –auto-scaling-group-names $1 –query AutoScalingGroups[*].LaunchConfigurationName –output text)
echo "Currently used Launch Configuration is $_lc"
}

function findAMIinLaunchConfig(){
_lc_ami_id=$(aws autoscaling describe-launch-configurations –launch-configuration-names $1 –query LaunchConfigurations[*].ImageId –output text)
echo "AMI used in current launch configuration is $_lc_ami_id"
}

function findAMIfromAutoscalingGroup(){
_launch_configuration=$(findLaunchConfig $1)
_current_ami_id=$(findAMIinLaunchConfig $_launch_configuration)
echo "Currently used AMI ID in launch configuration is $_current_ami_id"
}

_file_name=$1
_changes=$3

if [ $# -gt 0 ] &&  [ -f "$_file_name" ];then

source $_file_name
_ami_id=$(findAMIfromAutoscalingGroup $AUTO_SCALING_GROUP)
echo "New AMI ID to be inserted in Packer configuration from which instance will be rolled $_ami_id"

sed -i "s/$(grep source_ami "$PACKER_JSON" | cut -d’"’ -f4)/$_ami_id/g" $PACKER_JSON
new_ami_id=$(packer build "$PACKER_JSON" | tail -n 1 | cut -d" " -f2)
echo "Final AMI ID to be updated in new launch configuration $new_ami_id"
aws autoscaling create-launch-configuration –cli-input-json $LC_SKELETON –launch-configuration-name $new_lc_name

else
echo "bash updater.sh <config-file>"
fi
[/shell]

variables.config will include values of variables used in the script. These variables are shown below,

[shell]
AUTO_SCALING_GROUP="<autoscaling-group-name>"
PACKER_JSON=<json filename needed by packer>
AWS_ACCESS_KEY_ID=<aws_access_key>
AWS_SECRET_ACCESS_KEY=<aws_access_secret_key>
AWS_DEFAULT_REGION=<your_default_region>
LC_SKELETON=<LC json skeleton filename>
export PATH=$PATH:<absolute path to packer folder>
[/shell]

PACKER_JSON, as mentioned in the above variables list, holds the name of a JSON file that will be used by the packer to create an instance, make relevant changes as mentioned in the changes_to_be_applied.sh shell script shown below,

[shell]
{
  "variables": {
    "aws_access_key": "$AWS_ACCESS_KEY_ID",
    "aws_secret_key": "$AWS_SECRET_ACCESS_KEY"
  },
  "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `$AWS_ACCESS_KEY_ID`}}",
    "secret_key": "{{user `$AWS_SECRET_ACCESS_KEY`}}",
    "region": "$AWS_DEFAULT_REGION",
    "source_ami": "ami-123456",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "subnet_id": "subnet-ab123cde",
    "associate_public_ip_address": "true",
    "security_group_id": "sg-12a34567",
    "ami_name": "packer-ami- {{timestamp}}"
  }],
  "provisioners": [{
    "type": "shell",
    "script": "changes_to_be_applied.sh"
  }]
}
[/shell]

The last file that will be used by the parent script is the LC_SKELETON. Since it is a constraint that we can not copy a launch configuration using AWS CLI. We generated a JSON skeleton of AWS launch configuration in which most of the values can be copied from the previous launch configuration used by auto-scaling group except AMI ID that has to be replaced by the one created by packer. A demo LC_SKELETON file is as shown below,

[shell]
{
"ImageId": "<new AMI ID created by packer>",
"KeyName": "<keyname>",
"SecurityGroups": [
"<security group of exisitng LC>"
],
"UserData": "<commands to be executed in userdata, if any>",
"InstanceType": "m3.xlarge",

"BlockDeviceMappings": [
{
"DeviceName": "/dev/sda1",
"Ebs": {
"DeleteOnTermination": true,
"VolumeSize": 50,
"VolumeType": "gp2"
}
},
{
"DeviceName": "/dev/sdb",
"VirtualName": "ephemeral0"
}
],

"InstanceMonitoring": {
"Enabled": true
},
"EbsOptimized": true,
"AssociatePublicIpAddress": true
}[/shell]

Regards
Sharad Aggarwal