I know it’s been a while since I had a post, but this one was a doozy in the making. At work I am often using CloudFormation and when AMI’s get updated and then retired I have to spend time hunting down AMI’s in the console or using the CLI. It’s a thorn in my side and time consuming. Here, I will show you how you can use a process to automate the whole process.
Let’s being…
I have recently developed a new process for grabbing the latest AMI’s for Ubuntu 16.04, Ubuntu 18.04, Windows 2012 (R2)/2016, Windows 2012/2016 with SQL 2016, AND Windows 2016/2019 with SQL 2017 for the following AWS Regions:
ap-northeast-1
ap-northeast-2
ap-south-1
ap-southeast-1
ap-southeast-2
ca-central-1
eu-north-1
eu-west-1
eu-west-2
eu-west-3
sa-east-1
us-east-1
us-east-2
us-west-1
us-west-2
The process is a crontab task that invokes a wrapper bash script that runs another bash script that reaches out to AWS and gets each AMI and writes it to file. The second part of the wrapper bash script then copies that file up to a publicly facing S3 bucket. The output is proper JSON and can be used in CloudFormation templates invoking the Fn::Transform intrinsic function, much like an include file. I will demonstrate how to set this up and use it in CFN templates.
The process can run on a t3.nano, or your free tier instance, in your account. The cost is $3.796 per month. It sends the document to your S3 bucket, the bucket should be publicly accessible. The EC2 instance should have a profile attached that has read only access to SSM (describe) and EC2 (describe) and full access to the S3 bucket, and no no other permissions.
How to Setup the Process
Step 1 – Build a t3.nano on a public Internet subnet, or in a private subnet with a VPN. You can leave all defaults to the build and let it spin up.
Step 2 – Create an IAM policy with this JSON:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:Describe*", "ssm:Get*", "ssm:List*" ], "Resource": "*" }, { "Effect": "Allow", "Action": "s3:*", "Resource": "arn:aws:s3:::##YOUR BUCKET NAME GOES HERE##/*" }, { "Effect": "Allow", "Action": "ec2:Describe*", "Resource": "*" } ]} |
Attach that to a new role, then attach role to the EC2 instance.
Step 3 – Using the code from this github repo/directory, https://github.com/davidadixon/getamis.git, login into the EC2 instance and create 2 files at the current directory level called rungetami.sh and getami.sh. Copy the contents of each file from the github code, respectively. – Or – you can clone the repo to your local folder, be careful to make sure the paths in all the files are correct.
Step 4 – Create a cron job using the format and task found in the cron.job file in the github repo.
crontab -e is your friend here.
This will run the task, get the AMI ID’s, write them to a file and copy up to the S3 bucket.
Step 5 – Access the bucket to find the AMI you need or put it in CloudFormation using the demo below.
The process runs every day at 6am Eastern Time and takes about 1-2 minutes, from cron job to file being available. Assuming you build it ins us-east-1.
How to Use in CloudFormation
Whether you use JSON or YAML, it does not matter and this will work.
The best way to use this is by inserting it as a Mapping under the Mapping section of your file, done by doing this:
JSON
1 2 3 4 5 6 | "Fn::Transform": { "Name": "AWS::Include", "Parameters": { "Location": "s3://##YOUR BUCKET NAME GOES HERE##/amis.json" }} |
YAML
1 2 3 4 | Fn::Transform: Name: AWS::Include Parameters: Location: s3://##YOUR BUCKET NAME GOES HERE##/amis.json |
This will pull the Mapping into your CloudFormation template automatically. To use the Mapping, when you want to build an EC2 instance call it using this:
JSON
1 2 3 4 5 6 7 8 9 | "ImageId": { "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region" }, "Ubuntu18" ]} |
YAML
1 2 3 4 5 | ImageId: Fn::FindInMap: - RegionMap - Ref: AWS::Region - Ubuntu18 |
In each case, make sure to enter your OS type where here it is Ubuntu18. The options are as follows:
Ubuntu16
Ubuntu18
Windows2012
Windows2016
Windows2012SQL2016
Windows2016SQL2016
Windows2016SQL2017
Windows2019SQL2017
You can add a parameter to make it an option to choose when building using this these two snippets, one for the Parameters section and the other in the EC2 resource section, like this:
JSON
1 2 3 4 5 6 7 8 9 | "WindowsOS": { "Description": "OS Version for all non SQL servers.", "Type": "String", "Default": "Windows2016", "AllowedValues": [ "Windows2016", "Windows2012" ]} |
WITH:
1 2 3 4 5 6 7 8 9 10 11 | "ImageId": { "Fn::FindInMap": [ "RegionMap", { "Ref": "AWS::Region" }, { "Ref": "WindowsOS" } ]} |
YAML
1 2 3 4 5 6 7 | WindowsOS: Description: OS Version for all non SQL servers. Type: String Default: Windows2016 AllowedValues: - Windows2016 - Windows2012 |
WITH:
1 2 3 4 5 | ImageId: Fn::FindInMap: - RegionMap - Ref: AWS::Region - Ref: WindowsOS |
A few notes…
When using this inline with CloudFormation, the event log in CloudFormation will have it’s second entry say “Transform Succeeded”. This means the “include” worked and was properly parsed. Otherwise, it will instantly roll back.
I will try in the future make the entire build process a CloudFormation template, but as it’s working now, that is not high priority.
I will also try to make this entire process serverless in Lambda, but as this costs under $4 a month right now, and Lambda charges you by the ms (this process takes up to 120,000 ms) it cost more in Lambda as written.
If you use this in a CloudFormation template, be aware that an updated AMI may rebuild your instance in the case of a stack update. This is not always the case and there are two many situations to review as an example, but just be aware that is a possibility.
Also, if you have an recommendations on how to improve or AMI’s that should be added feel free to let me know or experiment on your own.
Lastly, you may have to install awscli depending on what Linux version you use. I use Ubuntu the most and I need to install it. On Amazon’s Linux instance I believe it is already installed.
Reference links:
Fn::Transform
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-section-structure.html
Mappings
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/mappings-section-structure.html
Parameters
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html
Github (source code)
https://github.com/davidadixon/getamis.git
Happy Building!
