A very quick introduction to the pain of AWS CloudFormation

One of the main reasons IT has run like a dangerous road is that infrastructure historically was provisioned manually. However, cloud computing has enabled us to automate it. Infrastructure as code enables us to build an automated delivery process for infrastructure which gets deployed in a matter of seconds as long as that infrastructure is expressed as code. AWS CloudFormation is one of the pioneer tools that allowed infrastructure to be expressed as code.

CloudFormation 101

CloudFormation, or CFN as it's also called, is a tool to automate the creation, updating, and management of the entire infrastructure stack on AWS in a centralized fashion. Many say it's a cheesier version of TerraForm which can do more than just AWS cloud, but sometimes you just need to use that. If your infrastructure is already on AWS and you want to leverage Infrastructure as a Code (IaC), you should consider using AWS CloudFormation.

The idea behind CloudFormation is pretty simple: creating resources manually can be tedious, and while it can be automated with CLI or API queries, if you need to update the stack in the future, it will always be mostly a manual process. Instead of worrying about this, CloudFormation offers a way to model the structure and configuration of all your infrastructure and automate its deployment. So instead of writing a shell script with a bunch of AWS API calls, wait loops, and retry logic, you just specify what you want in a declarative manner and CloudFormation figure out the steps for you.

Concepts

You define all the resources that AWS needs to work in a JSON or YAML text file. In CloudFormation parlance, this file is called a template, which contains all the information you need for your product infrastructure, including resources, parameters, and configuration of those resources. You can control the version of the template file using VCS and even use it to create continuous deployment pipelines.

Aside from template, CloudFormation introduces other concepts:

Template is a declarative JSON or YAML code file that describes the proper state of the resources needed to deploy the application.
Stack implements and manages a group of resources listed in a template, and allows you to manage the state and dependencies of those resources together.
Change Set is an introductory version of the changes that will be performed by stack operations to create, update, or delete resources.
Stack Set is a collaboratively managed group of stacks, which can be replicated as a group.

CloudFormation concepts

High-level process

With these four actions, we can control the entire life cycle of a CloudFormation stack:

Create. It refers to the creation of a CloudFormation stack from a CloudFormation template. During creation, all resources specified in a template are created.
Update. When an update is triggered, CloudFormation computes a chain set, which is the action required to make the current stack match the requested state. It can then apply those changes to complete an update.
Delete. Delete action deletes all related resources. This may seem dangerous, as a user may accidentally delete a stack and take down production. But, there are multiple levels of protection against this. First, we enable termination protection on the stacks to prevent accidental deletion. Next, we specify deletion policies for stateful resources such as databases and S3 buckets that we do not want to be deleted even if the stack delete is triggered.
Rollback. If a delete, create, or update operation fails, CloudFormation attempts a rollback. Rollback is an operation the CloudFormation triggers itself in attempts to roll back all changes made by the operation to restore the stack to its state prior to the operation.

Once the template is defined, you can configure and deploy it. Once deployed, all associated stack and underlying resources will be magically created in your account. When the stack needs to be updated, existing resources will be updated with the new configuration, old resources will be deleted if they are no longer in use, and new resources will be created when they are added.

CloudFormation ensures that all the dependent resources in your template are created in the correct order, so you don't have to worry about it. For example, if you want to create an EC2 instance with a security group, CloudFormation will first create a security group and then create an instance with that security group. All you have to do is explicitly set the connection in the template.

All that remains is to figure out how to make this template that can be passed to the CloudFormation service.

CloudFormation template

As already mentioned, each template is a JSON or YAML file that describes all the resources needed to create a stack in AWS. A CloudFormation template is composed of multiple sections. The most important ones:

AWSTemplateFormatVersion — indicates the version of the AWS CloudFormation template.
Description — comments or description of the template
Metadata — can be used in the template to provide further information using JSON or YAML objects.
Parameters — for creating and passing variables when creating a stack from a template. Parameters allow our templates to be reused for creating multiple stacks with varying properties.
Mappings — mapping of keys and related values that can be used to specify conditional parameter values. This is a version of the "case" operator in CloudFormation
Conditions — there are times when you would like to have the equivalent of an if-else conditional in a CloudFormation template. Conditions define whether certain resources are created or when resource properties are assigned to a value during stack creation or updating. Conditions can be used when you want to reuse the templates by creating resources in different contexts.
Transform — builds a simple declarative language for AWS CloudFormation and enables the reuse of template components. Here, you can declare a single transform or multiple transforms within a template.
Resources — The most important section of the template, describing the actual AWS resources included in the stack
Outputs — describes the values that are returned whenever you view your stack properties. This is displayed in the AWS CloudFormation console

Template example

The best way to understand CloudFormation is to go hands on with a simple CloudFormation template.

Let's take a look at the anatomy of the CloudFormation template example. We are going to deploy an EC2 instance and a security group from the official AWS example.

I prefer the YAML format, so the first thing to do is to convert JSON to YAML using cfin flip service. Just copy the result into the separate local YAML file.

EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType:
        Ref: InstanceType
      SecurityGroups:
      - Ref: InstanceSecurityGroup
      KeyName:
        Ref: KeyName
      ImageId:
        Fn::FindInMap:
        - AWSRegionArch2AMI
        - Ref: AWS::Region
        - Fn::FindInMap:
          - AWSInstanceType2Arch
          - Ref: InstanceType
          - Arch

Here we have several uses for Ref. Ref is a way to refer to values from other parts of the template.

For example, Ref: InstanceSecurityGroup refers to the only other resource in this pattern-the SecurityGroup to be created. Here is the definition of that resource:

InstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
      - IpProtocol: tcp
        FromPort: '22'
        ToPort: '22'
        CidrIp:
          Ref: SSHLocation

The above shows the Security Group configuration for our EC2 instance. We apply one rule for incoming traffic SecurityGroupIngress — we open 22 port. This one is to be able to ssh to the machine.

Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair to enable SSH access to the instance
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: must be the name of an existing EC2 KeyPair.
  InstanceType:
    Description: WebServer EC2 instance type
    Type: String
    Default: t2.small
    AllowedValues:
    - t1.micro
    - t2.nano
    - t2.micro
    - t2.small
    - t2.medium

Now we define any values that might be customized by the end-user are defined as parameters. Here we expose two parameters:

The instance type, defines the hardware associated with our VM.
The key pair is used to secure the instance. It is expected that a key pair has already been created in the account.

Stack creation

Once you have written the template, you can create entire environments in a matter of minutes (ahem, tens of minutes) which will be needed for entire groups or even departments of developers, testers, and so on. You can even put parameters that can be adjusted in the template parameters: machine names and sizes, disk sizes, etc.

A big advantage of all this is also that all environments are uniform and identical, which means that everything tested in one place will work in the same way in another (with few access headaches).

Now that we have everything in place, let’s tell AWS to deploy our infrastructure! Wait a second, let’s validate it first:

aws cloudformation validate-template \
	--template-body file://EC2InstanceSample.yml

Output:

{
    "Parameters": [
        {
            "ParameterKey": "KeyName",
            "NoEcho": false,
            "Description": "Name of an existing EC2 KeyPair to enable SSH access to the instance"
        },
        {
            "ParameterKey": "SSHLocation",
            "DefaultValue": "0.0.0.0/0",
            "NoEcho": false,
            "Description": "The IP address range that can be used to SSH to the EC2 instances"
        },
        {
            "ParameterKey": "InstanceType",
            "DefaultValue": "t2.small",
            "NoEcho": false,
            "Description": "WebServer EC2 instance type"
        }
    ],
    "Description": "AWS CloudFormation Sample Template EC2InstanceWithSecurityGroupSample: Create an Amazon EC2 instance running the Amazon Linux AMI. The AMI is chosen based on the region in which the stack is run. This example creates an EC2 security group for the instance to give you SSH access. **WARNING** This template creates an Amazon EC2 instance. You will be billed for the AWS resources used if you create a stack from this template."
}

Seem legit. Next, we are going to estimate the cost of the CloudFormation stack:

aws cloudformation estimate-template-cost \
	--template-body file://EC2InstanceSample.yml \
	--parameters ParameterKey=KeyName,ParameterValue=newcluster ParameterKey=InstanceType,ParameterValue=t2.micro

As a result of this operation you will get a URL to the AWS calculator. Click it and go through it — to understand what we're paying for.

Next, creating the CloudFormation stack:

aws cloudformation create-stack \
	--template-body file://EC2InstanceSample.yml \
	--stack-name single-instance \
	--parameters ParameterKey=KeyName,ParameterValue=cfn-test ParameterKey=InstanceType,ParameterValue=t2.micro

As an output you will get the stack ID of the stack that is currently being created:

{
    "StackId": "arn:aws:cloudformation:us-west-2:000000000001:stack/single-instance/a34bb5e0-8fc5-11ec-8f24-0ad6c0c87207"
}

In the AWS console, you can see that the EC2 instance and the EC2 security group have been created. Let's find a working instance so we can get the public DNS to log in. We can find this in the EC2 console.

Now connect to the instance via ssh.

$ ssh -i cfn-test.pem ec2-user@ec2-35-222-67-233.us-west-2.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
32 package(s) needed for security, out of 57 available
Run "sudo yum update" to apply all updates.
[ec2-user@ip-172-35-0-222 ~]$ uptime
 08:00:21 up 2 min,  1 user,  load average: 0.05, 0.05, 0.02
[ec2-user@ip-172-35-0-222 ~]$

$ uptime
08:01:23 up 3 min,  1 user,  load average: 0.02, 0.04, 0.01

Cloud Formation does not watch over your resources after they’re created. Therefore, if you make any manual modifications to the resource that is maintained by CloudFormation stack, the stack itself won’t be updated. A situation where the actual infrastructure state is different from its definition is called configuration drift. CloudFormation comes in handy and lets you see the actual drift for the stack in the console – change something in the console and try this:

aws cloudformation detect-stack-drift \
    --stack-name single-instance

Let's destroy the resources so you don't get charged more money than you need to:

aws cloudformation delete-stack --stack-name single-instance

Service Catalog

Although AWS Service Catalog is built on top of CloudFormation, unlike CloudFormation it is used for different purpose. It designed to share common applications. This means you can share the Service Catalog with multiple accounts and copy the same security restrictions and common best practices. In this way, it will allow you to highlight and reuse recurring services or products.

AWS Service Catalog can be used to create complex catalogs in the areas of multi-account setup, configuration, and creating and updating AWS services. AWS Service Catalog also provides the flexibility to manage and organize multiple accounts from a central account.

It operates with the following entities:

Products. Each imported CloudFormation template can be termed as a product, and the CloudFormation template can have as many numbers of valid AWS resources. Multiple version of the same product is created by importing the updated CloudFormation template.
Portfolio. Multiple products can be assigned to a Portfolio. Portfolios help manage product configuration, who can use specific products, and how they can use them. Users, Roles, and Groups can be added to Portfolio for granting access to the products added in Portfolio. A portfolio can be shared with member accounts by referencing an existing organizational unit or organization ID without leaving the AWS Service Catalog.
Catalog: A collection of products that the administrator creates by importing AWS CloudFormation templates.
Constraints: Rules that limit how the AWS CloudFormation template for a product is deployed.

The Service Catalog is used to standardize, structure teams, and reuse products within an organization — ultimately to increase the productivity of entire development teams.

As an example, let's say the development team typically runs a test server that updates a security group with access to a VPN. Chances are there are other developers who perform similar actions by writing their own versions of scripts.

This is where Service Catalog brings value – by providing self-service to end users, allowing them to provision preapproved resources defined for their use. This in turn improves team productivity and unifies the company's approach toward creating similar types of infrastructure in the future. It also promotes resource standardization, compliance, and consistent service offerings across different environments by launching resources through a predefined products.

Summary

AWS CloudFormation offers an easy way to model the necessary AWS resources, allocate them and manage them throughout their lifecycle by working with them as code.

As the post title suggests, Cloudformatiin does not provide the best user experience. There are a lot of articles about the pains of using CloudFormation, for me the top four:

Slow as f**k
Lack of debugging information
Many unnecessary limitations on the stack size and values
Not flexible

CloudFormation is free of cost, so at least we shouldn't pay for that. The only fee that users incur is the cost of AWS service provisioned by CloudFormation and, of course, the nerves spent on testing and configuring a working template. It's better be used with AWS CDK or sceptre or switch to Terraform-based IaC.