Demystifying AWS API Gateway Deployments Using CloudFormation Templates

7/23/2020

Have you ever come up with a simple use case for an AWS serverless API Gateway and Lambda Function integration, but weren't quite sure how to get your API up and running from an infrastructure as code perspective? Maybe you want to be able to make a simple API call, passing in a couple of parameters, and then have a Lambda Function run a process to download some data and store a file in S3? The process seems pretty simple, afterall there are only three components to work with on the AWS side. You might get away with writing just a few lines of Python code to get the data you need and send it to an S3 Bucket for storage. But how do you make sense of all of the API Gateway bells and whistles? Better yet, how do you keep with the infrastructure as code best practices and package all of your API Gateway configs and Python Lambda Function into a CloudFormation template so that you can easily deploy and make updates?

I've got to be honest, REST APIs in Amazon's API Gateway have always seemed mystical to me, and they seem even worse when you want to configure them in a CloudFormation template. After all, you wouldn't want to have to click around the web interface to build out API endpoints every time you want to update or need to deploy your API to a different environment would you?

Now, maybe you're a fan of using a tool like SAM or The Serverless Framework to spin up and deploy your serverless projects, and I get that. Those are great tools, and I am personally a fan of The Serverless Framework, but I am also a big fan of learning things the hard way so that I can get a better understanding of how things work. So, in this post, I want to go over a little trick I figured out that could help you demystify the configuration of simple API endpoints using CloudFormation templates. It will require us to build out a "test" version of the API using the AWS Console, but once we get that built, we will see how easily it gets translated into a CloudFormation template. Then you should be able to go straight to using a CloudFormation template the next time you want to build a new API, just by referencing back to this example.

What Are We Building?

We are going to keep it simple by writing a Lambda Function in Python that will start up (or stop) an EC2 instance. The Lambda Function will be triggered by a POST request to an API Gateway endpoint, and the POST request will take three parameters (region, instanceID, and action) and pass those on to the Lambda Function. The architecture is pretty straightforward.

IAM Roles

Before we start building anything, let's create an IAM Role that will allow our Lambda Function to perform the EC2 actions with least privileges. From the IAM Console, go ahead and create a new Role and choose "Lambda" as the use case:

Next in the "Choose Permissions" stage, we want to select "Create Policy". When creating a policy, we want to select "EC2 as the Service, and then under the "Write" access level, we want to search for and select "StartInstances" and "StopInstances"

Note that in the "Resources" section of Create Policy, you can either specify certain EC2 instances to include with these permissions, or you could select "Any". Give the Policy a name of "StartStopEc2sPolicyTest" and a description and Create it.

Back to the Create Role page, we search for the new Policy we created and attach it to the Role.

Finally, give your Role a name of "StartStopEC2sRoleTest" and a description, and Create the Role.

The Lambda Function

In the AWS Console, navigate to Lambda and create a new function. Give it a name like "StartStopEC2sTest" which we will delete later in favor of ultimately deploying everything in a CloudFormation template. Select Python 3.7 as the runtime (Python 3.8 will not work with the method we will be using in the CloudFormation template later). Choose the existing IAM Role, StartStopEC2sRoleTest, that we created earlier.

Utilizing the boto3 package to interface with EC2 commands, our Lambda Function is completed with only a few lines of code. Note that when we setup API Gateway with query parameters, those query parameters will be passed through the "event" and we can fetch them by specifying the event["queryStringParameters"] keys.

import boto3

def lambda_handler(event, context):
    '''
    lambda_handler takes query parameter input from an API Gateway event,
    and starts or stops an EC2 with region and instanceId and action query 
    parameters.
    '''
    region = event["queryStringParameters"]["region"]
    instanceId = [event["queryStringParameters"]["instanceId"]]
    action = event["queryStringParameters"]["action"]
    ec2 = boto3.client('ec2', region_name=region)
    
    if action == 'start':
        ec2.start_instances(InstanceIds=instanceId)
        return {
            'statusCode': 200,
            'body': 'EC2 ' + instanceId[0] + ' in region ' + region + ' is starting.'
    }
    elif action == 'stop':
        ec2.stop_instances(InstanceIds=instanceId)
        return {
            'statusCode': 200,
            'body': 'EC2 ' + instanceId[0] + ' in region ' + region + ' is stopping.'
    }
    else:
        return {
            'statusCode': 200,
            'body': 'Invalid action ' + action + '. Nothing to do.'
    }

One last thing we'll do here is create a test event for our Lambda Function like the following:

Replace the "instanceId" with one of your own EC2 Instances, and run the test. You should see your EC2 starting up. You can also try using "action": "stop" in your test configuration to stop your EC2.

API Gateway

Now let's navigate over to the API Gateway console and "Create" a new REST API (don't choose "Private" REST API as we will not be accessing this API from within a VPC for our demonstration purposes.) Give your new REST API a name of "StartStopEC2sAPIGTest" and a description and create it.

Create Resource

Under "Resources" and "Actions", choose to create a Resource and configure it as below:

Under "Resources" and "Actions", create a new POST method, and configure it with Lambda Function integration and Lambda Proxy Integration. Don't forget to use the name of the Lambda Function that you created previously.

Now we need to configure the POST Method Execution with the three query parameters and the API Key Required settings shown below.

That's it. Now we can test by clicking on the "Client Test" and enter the Query String region=us-east-1&action=start&instanceId=i-0e2bf00f645a2cd2f. Run the test and you should see a 200 response with a body like EC2 i-0e2bf00f645a2cd2f in region us-east-1 is starting.

Now we can deploy the API by clicking on "Actions" and "Deploy API". Configure your deployment like shown below:

Once deployed, you will be able to see your API invoke endpoint URL on the next screen:

However, this is not quite ready because we chose to create the API method to use an API Key.

API Key

The last thing we need to do is create an API Key, and make it available to use with this API. Under "API Keys" choose "Actions" --> "Create API Key" and give your Key a name and description.

We also need to create a Usage Plan to associate with this API Key. You can give a low number of API requests for demo throttling purposes, and associate it with the "v0" stage we deployed earlier.

Then, on the "API Keys" tab, we can associate the API Key we created earlier with this particular API and Usage Plan:

Let's Give This A Shot!

Whew! That was a lot of point and click! But, we can finally test this out with a tool like Postman. In Postman, you can create a new POST request to your API Gateway invoke endpoint URL (don't forget to add the "Resource name" to the end of the URL) and configure the three query parameters like shown below:

Now configure the Authorization by setting the API Key's Key as "x-api-key" and "Add to Header". Use the API Key that you created earlier because these endpoints and Keys I have created won't exist anymore by the time you have read this blog :)

Now you're ready to Send the request, and your chosen EC2 should start up from this API request.

CloudFormation Template

In our CloudFormation template, we will define resources for our Lambda Function, API Gateway REST API and it's various components, and some IAM Roles and Policies, essentially just as we did in the AWS Console above. You can find the entire CloudFormation template in my Github repo if you want to follow along from there as we will only highlight a few parts of it here in the blog post.

API Gateway Resources

We define eight resources related to API Gateway in our CFT listed below by their Types:

AWS::ApiGateway::RestApi
AWS::ApiGateway::Model
AWS::ApiGateway::Stage
AWS::ApiGateway::Deployment
AWS::ApiGateway::ApiKey
AWS::ApiGateway::UsagePlan
AWS::ApiGateway::UsagePlanKey
AWS::IAM::Role

The heart of our API Gateway Resources is going to be in the type AWS::ApiGateway::RestApi. Shown in the code block below, this is where we actually define the Resources and Methods with query parameters, and such.

#cft.yml
ApiGatewayRestApi:
    Type: AWS::ApiGateway::RestApi
    Properties:
      ApiKeySourceType: HEADER
      Description: An API Gateway with a Lambda Integration to start or stop EC2s via POST requests to this API.
      EndpointConfiguration:
        Types:
          - REGIONAL
      Name: start-stop-ec2s
      Body:
        swagger: "2.0"
        info:
          description: "StartSporEC2sAPIGTest"
          version: "2020-07-27T15:58:25Z"
          title: "StartSporEC2sAPIGTest"
        basePath: "/v0"
        schemes:
        - "https"
        paths:
          /start-stop-ec2s:
            post:
              produces:
              - "application/json"
              parameters:
              - name: "instanceId"
                in: "query"
                required: true
                type: "string"
              - name: "action"
                in: "query"
                required: true
                type: "string"
              - name: "region"
                in: "query"
                required: true
                type: "string"
              responses:
                '200':
                  description: "200 response"
                  schema:
                    $ref: "#/definitions/Empty"
              security:
              - api_key: []
              x-amazon-apigateway-integration:
                uri: !Join 
                  - ':'
                  - - 'arn:aws:apigateway'
                    - !Ref 'AWS::Region'
                    - 'lambda:path/2015-03-31/functions/arn:aws:lambda'
                    - !Ref 'AWS::Region'
                    - !Ref 'AWS::AccountId'
                    - function
                    - !Join 
                      - ''
                      - - !Ref LambdaFunction
                        - /invocations
                responses:
                  default:
                    statusCode: "200"
                passthroughBehavior: "when_no_match"
                httpMethod: "POST"
                contentHandling: "CONVERT_TO_TEXT"
                type: "aws_proxy"
        securityDefinitions:
          api_key:
            type: "apiKey"
            name: "x-api-key"
            in: "header"
        definitions:
          Empty:
            type: "object"
            title: "Empty Schema"

Notice in the properties.body.paths.parameters block, we could easily add or remove any query parameters that we wish. You could also change security settings such as the API Key setting in the securityDefinitions block.

Now you may be asking how did I just come up with all of these settings to throw into the CFT like this? The answer lies in the API Gateway Console. Take a look in your Console under "Stages" and "Export"

See that section of YAML there? Everything in that code display section could be copied and pasted into our CFT in the properties.body block that we showed above. A couple of things to note when you copy/paste that section:

You can leave out the "host" key-value
Rather than using a one-liner for the x-amazon-apigateway-integration.uri, you could instead utilize the !Join methods I have shown above in my CFT to keep the API Resources generalized to your own AWS Accounts and Regions, etc.

Lambda Resources

We define five Resources related to our Lambda Function of Types:

AWS::Lambda::Function
AWS::Lambda::Permission
AWS::IAM::Role
AWS::IAM::Policy
AWS::Logs::LogGroup

Our Python code is simple enough that we can simply include it in the body of the CFT as seen below. However, this is not always the case. If your code is too much more complex, or especially if you use any third-party libraries that are not included in the Lambda runtime, then you should generally zip your code and store it in an S3 Bucket where you CFT can fetch it to deploy the Lambda Function.

LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import boto3
          import json


          def lambda_handler(event, context):
              '''
              lambda_handler takes query parameter input from an API Gateway event,
              and starts or stops an EC2 with region and instanceId and action query 
              parameters.
              '''
              region = event["queryStringParameters"]["region"]
              instanceId = [event["queryStringParameters"]["instanceId"]]
              action = event["queryStringParameters"]["action"]
              ec2 = boto3.client('ec2', region_name=region)
              
              if action == 'start':
                  ec2.start_instances(InstanceIds=instanceId)
                  return {
                      'statusCode': 200,
                      'body': 'EC2 ' + instanceId[0] + ' in region ' + region + ' is starting.'
              }
              elif action == 'stop':
                  ec2.stop_instances(InstanceIds=instanceId)
                  return {
                      'statusCode': 200,
                      'body': 'EC2 ' + instanceId[0] + ' in region ' + region + ' is stopping.'
              }
              else:
                  return {
                      'statusCode': 200,
                      'body': 'Invalid action ' + action + '. Nothing to do.'
              }
      Description: AWS Lambda function
      FunctionName: 
        Ref: LambdaFunctionName
      Handler: index.lambda_handler
      MemorySize: 256
      Role: !GetAtt LambdaIamRole.Arn
      Runtime: python3.7
      Timeout: 30
      Tags:
      - Key: Name
        Value:
          Ref: LambdaFunctionName

In the IAM Role and Policy in our CFT we create the same least privilege access like we did in the AWS Console, and you can checkout those resource sections from the CFT in the Github repo.

The only thing left to do is to deploy your CloudFormation Stack. You can simply deploy the CFT from my Github repo into your own AWS account to create all of the resources.

Once the CFT has finished deploying, you will need to navigate to your newly created API in the AWS Console and get the API Key value to use in your API calls with Postman like we did earlier.

Wrapping Up

The resources created in this blog are included in the AWS free-tier so they are essentially zero-cost unless you go on to run the API requests and Lambda Function millions of times a month. It is up to you whether you want to delete the CFT Stack you created, however, you may find this Stack useful and hang on to it.

We saw how to create a simple API with query parameters in a POST request and integrated it with a Lambda Function using the AWS Console. However, there was a lot of clicking and navigating around which is not ideal for deploying to different environments or accounts. Utilizing a CloudFormation template can be much simpler, and we saw how the heart of the API Gateway functionality can be translated in to a CFT.

Where To Go From Here?

I would encourage you to play around with the CFT a little more. Try adding a GET method to the API to query for EC2s that are running. Modifying the Lambda Function for this use case should be simple to.

To further experiment with serverless architectures, I would encourage you to explore SAM and/or The Serverless Framework for building API Gateway and Lambda integrations. See the links below for articles that can help you get started with either of those frameworks. Keep in mind that you will need to have Docker installed on your machine for these frameworks to zip your Python code for deployments, but oh-my does that make it so easy!

SAM - https://blog.heyitschris.com/posts/get-your-foot-in-the-door-with-sam/
The Serverless Framework - https://www.serverless.com/blog/serverless-python-packaging

I hope this has helped demystify API Gateway deployments for you!

Happy Hacking!