Deploying Lambda Functions with Terraform

Tue, 16 Jun 2020 20:05:28 +0000

Serverless architectures are built from a collection of Managed service wired together to create a specific service. This could be anything from a shopping cart to a full ETL solution. The advantage of building applications this way is that each of the component parts are fully managed by AWS. That means you (either the developer or company) don’t have to worry about them functioning, you just need to make sure they’re wired together correctly.

Managed services are great, but these service only offer basic functionality such as queues or databases. What they can’t do is know how your business actually adds value to the world.

Lambda functions are the Serverless super-hero because they let you run your own code without ever having to worry about how the server operates (yes, there are servers!) or even how how to install the software that runs your code.

There are a lot of ways to deploy AWS infrastructure, and even more for Serverless infrastructure, but my personal favourite is Terraform. Its designed to allow you to write your infrastructure as code (IaC) and is surprisingly flexible. It also allows you to interact with non-AWS services which can really help tie everything together.

Terraform can be a little verbose, but its very easy to understand, and very simple to work with. On the face of it Terraform doesn’t obviously lend itself to Serverless, but that’s because it designed to give you extremely low-level access to APIs in the form of Resources.

In this article I will show you the minimal set of code required to deploy a lambda function. Its not going to be production-ready, or even do anything very impressive, but its going to introduce at an entry level the minimal components required.

About Terraform

Terraform works by creating a direct-graph of resources. The resource dependency order can be determined by using outputs from one resource as the inputs to another. Terraform can then deploy resources which have no outstanding dependencies recursively until all resources have been deployed with their dependencies satisfied.

Terraform then creates a state file which it uses on further deployments to identify which specific resources in AWS it should be adding, deleting or modifying. State files can be stored in local files or remotely, but for now we will just keep them locally.

Terraform Meta-Resources

Terraform is incredibly configurable, and to do that it happily eats it own dog-food by configuring with Terraform resources! This is the content of a file I like to call meta.tf:

terraform {
 required_version = "> 0.12"
}
provider "aws" {
 region = "eu-west-1"
 version = ">= 2.61.0"
}

The terraform block tells Terraform to use a version no less that 0.12.0. While older versions are available, this version added some functionality that makes it a little easier to code and so is a good base.

The provider block tells Terraform that we want to use AWS resources, we want to place them in the eu-west-1 region and we want to use a provider version no less than 2.61.0

Creating the Lambda Function Bundle

A Lambda Function needs some code to run. The code can be in a variety of languages but in this case we’re going to use Javascript. Here is some simple code that we’re going to deploy:

To supply the code to the Lambda Function it needs to be in a Zip file. Terraform can create these zip files for us, and for simplicity we’re going to define the Function code inline:

data "archive_file" "lambda" {
 type = "zip"
 output_path = "${path.module}/.terraform/lambda.zip"
source {
 filename = "index.js"
 content = <<-EOF
 module.exports.handler = async (event, context) => {
console.log("EVENT: \n" + JSON.stringify(event, null, 2))
console.log("CONTEXT: \n" + JSON.stringify(context, null, 2))
return Promise.resolve()
}
EOF
}
}

Lines 7-10 will be written to a file called index.js which will be added to a zip file called lambda.zip. ${path.module} will be interpolated as the local path during deployment. The code exports a function called handler. Keep these in mind as they will be useful when we actually deploy our function.

The code is extremely simple: it’s going to print out the Event and Context objects which will be passed to it upon execution. This is how the function will get information related to its execution.

The Log File

Since the Lambda Function is running inside of AWS, we need some way to capture its output. The Lambda service automatically captures anything which would normally be sent to stdin and stderr and store that in a CloudWatch Log Stream. Although the Lambda service will create the Log Group, its best practice to create this yourself and set an expiry period so that the logs are automatically cleared out after a period of time:

resource "aws_cloudwatch_log_group" "lambda_log_group" {
 name = "/aws/lambda/${aws_lambda_function.lambda.function_name}"
 retention_in_days = 30
}

All Lambda functions log to a log group called /aws/lambda/<lambda name> and so we’re creating that and capturing the name from the Lambda Function resource (created very soon!)

We also set the logs to expire after 30 days. Logs are pretty cheap, but if the function is called a lot or it generates a lot of log output the cost can quickly mount up so after a short period we want AWS to delete them on our behalf.

Permissions

In order to be able to interact with any other service, AWS resources need an IAM Role which can be “assumed” by the calling service. In our case that’s the Lambda service:

resource "aws_iam_role" "lambda" {
 name = "simplest_lambda_role"
 assume_role_policy = <<-EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
]
}
EOF
}

The role needs to have policies associated with it in order to define what its allowed to do. In our case the only thing we need permission to do is to create Log Streams within the Log Group, and write to them. Policies can be stand-alone, re-usable policies, or in-line policies specific to this function. For simplicity in this case we’re going to create an in-line policy:

resource "aws_iam_role_policy" "logging" {
 name = "Cloudwatch-Logs"
 role = aws_iam_role.lambda.name
 policy = <<-EOF
{
"Statement": [
{
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": "arn:aws:logs:*:*:*"
}
]
}
EOF
}

The Lambda Function

At last we have all the pieces in place to be able to create our Lambda Function resource. Here we will give it a name, point it to our code bundle, tell it what runtime to use and what exported function to execute and what Role it should be assuming to give itself the correct permissions:

resource "aws_lambda_function" "lambda" {
 function_name = "simplest_lambda_function"
 filename = data.archive_file.lambda.output_path
 source_code_hash = data.archive_file.lambda.output_base64sha256
 runtime = "nodejs12.x"
 handler = "index.handler"
 role = aws_iam_role.lambda.arn
}

Terraform hides some nasty internal values such as ARNs and file hashes by taking the outputs from some of the resources we created earlier as inputs to this resource. In doing so we have also implied the dependencies of resources and helped Terraform work out what order things should be deployed.

Deploying the Function

To start the deployment process, Terraform needs to be initialised with the command:

terraform init

This will ensure we’re using a valid version of Terraform and download the providers specified in meta.tf (or a default if we didn’t specify a version). All of its internal “stuff” will be stored in a directory called .terraform

To see what Terraform is going to do execute the following command:

terraform plan

You should get an output something like:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
data.archive_file.lambda: Refreshing state...
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:

 # aws_cloudwatch_log_group.lambda_log_group will be created
 + resource "aws_cloudwatch_log_group" "lambda_log_group" {
 + arn = (known after apply)
 + id = (known after apply)
 + name = "/aws/lambda/simplest_lambda_function"
 + retention_in_days = 30
}

 # aws_iam_role.lambda will be created
 + resource "aws_iam_role" "lambda" {
 + arn = (known after apply)
 + assume_role_policy = jsonencode(
{
 + Statement = [
+ {
 + Action = "sts:AssumeRole"
 + Effect = "Allow"
 + Principal = {
 + Service = "lambda.amazonaws.com"
}
},
]
 + Version = "2012-10-17"
}
)
 + create_date = (known after apply)
 + force_detach_policies = false
 + id = (known after apply)
 + max_session_duration = 3600
 + name = "simplest_lambda_role"
 + path = "/"
 + unique_id = (known after apply)
}

 # aws_iam_role_policy.logging will be created
 + resource "aws_iam_role_policy" "logging" {
 + id = (known after apply)
 + name = "Cloudwatch-Logs"
 + policy = jsonencode(
{
 + Statement = [
+ {
 + Action = [
+ "logs:CreateLogGroup",
+ "logs:CreateLogStream",
+ "logs:PutLogEvents",
]
 + Effect = "Allow"
 + Resource = "arn:aws:logs:*:*:*"
},
]
}
)
 + role = "simplest_lambda_role"
}

 # aws_lambda_function.lambda will be created
 + resource "aws_lambda_function" "lambda" {
 + arn = (known after apply)
 + filename = "./.terraform/lambda.zip"
 + function_name = "simplest_lambda_function"
 + handler = "index.handler"
 + id = (known after apply)
 + invoke_arn = (known after apply)
 + last_modified = (known after apply)
 + memory_size = 128
 + publish = false
 + qualified_arn = (known after apply)
 + reserved_concurrent_executions = -1
 + role = (known after apply)
 + runtime = "nodejs12.x"
 + source_code_hash = "TFoUgCw/pW7Sy3/gY/TB3AtGzXoqRjN1L6YT066iPGg="
 + source_code_size = (known after apply)
 + timeout = 3
 + version = (known after apply)
+ tracing_config {
 + mode = (known after apply)
}
}
Plan: 4 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

This shows that Terraform will create 4 new resources:

The Log Group
An IAM Role
An Inline IAM Policy attached to the Role
Our Lambda Function

To actually create resources in AWS, execute:

terraform apply

When prompted, enter yes

Testing our Function

To invoke our function from the command line, execute:

`aws lambda invoke --function-name simplest_lambda_function /dev/null --log-type Tail --query 'LogResult' --output text | base64 -d`

which should respond with something similar to:

START RequestId: ab934699-7094-4829-8cee-aecd2c193583 Version: $LATEST
2020-06-16T16:58:46.098Z ab934699-7094-4829-8cee-aecd2c193583 INFO EVENT:
{}
2020-06-16T16:58:46.098Z ab934699-7094-4829-8cee-aecd2c193583 INFO CONTEXT:
{
"callbackWaitsForEmptyEventLoop": true,
"functionVersion": "$LATEST",
"functionName": "simplest_lambda_function",
"memoryLimitInMB": "128",
"logGroupName": "/aws/lambda/simplest_lambda_function",
"logStreamName": "2020/06/16/[$LATEST]759235165de84d3bbe7d3a87a7bd2db4",
"invokedFunctionArn": "arn:aws:lambda:eu-west-1:522711524578:function:simplest_lambda_function",
"awsRequestId": "ab934699-7094-4829-8cee-aecd2c193583"
}
END RequestId: ab934699-7094-4829-8cee-aecd2c193583
REPORT RequestId: ab934699-7094-4829-8cee-aecd2c193583 Duration: 2.86 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 63 MB Init Duration: 129.60 ms

While the command was a little long-winded, this is the log data you will also find in the CloudWatch Log Group. We can see the invocation id ab934699-7094-4829-8cee-aecd2c193583 and the content of the event and context objects. Finally we can see the REPORT line with tells us

The function took 2.86ms to run
It will be billed at 100ms (always rounded up to the nearest 100ms)
It was allocated 128MB of memory
At its peak, it actually used a total of 63MB of memory
It “Cold-Start” initialisation time was 129.6ms

You can use these values to tune the function which I will address in a future post.

Cleaning Up

Since this was just a simple demo, you probably want to remove all these resources at some point. To do so execute:

terraform destroy

Final Remarks

This function doesn’t do a great deal, and right now we can only invoke it from either the command line or from the AWS Lambda console, but it covers the basics of using Terraform to get it, and all its supporting resources deployed.

For more information see the official Terraform site.

serverless on Barney Parker