Making AWS Lambda deploy faster using layers

Advertisements

I’ve been playing with AWS Lambdas for a little while, at the same time as learning NodeJS. Let’s say that I’ve uploaded quite a few new versions of my test Lambdas. Even with a small code base and minimal dependencies, I always feel like deploying is slow. Here is how I ended up making AWS Lambda deploy faster using layers. I decided to update the scripts I was using for my previous post, therefore everything can be found in this repository. More precisely, update_dependencies.sh shows an example of how to deploy a new version of a layer and update your lambda.

The problem

I timed a few parts of my deploy script to find out that the zipping process was taking quite a bit of time:

This is a whole 15 seconds to deploy a change to the lambda. You might think that this is millions of files, images, and such. But it is merely 1900 files, the Alexa SDK. Okay, some of that might be my fault, I’m using a Raspberry Pi to develop my things.

Fixing the issue

I already know about Lambda layers since I’ve used HashiCorp’s Vault AWS Lambda extension in a previous job. I was not sure if that concept could be used to store libraries. Turns out it can!

Using layers might even allow you to directly edit your code in the web interface of the lambda if it is small enough.

The first thing that I did was to manually create the layer one time through the web page. This will give you an ARN and make things easier later. The zip file that I uploaded was simply containing a random file of mine.

Uploading the layer version

Once you have the ARN you’ll be able to use the AWS cli to upload the new version of the layer. In order to do so, you’ll need to use the command: aws lambda publish-layer-version. In my case, I invoke that command line like so:

The one ambiguous part is the creation of the zip file. In my case, the only thing that will be included in the zip file will be libraries, the documentation explains the directory structure that needs to be used. Basically, for nodejs package that is not specific to a given version, you’ll need to have all your dependencies in a nodejs/node_modules directory in the zip file. There are ways to make this specific to a given runtime, the documentation explains these.

I decided to make it simple to create the dependency tree: create a temporary directory with a sub-directory named nodejs, and run npm install in there to get the dependencies. Once this is done, zip everything starting at the root of the temporary directory.

For the next step, I will need the version number of the new layer. Luckily, the return value of the command used to update the layer will give you that information. The return format is JSON formatted, I decided to use jq to extract the version. Using the great tool, extracting the version is as simple as piping the result of the update command into  jq .Version.

Updating the function to use the new version

Do note that updating the function at this point is probably a bad idea for production code. Any instance of the lambda will use the new dependencies, and might simply break.

Since I’m only playing with all this and do not mind if my lambdas break, I decided to update the functions directly in the script that uploads a new layer version. It can be done using update-function-configuration:

A few related links

Advertisements
Tagged , , . Bookmark the permalink.

Leave a Reply