Serverless Loose Ends

I amassed a few tidbits of nice information while playing with Serverless over time. These are all too small to do a real post about, so I decided to outline a few in a single one. This time, I will go over two subjects: make simple project deployment faster and using the same S3 bucket for all your projects. Here is how I ended up fixing these two Serverless loose ends.

Make simple project deployment faster

Most projects for this blog have the same shape: they require no external dependency except the AWS SDK and serverless. Both of these are completely useless at runtime: serverless is dev-only and AWS SDK is already provided in lambdas.

My issue is that it looks like serverless removes development dependencies instead of installing them in a clean directory. The side effect of this is that it takes quite a bit of time to remove all those small files. And all that work to end up with an empty directory anyway. For example, deploying a simple project would take almost 6 minutes:

I decided to look around and found a forum post trying to fix this issue (see here). The fix described in the post is to simply tell serverless to not care about your dependencies and exclude the node_modules folder completely:

Once this was done, deployment time went down quite a bit to less than 4 minutes:

Using a single S3 bucket for multiple projects

By default, serverless will generate an S3 bucket per project. This might not be an issue for small businesses or a simple playground, but it can become an issue later. Amazon has put a soft limit to 100 S3 buckets per account. This limit can be increased to a maximum of 1000 (source: link). This might seem like a lot, but you could also be using buckets for a lot of things. Of course, this limit can be bypassed by creating new AWS accounts, but it becomes a nightmare once you start accessing resources cross-account.

Following a forum post (here), I discovered the deploymentBucket option. This needs to be added to your provider section and needs to contain the ARN of the bucket to use. For example, I use the following in my playground:

I wanted to make things simple for me and not have to hardcode anything. At the same time, I wanted to play with CloudFormation, so I decided to expose the bucket through a CloudFormation stack. In the previous sample, the name I use for the deployment bucket is ${cf:ServerlessRoot.ServerlessDeploymentBucketArn}. This tells serverless to get the CloudFormation stack called ServerlessRoot. Then, it extracts the output named ServerlessDeploymentBucketArn from it.

The Stack I deployed can be found here. And a quick helper script to deploy it. Right now, it contains only the deployment bucket, but maybe I’ll add some IAM stuff in there too.

Serverless Plugin for S3 security and Git status

I’ve been using Serverless at work for a little bit of time. While doing that, I’ve been exploring it even more on my own time. There are a few things that I want on all my projects and don’t want to do manually every time. Amongst other things, I want my S3 buckets to never be public and I want some kind of git status on my lambda. I know there are plenty of Serverless plugins to do just that, but I want to learn how to do a plugin. Here is how I ended up writing a Serverless Plugin for S3 security and Git status.

As always, the code I worked on for this sample can be found on my GitHub.

Plugin goals

Basically, I wanted my plugin to be used as a master plugin for my other projects. I want it to do the following:

  • Store the SHA, branch, and user that deployed
  • Ensure that all my S3 buckets can’t be made public (using PublicAccessBlockConfiguration)
  • Ensure that all my log groups have a retention policy
  • Validate the stage that I’m using for deployment to select environment configuration

Since I wanted to see how to setup custom options, I allowed the user to whitelist some S3 buckets to not receive the block.

Writing the plugin

Plugin bootstrap

Creating the base of the plugin is simple enough. Still, I decided to use serverless to create the skeleton for me using serverless create --template plugin. Running this command will create a index.js file containing the boilerplate for the plugin. You will then need to build a package around that file.

Registering to the hooks

Integrating your plugin into Serverless’s workflow is done through hooks. Everything is a step and you can hook before or after any step. You can even create new ones when you add commands. Finding documentation on these steps is quite hard. The best documentation of the flows I could find is on this gist.

After investing quite a bit of time I found the two places where I needed to hook my plugin. The first one was after the package:initialize step. In my code, hooking to this step can be found here. Using this hook, I validate that the configuration used makes sense for me.

The second hook I needed is used to inject into the resources. This needed to be far enough in the flow so that resources are already created, but not so far so the CloudFormation templates are not yet generated. I found that hooking before the package:finalize step gets that done. This part can be found here in my code.

Finding all generated resources

Serverless can store your resources in two different places depending on how they got created.

If a resource is added through the resources section of serverless.yml it will end up in serverless.service.resources.Resources. An example of this is my public assets S3 bucket.

If a resource is generated by Serverless itself, for example, a deployment bucket, it will end up in serverless.service.provider.compiledCloudFormationTemplate.Resources.

To make handling these two types of resources easier, I wrote a quick wrapper to iterate on both of these.

Using custom options from serverless.yml

To make my plugin a bit more usable, I wanted to see how to interact with options. These are added in the custom section of your serverless.yml file.

In your plugin code, you’ll be able to access the configured values through the serverless object. For example, in my code, I access  serverless.service.custom.core.skippedBuckets here. I use this value to skip some buckets and allow these to be public.

Getting the git repository status

I wanted to keep my code as self-contained as possible. In order to do so, I wanted to refrain from using a shell command to get the git information. I looked at NodeGit and that was exactly what I needed. Sadly, installing the package on my RaspberryPi ended up taking way too long and I decided to simply use shell commands.

I ended up using 4 commands to get the information I needed. These can be found here in my code, nothing too fancy around there.

Using the plugin

In order to use the plugin, you’ll need to do two things. The first one is to add the dependency to your package.json. In my case, I decided to simply use a file path to refer to the project. Once this is done, you’ll need to register the plugin in your serverless.yml file.

If your plugin requires any special configuration, you’ll be able to add it into the custom section of your serverless.yml file. In my case, I added a whitelisting for one of my S3 buckets. This was done by adding the following code section:

Related links

Service Discovery Through Lambda Extension

I started to read more and more about serverless architecture and the power of AWS Lambda. Amongst other things, Lambda extensions caught my eye. In a few words, this allows you to run code before or after your Lambda executes and also during the initialization or shutdown of your Lambda. There are already quite a few extensions in the wild. The first scenario I wanted to explore was a poor-man Service Discovery Through Lambda Extension.

As always, the code associated with this experiment is available on my GitHub repository.

Lambda extensions

Lambda extensions are a relatively new thing: they were introduced at the end of 2020. In order to keep things simple, Amazon built extensions on top of layers. Basically, you’ll need a special directory structure to signal to the runtime that your layer is an extension. Then, you register your extension like any other lambda layer.

I could not find any official documentation on that directory structure and had to rely on samples to figure it out. It turns out that you will need the following structure in your ZIP for your layer:

  • extensions: this directory needs to contain the executable that will start your extension
  • myExtensionName: this directory will contain your extension code

It is worth noting that nothing forces you to use a specific language to write your extension. You could write your extension in GO and still use it in a nodejs lambda.

Basically, you can see the extension as an external process that runs in the same context as your lambda. A little like a Kubernetes Sidecar.

My actual extension

Service discovery is always a complex topic in a microservices system. Therefore the extension I wrote tries to solve this problem. It allows a deployment tool to insert services, their version, and ARN into DynamoDB. Users of the extension can then set an environment variable named  NEEDED_SERVICES on their lambda. This environment variable supports a comma-separated list of services. It even handles a quick notation  MyService==1.2.0 to get a specific version and not the latest one.

When the extension is initialized, it does a few things:

  1. Read the environment variable to get the needed services
  2. Get the ARNs for the services from DynamoDB
  3. Write a file containing the discovered services
  4. The lambda can then read the file and knows how to contact the other services

Of course, none of this is usable as-is and is simply me playing. There are better ways of doing the same thing as this extension is doing.

Writing the extension

Entry point

The entry point of your extension is an executable file found in the extensions/ directory of your layer’s ZIP file. Make it easy for yourself and name it the same as the directory containing your code. A sample of this executable can be found here. The name of this executable will be important later, choose it well. Remember to set that file as executable, else your extension won’t work.

Registering the extension

One of the first things to do in your extension is to register it to the lambda service. This is done through an HTTP call. For example, the register call in my extension can be found here. There are two points to note in this call:

  • The events types to listen for, the only valid values are  INVOKE and SHUTDOWN.
  • Most importantly, the  Lambda-Extension-Name header needs to be exactly the same string as your executable from the extensions directory

If you are registering for SHUTDOWN events and are getting errors like ShutdownEventNotSupportedForInternalExtension: you haven’t passed the right extension name in the register call.

Polling for events

Once your extension is registered, you will be able to poll for events associated with it. Doing so is, again, done through an HTTP call and can be found here in my playground. You can then do a good old  while ... true to poll and process the events, like here. In my example, I do not care about invocations. Therefore, only the shutdown event is handled.

Remember to be a good neighbor and handle SIGINT and SIGTERM. These two signals could be received in some cases and you should shut down your extension. Handling process signals in nodejs is done trivially using process.on(signal, function), for example: here.

Using the extension

Amazon made it really easy to use extensions: they are configured as layers on a lambda. Therefore, you can add an extension really easily through any deployment tool. In my case, I decided to use serverless, so adding the layer is done when declaring the function, see this section.

Related links

Empty space between tiles in Unity

I decided to dabble with Unity recently. The first project I wanted to play required me to use tilesets and tilemaps. I ended up finding a free tileset and followed a few tutorials, but whatever I did I always ended up with weird space between my tiles. Here is how I ended up fixing the empty space between tiles in Unity.

The problem

Empty space between tiles in Unity

Empty space between tiles in Unity

While looking at the scene view, I could not see the empty space but when I was switching to the game view I saw the image on the right. Since I was spending most of my time in the scene view, it took a while before I found that things were broken.

Once I saw the issue, I played with a few settings, trying various things. Double checked my tiles, their size, the import flow. Everything looked right. The one thing I never modified was my tileset. I assumed that it was built correctly since the import showed the tiles correctly. In hindsight, I should have investigated the tileset earlier.

After quite a bit of digging, I figured that in some cases (depending on the camera, pixel rounding, and such), Unity will go and pick pixels a little bit outside of the tile, taking the pixel next to the tile from the tileset. In my case, that pixel was transparent, so I ended up with transparent pixels in my tilemap.

The Solution

Tilemap using the fixed tileset

Tilemap using the fixed tileset

After reading a few proposed solutions, and trying them, the only one that worked for me was quite simple: add a bleed around my tiles. At that point, I was still exploring and was using only one tileset with a few tiles. This made manually editing the tileset easy enough.

A few sources that I investigated

Filesystem auto-complete while using docker in WSL

Besides simply using the command line, mounting directories is quite painful when using docker on Windows. Therefore, I tend to use WSL to run my docker commands. Still, auto-complete of local file and directories (when using docker’s -v option) is quite helpful. Here is a little bit of information on getting filesystem auto-complete while using docker in WSL.

For my local setup, I use Virtual Box and Docker Toolbox to run docker in a linux VM and you can find how I linked WSL with that VM in my other post.

My final goal was to be able to do a simple:
docker run –rm -v /mnt/d/data:/data -i -t calestar/ebook-utils COMMAND
where autocomplete for local directories (/mnt/d/data) works correctly in WSL.

In order to get to that point, I needed one simple thing: have part of the Windows filesystem mounted in the same directory on both (a) the linux VM and (b) WSL. Luckily for me WSL already links to the Windows filesystem: the D: drive is automatically mounted as /mnt/d.

Next step was to get the linux VM to get access to the files in the D: drive, and have them under /mnt/d. This part is specific to VirtualBox, I’m quite certain this can be done using Hyper-V (and Docker for Windows), but I haven’t tried it.

VirtualBox configuration

VirtualBox configuration

First thing to do is open VirtualBox and find the VM that is used by docker. In order to get the name of the VM to modify, open a Windows command prompt and run docker- machine env. In the output, you will find a a value associated with DOCKER_MACHINE_NAME. You will now shutdown the VM in order to modify it. To do so, simple right-click on the VM and select Close->ACPI Shutdown and confirm that you want to shut down the VM. With the VM shutdown, right-click on it and select Settings.

In the settings page, go into the Shared Folders category. This page is where the magic happens. You will need to add a new shared folder. The folder path is the path on you Windows filesystem (D: for me). The name of the share is in fact the path on the linux VM where docker runs (/mnt/d for me).

With this configuration done, it is time to start the VM once again. To do so, select your VM from the main menu, right-click on it and select Start->Headless Start.

It might take a little bit of time before the VM is up-and-running, but it should all be working now.

Running screen in WSL

I recently started using WSL (Windows Subsystem for Linux) quite a bit and ended up needing multiple tabs opened. Right now, WSL does not support having multiple tabs natively, and I had to find an alternative. I decided to go with the good old screen but ran into some issues while running screen in Windows Subsystem for Linux.

Since screen is pre-installed in WSL (Ubuntu), I assumed that it would be working out of the box. But while trying to run it, I ended up with various errors:

The quick fix would be to run screen as root (with sudo) but that annoyed me, I really wanted to run it as my user.

After a bit of googling around, I managed to find a fix that works for me: change the directory screen uses to store its internal information. Luckily, screen allows an environment variable to control this directory: SCREENDIR. I have zero credit to take for this solution, I only googled around and found a nice little one already working 🙂

I decided to go with a simple sub-directory under my home and use that:

Remember to edit ~/.bashrc to add the export line to make this permanent!


Docker command line in WSL

Using containers and Docker is the norm for a lot of developers out there nowadays, but using it on Windows can be painful. Here is a little documentation on how I’ve been using Docker command line in WSL (Windows Subsystem for Linux).

The main problem I was having with using Docker on Windows is quite simple: Docker is basically a series of command line tools (docker, docker-compose, docker-machine, …) and the Windows command prompt is not quite nice to use.

For a while, I was using Docker from a git-bash shell. It is better than nothing, but then I needed more and more Linux tools and got quite annoyed. In order to make things easier, I decided to install Ubuntu directly on my Windows 10 machine through WSL. In order to get that installed, simply follow this tutorial.

To run Docker on Windows, you have two choices:

  1. Docker for Windows, uses Hyper-V to run a Linux VM for Docker;
  2. Docker Toolbox, uses VirtualBox to run the Linux VM for Docker

Both of these will get the job done. I had to go with Docker Toolbox for one simple reason: I don’t have a Enterprise, Professional, or Education version of Windows 10, and it is needed to get Hyper-V running.

Once you have Docker (for Windows or Toolbox) running, you should be able to use all the command line commands through the Windows command prompt or git-bash. In order to get everything running on Ubuntu you will need to do one last thing: configure your Linux environment.

The first thing you will need to get your environment up-and-running is the value of your DOCKER_HOST environment variable. To get this value, simply run echo %DOCKER_HOST%  in a Windows command prompt (or  echo $DOCKER_HOST  in a git-bash console). In my case, the value is tcp:// . This gives you the IP/Port where your Docker daemon is listening.

Armed with this value, you will need to edit your bash initialization script in your Ubuntu installation. In order to do so, simply add the following at the end of your ~/.bash_profile file:

The value associated with DOCKER_HOST is the one we got a bit earlier, and the value associated with DOCKER_CERT_PATH is a file inside your Windows user’s home directory.

You should now be able to install the docker package and use any command line tools contained in it!