AWS Lambda Layers - The Complete Guide

In this step-by-step tutorial, you'll learn about why we need Lambda layer and how to create one. And, you'll also learn about how to use a lambda layer inside a lambda function.

Before discussing about lambda layers - let us discuss about the problem it solves.

Why do we need AWS Lambda Layers?

Scenario 1

You have some business functionality which is used across many lambda functions. You don't want to duplicate this business functionality across many lambda functions as this would be tedious to change in many places - if the business functionality undergoes some logic change. And, you want to follow DRY and Single responsibility principles.

Scenario 2

Your company allows you to use only the dependencies that have been approved by the security team. We agree with them - as there are several attacks that happened previously from malicious packages (Hello malicious npm packages!)

One solution is to create your lambda project with the dependencies that you require and get it approved by your security team.

But the problem with this approach is that this could result in huge delays in development of actual application as you need to talk back and forth for each lambda project.

Lambda layers to the rescue.

What is Lambda Layer?

Lambda Layer is just a zip file containing either code or data. You can easily share functionalities or data across lambda functions using Lambda Layers.

How Lambda Layer works

Lambda service uses Linux for running your lambda functions.

When you use a lambda layer in your lambda function code - the code/data in the lambda layer will be extracted to your /opt directory whereas the actual lambda function code will be downloaded to /var/task directory.

Below is the high level diagram

Characteristics of Lambda Layers

Below are the characteristics of the lambda layer.

Layers are immutable: Once you upload a lambda layer, you'll not be able to change it.

Layers can be versioned: Even though layers are immutable, you can create a new version of the layer.

Layer version can be deleted: You can delete lambda layer version and when you do it - you'll not be able to create new lambda function using that layer. However, any existing lambda function which uses that version would still work.

Lambda functions can have up to 5 layers: Your lambda functions can refer up to 5 lambda layers

Layers count towards deployment package: The maximum size of Lambda deployment package is 250 MB (unzipped) and 50 MB(zipped). If you use lambda layers, that also might count towards the deployment package size

Order of Layers is important: As each lambda is extracted to the same /opt, overwriting the existing layer is very much possible. To avoid this, you need to use different paths within /opt . We'll see an example later in this article

Creating your first Lambda Layer

In this article, we're going to use AWS CDK - an Infrastructure as Code (IaC) tool - developed and maintained by AWS. If you're new to AWS CDK - I strongly suggest you to read this guide to understand AWS CDK. No problem, I can wait.

Create a new AWS CDK project

You can create a new aws-cdk project by executing following commands. Please make sure you've installed aws-cdk npm package globally before executing these commands.

mkdir aws-lamabda-layers
cd aws-lambda-layers
cdk init app --language=typescript

Once the cdk project is created, you can create a folder by name src. This folder will contain all the lambda functions and the code for the lambda layers.

Creating your first lambda layer:

We want to re-use business logic across different lambda functions. So, let's come up with innovative business functionality - addition of 2 numbers and returning the result :-)

export function add(a: number, b: number) {
  return a + b;
}

We're creating this method in logic.ts and will be in folder layers/business-logic.

Creating lambda layers using aws-cdk is pretty simple. You just need to mention the compatible runtimes and the code as shown below. Optionally, you can specify the layerVersionName and description.

    const logicLayer = new lambda.LayerVersion(this, 'logic-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_14_X,
        lambda.Runtime.NODEJS_16_X,
      ],
      layerVersionName: 'business-logic-layer',
      code: lambda.Code.fromAsset('src/layers/business-logic'),
      description: 'Business logic layer',
    });

When you deploy the stack using cdk deploy command, lambda layer will be created. You can see the same in AWS console.

Using Lambda Layer in Lambda function

Now, we've lambda layer available and we can use this lambda layer in lambda function.

Creating Lambda function which uses Lambda Layer

As mentioned earlier, lambda layer would be extracted to /opt directory. So, when your lambda function is invoked, your lambda function code will have to refer to the lambda layer code which would be in /opt directory.

Below is a simple lambda function which uses lambda layer that we created earlier.

import * as logic from '/opt/business-logic';

export const handler = async (event: any = {}): Promise<any> => {
  console.log(`Addition:${logic.add(2, 3)}`);
};

In VS Code, there will be red squiggly lines in /opt/business-logic path as there is no such path in my local system. As we're using typescript, we can use paths property in tsconfig.json to re-map the imports - as shown below.

"baseUrl": "./",
"paths": {
   "/opt/business-logic": ["src/layers/business-logic/logic"],
}

Now, red squiggly lines will be removed as typescript will map the /opt folder to src/layers folder locally. At runtime- when our lambda function is invoked, it will be referring to the /opt directory in linux environment.

You can create Lambda resource by using the following cdk code. We're configuring the runtime, timeout and memory size properties initially and creating lambda function

   const nodeJsFnProps: NodejsFunctionProps = {
      bundling: {
        externalModules: [
          'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
        ],
      },
      runtime: Runtime.NODEJS_16_X,
      timeout: Duration.minutes(3),
      memorySize: 256,
    };

    const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
      entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
      ...nodeJsFnProps,
      functionName: 'lambdaWithLayer',
      handler: 'handler',
      layers: [logicLayer],
    });

You can deploy the updated stack by using cdk deploy command and lambda function will be created and you can see the same in aws console.

Testing the lambda function:

In AWS console, you can test the lambda function and click Test button

Checking the lambda layer and lambda function in real time in AWS service

We can update the lambda function to print the contents of lambda layer folder( /opt ) and lambda function folder ( /var/task )

I wrote a simple function to execute shell command in lambda and I've updated the lambda function code to list the contents of these folders - as shown below.

export const handler = async (event: any = {}): Promise<any> => {
  console.log(`Addition:${logic.add(2, 3)}`);
  const commands = ['ls -R /var/task', 'ls -R /opt'];
  for (const cmd of commands) {
    try {
      const res = await execShellCommand(cmd);
      console.log(`Result of ${cmd}:`, res);
    } catch (err) {
      console.log(`error executing command - ${cmd}:`, err);
    }
  }
};

function execShellCommand(cmd: any) {
  return new Promise((resolve, reject) => {
    exec(cmd, (error: any, stdout: any, stderr: any) => {
      if (error) {
        console.warn(error);
      }
      resolve(stdout ? stdout : stderr);
    });
  });
}

When you deploy and invoke the lambda function, you'll get the following output

As expected, the business logic file is in /opt folder and our lambda function is in var/task folder

Creating lambda layer with npm or yarn packages as dependencies

Earlier, we've created lambd layer without any external npm packages. Now,we want to create re-usable library of npm packages - approved by security team. We want to create another lambda layer for this.

Create a new folder by name utils\nodejs  in layers folder as shown in the below picture. And, initialize a new project by executing npm init  here. The utils folder would be converted to lambda layer later.

Install npm package ulid and we want this package to be re-used across many different functions.

Creating utils Lambda layer

We're going to create a new lambda layer utils (with only ulid package available in it).

 const utilsLayer = new lambda.LayerVersion(this, 'utils-layer', {
      compatibleRuntimes: [
        lambda.Runtime.NODEJS_14_X,
        lambda.Runtime.NODEJS_16_X,
      ],
      code: lambda.Code.fromAsset('src/layers/utils'),
      description: 'Utils  layer',
    });

Updated lambda function code

For local development, we would be installing ulid package in our root folder too so that our lambda can access this package. But, we'll excluding this package from lambda while bundling.

Updated lambda function code is pretty simple. We're calling the ulid to generate unique lexicographically-sortable identifier.

import * as logic from '/opt/business-logic';
import { ulid } from 'ulid';

export const handler = async (event: any = {}): Promise<any> => {
  console.log(`Addition:${logic.add(2, 3)}`);
  console.log('Unique lex sortable id:', ulid());
};

Updated lambda function

There are 2 changes required for using the utils lambda layer in your lambda function

• Removal of ulid package when bundling the lambda as it is available in utils layer
• Pass utils lambda layer to the lambda function in addition to existing logic lambda layer

Below is the updated lambda function

 const nodeJsFnProps: NodejsFunctionProps = {
      bundling: {
        externalModules: [
          'aws-sdk', // Use the 'aws-sdk' available in the Lambda runtime
          'ulid',
        ],
      },
      runtime: Runtime.NODEJS_16_X,
      timeout: Duration.minutes(3),
      memorySize: 256,
    };

    const lambdaWithLayer = new NodejsFunction(this, 'lambdaWithLayer', {
      entry: path.join(__dirname, '../src/lambdas', 'lambda.ts'),
      ...nodeJsFnProps,
      functionName: 'lambdaWithLayer',
      handler: 'handler',
      layers: [logicLayer, utilsLayer],
    });

Please note that we've mentioned the npm package ulid in externalModules property as we don't want this package to be bundled.

And, in layers property of lambda function, we've added utilsLayer too.

Testing the lambda function

You can test the lambda function in AWS console and you would be able to see the generated id.

How we know the lambda function is referring to the ulid in lambda layer and not the local npm package in root folder?

You can easily check this. Just remove the utils lambda layer from layers property in lambda function properties and deploy the stack using cdk deploy

Test the lambda function and it'll throw Runtime.ImportModuleError  as shown below.

As we've excluded the ulid package while bundling the lambda function - the package would not be available as part of lambda. Earlier, we were referring to the this package from lambda layer. As we've removed it now, we're getting error.

Referring lambda layers code in lambda function

There are 2 ways to refer to the code/package in the lambda layer from the lambda function.

• You can use the /opt/.. path directly as we've used for referring the business logic layer in our lambda function
• You can use the npm package with no mention of custom path as we've used for ulid package. You can use the package as though it is installed locally for the lambda function.
  
  We've used both approaches in our code.

import * as logic from '/opt/business-logic';
import { ulid } from 'ulid';

You could rewrite the last line to something like below but it is not needed.

import { ulid } from '/opt/nodejs/node_modules/ulid';

When you don't specify the custom path, Lambda would check whether this code is associated with local lambda function and then, it will check the default library dependencies for the lambda layer ( /opt/nodejs/node_modules )

As mentioned earlier, all of your lambda layer code will be extracted to /opt folder. In your lambda layers folder structure, if your node_modules folder is inside nodejs folder - it will be extracted to /opt/nodejs/node_modules and lambda function can refer to this package without mentioning any custom path.

This is the reason why we've created additional nodejs folder inside utils folder in lambda layer so that the ulid package can be deployed to /opt/nodejs/node_modules/ folder.

Conclusion

You can use lambda layers when you're following Domain Driven Design so that all of your core business logic would be independent of any implementation or interface details. You can use this logic/packages in lambda or in fargate or any other service.

Hope you've learnt a bit about lambda layers.

Please let me know your thoughts in the comments section. If you like this article, you can subscribe to newsletter.