Blog
November 30, 2018 Marie H.

AWS Lambda Layers: Finally, Shared Dependencies Without the Pain

AWS Lambda Layers: Finally, Shared Dependencies Without the Pain

Photo by <a href="https://unsplash.com/@growtika?utm_source=cloudista&utm_medium=referral" target="_blank" rel="noopener">Growtika</a> on <a href="https://unsplash.com/?utm_source=cloudista&utm_medium=referral" target="_blank" rel="noopener">Unsplash</a>

AWS Lambda Layers: Finally, Shared Dependencies Without the Pain

Lambda Layers was announced at re:Invent two days ago and I've spent the last couple evenings getting my hands dirty with it. This is one of those features I didn't know I needed until I saw it and immediately thought of six places I would have used it this year. Let me walk you through how it works.

The Problem It Solves

If you have more than a handful of Lambda functions, you've probably hit this workflow: make a change to a shared utility module, rebuild the dependency ZIP for every function that uses it, redeploy all of them. It's tedious, it's slow, and every function is carrying its own copy of requests, boto3, pandas, or whatever you're using.

My worst offender was a project with about 15 functions that all used the same internal library plus requests and cryptography. Every deploy was a multi-MB upload for each function, and if I updated the shared library I was touching 15 deployment packages. The obvious solution is LET'S AUTOMATE IT, but the real solution is Lambda Layers.

A Layer is a ZIP archive that gets mounted into your function's execution environment at /opt. You define it once, version it, and attach it to as many functions as you want. The functions don't bundle that code — they just reference the layer. Update the layer, bump the version, update your function's layer reference, done.

Layer ZIP Structure

This is the part that trips people up. The directory structure inside the ZIP has to follow a convention that matches the runtime's module search path.

For Python, your dependencies need to live at:

python/lib/python3.6/site-packages/

So to create a layer for the requests library:

mkdir -p layer/python/lib/python3.6/site-packages
pip install requests -t layer/python/lib/python3.6/site-packages/
cd layer
zip -r ../requests-layer.zip .
cd ..

Let's check what we've got:

$ ls -lh requests-layer.zip
-rw-r--r--  1 marie  staff   412K Nov 30 09:14 requests-layer.zip

$ unzip -l requests-layer.zip | head -20
Archive:  requests-layer.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  11-30-2018 09:14   python/
        0  11-30-2018 09:14   python/lib/
        0  11-30-2018 09:14   python/lib/python3.6/
        0  11-30-2018 09:14   python/lib/python3.6/site-packages/
     9906  11-30-2018 09:12   python/lib/python3.6/site-packages/requests/__init__.py
     ...

Publishing the Layer

aws lambda publish-layer-version \
  --layer-name requests-layer \
  --description "requests 2.20.0 for Python 3.6" \
  --zip-file fileb://requests-layer.zip \
  --compatible-runtimes python3.6

Output:

{
    "LayerArn": "arn:aws:lambda:us-east-1:123456789012:layer:requests-layer",
    "LayerVersionArn": "arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1",
    "Description": "requests 2.20.0 for Python 3.6",
    "CreatedDate": "2018-11-30T14:22:03.071+0000",
    "Version": 1,
    "CompatibleRuntimes": ["python3.6"]
}

Note the version number at the end of the ARN — :1. Every publish-layer-version call creates an immutable new version. Layer versions are forever; you can't modify one in place. This is actually good behavior — it means attaching a specific layer version to a function is reproducible and won't silently change under you.

Attaching the Layer to a Function

aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1

You can attach multiple layers (up to 5 — more on that limit shortly):

aws lambda update-function-configuration \
  --function-name my-function \
  --layers \
    arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1 \
    arn:aws:lambda:us-east-1:123456789012:layer:my-utils:3

Using It in Your Function

Your function code doesn't need to change at all. Python finds packages in /opt/python/lib/python3.6/site-packages automatically because Lambda adds /opt/python to sys.path. So this just works:

import requests  # from the layer, not bundled in the deployment package

def handler(event, context):
    response = requests.get("https://api.example.com/data")
    return {
        "statusCode": 200,
        "body": response.json()
    }

Your function's deployment package is now just the handler code — no dependencies. My 15-function project went from multi-MB deployment packages down to a few KB each. Deploys are noticeably faster.

Version Management in Practice

When you update a dependency — say requests 2.20.1 comes out with a security fix — the workflow is:

  1. Rebuild the layer ZIP with the new version
  2. publish-layer-version → get back version 2 of the layer
  3. Update your functions to reference :2

Step 3 is still something you want to automate. I use a small script that lists all functions using the old layer ARN and updates them to the new one:

#!/bin/bash
OLD_LAYER="arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:1"
NEW_LAYER="arn:aws:lambda:us-east-1:123456789012:layer:requests-layer:2"

aws lambda list-functions --query 'Functions[*].FunctionName' --output text | tr '\t' '\n' | while read fn; do
  layers=$(aws lambda get-function-configuration --function-name "$fn" \
    --query 'Layers[*].Arn' --output text 2>/dev/null)

  if echo "$layers" | grep -q "$OLD_LAYER"; then
    echo "Updating $fn..."
    new_layers=$(echo "$layers" | tr '\t' '\n' | sed "s|$OLD_LAYER|$NEW_LAYER|" | tr '\n' ' ')
    aws lambda update-function-configuration \
      --function-name "$fn" \
      --layers $new_layers
  fi
done

The 5-Layer Limit

Each function can have at most 5 layers attached. In practice this is rarely a problem, but it does mean you want to think about how you organize layers. I group related dependencies into a single layer rather than one layer per package. "Data processing" layer (pandas, numpy, scipy), "HTTP clients" layer (requests, urllib3), "internal utils" layer. Keeps you well within the limit and makes the groupings meaningful.

Total unzipped size across all layers plus your function code can't exceed 250MB. Also rarely an issue unless you're doing something like bundling a full ML model, in which case you have bigger architectural questions to answer.

Honestly

This is one of the better QoL improvements to Lambda in a while. It doesn't solve every Lambda pain point — cold starts are still a thing, the 15-minute timeout is still the 15-minute timeout — but the dependency management story is genuinely better now. If you're running more than a few functions with shared code, layers should be part of your workflow.