How to Create AWS Lambda Layers for Python Dependencies

Using AWS Lambda for your serverless Python app can give significant cost and management benefits. However, unless your needs are simple you will quickly need to bundle your dependencies along with your functions. Doing so with an AWS Lambda Layer works much like installing dependencies within a local environment. You are able to reduce each functions size, maintain your core code separately and simply tell the function to use the layer when setting it up.

Layers are bundled as a zip archive, but you’ll be running a different architecture on a dev machine than the Lambda does itself which could present problems for certain packages. Therefore, you need to create your lambda layer within docker.

After trying a few different approaches, I found the following script by haranjackson the best solution. It uses a docker image from lambci and the AWS CLI to build and publish a Lambda Layer to AWS. I’ve modified this below, and created a gist which will install from a requirements.txt file within in the same folder.

REGION=eu-west-2
RUNTIME=python3.8
LAYER_NAME=app

docker run -v $(pwd):/out -it lambci/lambda:build-$RUNTIME \
    pip install -r /out/requirements.txt -t /out/build/$LAYER_NAME/python

cd build/$LAYER_NAME
zip -r ../../$LAYER_NAME.zip python/
cd ../..

aws lambda publish-layer-version \
    --layer-name $LAYER_NAME \
    --region $REGION \
    --zip-file fileb://$LAYER_NAME.zip \
    --compatible-runtimes $RUNTIME

rm -rf build *.zip

You’ll then be able to see the Layer under “Additional Resources” from the AWS Lambda console and make use of it from any of your Lambda functions.

Alternative approaches

This works great for runtimes up to Python 3.8, but later ones are unavailable from the lambci dockerhub images. Amazon have their own python lambda images, though I’ve not had success using them to build layers using the approach above.

For other projects, I’d normally tackle bundling dependencies using something like the serverless.com framework though I’ve grown a bit tired of Python playing a second class citizen to JS - debugging errors quickly becomes difficult.

Another alternative approach could be to use the Serverless Application Model (SAM) by Amazon themselves. This is definitely something I want to check out. For now, I decided I wanted to understand what was going on and keep it simple rather than jumping through multiple frameworks.