Exploring ways to share Python packages across multiple SAM projects.
$ tree .
.
├── project1
│ ├── function
│ │ ├── app.py
│ │ └── requirements.txt
│ └── template.yaml
├── project2
│ ├── function
│ │ ├── app.py
│ │ └── requirements.txt
│ └── template.yaml
...
.
└── mylib
└── mylib.py
Considering packaging during deployment, one approach is to copy the shared package directory into each project. By specifying BuildMethod: makefile, the CustomMakeWorkflow copies the function code to ARTIFACTS_DIR during build and calls make build-{logicalId}, allowing this to be automated. Note that to reference the parent directory with a relative path, you need to pass the –build-in-source flag.
$ cat template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
TestFunction:
Type: AWS::Serverless::Function
Properties:
...
Metadata:
BuildMethod: makefile
$ cat function/Makefile
build-TestFunction:
pip install -r requirements.txt -t "$(ARTIFACTS_DIR)"
cp -r ../../mylib "$(ARTIFACTS_DIR)/"
This method has some drawbacks: the imported package cannot be found during development, and if the shared code has dependencies, they must be managed by the caller, making it fragile to changes. It’s preferable to create a pyproject.toml and use pip install. Note that the default PythonPipWorkflow doesn’t support –build-in-source, so BuildMethod: makefile is still necessary.
$ cat mylib/pyproject.toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "mylib"
version = "0.1.0"
dependencies = [
"numpy>=1.24.0,<2.0.0",
]
Unfortunately, installing via relative paths doesn’t work with sam build –use-container since the parent directory isn’t mounted to the container. Another approach is to pip install from a Git repository. Using pip install git+(repository) clones the repository and installs the library from the directory specified with #subdirectory=.
$ cd example-app
$ cat requirements.txt
git+https://github.com/sambaiz/pip-install-from-git-test.git@main#subdirectory=stats-lib
git+https://github.com/sambaiz/pip-install-from-git-test.git@main#subdirectory=text-lib
$ pip install -r requirements.txt
Collecting git+https://github.com/sambaiz/pip-install-from-git-test.git@main#subdirectory=stats-lib (from -r requirements.txt (line 1))
Cloning https://github.com/sambaiz/pip-install-from-git-test.git (to revision main) to /tmp/pip-req-build-rss0j5g6
Running command git clone --filter=blob:none --quiet https://github.com/sambaiz/pip-install-from-git-test.git /tmp/pip-req-build-rss0j5g6
Resolved https://github.com/sambaiz/pip-install-from-git-test.git to commit 605fa82f41ecf763308190fb6b0ae5113c05d893
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting git+https://github.com/sambaiz/pip-install-from-git-test.git@main#subdirectory=text-lib (from -r requirements.txt (line 2))
Cloning https://github.com/sambaiz/pip-install-from-git-test.git (to revision main) to /tmp/pip-req-build-ivwe3ami
Running command git clone --filter=blob:none --quiet https://github.com/sambaiz/pip-install-from-git-test.git /tmp/pip-req-build-ivwe3ami
Resolved https://github.com/sambaiz/pip-install-from-git-test.git to commit 605fa82f41ecf763308190fb6b0ae5113c05d893
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.24.0 in ./venv/lib/python3.14/site-packages (from stats-lib==0.1.0->-r requirements.txt (line 1)) (2.4.0)
Even for the same repository, cloning is required, and when using this in CI/CD, you need to grant appropriate permissions, but this approach still works even if moved to a different repository. If you plan to do so, using an external repository like CodeArtifact is also an option.
Alternatively, you could use a Lambda Layer. sam local invoke works as-is. However, since you still need to pip install during development and testing, it may not be much different from other approaches. While using relative paths works, it makes it difficult to fix the version of shared code. Ideally, you’d be better to specify release versions rather than relative paths. From there, whether to deploy each project with the shared code included or share a Lambda Layer is a choice that doesn’t matter much unless the size becomes large.
Lambda Layerでバイナリやライブラリを切り出す - sambaiz-net
Resources:
MyLibLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: mylib-layer
ContentUri: ../mylib/
CompatibleRuntimes:
- python3.11
Metadata:
BuildMethod: makefile
TestFunction:
Type: AWS::Serverless::Function
Properties:
...
Layers:
- !Ref MyLibLayer
Metadata:
BuildMethod: makefile