Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for uv to package python lambda function #654

Open
mazzma12 opened this issue Jan 8, 2025 · 4 comments
Open

Add support for uv to package python lambda function #654

mazzma12 opened this issue Jan 8, 2025 · 4 comments

Comments

@mazzma12
Copy link

mazzma12 commented Jan 8, 2025

Is your request related to a new offering from AWS?

No 🛑 , this is related to using an alternative Python package installer (UV) that would improve the efficiency and reliability of Lambda deployments.

Is your request related to a problem? Please describe.

The current process of installing Python dependencies for AWS Lambda functions using pip can be slow and sometimes unreliable. Common issues include:

  • Pip is notoriously not handling safely packages resolution through multiple indices, exposing the user to potential "dependency confusion" attack (see https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes )
  • Slow installation times during deployments, especially with large dependency trees
  • Inconsistent package resolution across different environments
  • High memory usage during package installation
  • Longer cold starts due to suboptimal dependency handling

Describe the solution you'd like.

Add support for using UV (https://github.com/astral-sh/uv) as an alternative Python package installer in the Lambda module. UV is a new, extremely fast package installer written in Rust that offers:

  • Up to 10-30x faster package installation compared to pip
  • Reliable dependency resolution
  • Lower memory usage
  • Deterministic builds
  • Full pip compatibility

The module could support UV through:

  1. A new option in the module configuration to specify UV as the package installer - similar to poetry
  2. Integration with UV's lockfile support for reproducible builds
  3. Documentation on how to use UV with the module

Describe alternatives you've considered.

  1. Using custom Docker builds to pre-install dependencies with UV
  2. Creating a separate Lambda layer with UV-installed dependencies

However, native module support would provide a much better developer experience and ensure consistent behavior across teams.

Additional context

UV has gained significant adoption in the Python community due to its performance benefits. Companies like Datadog have reported significant improvements in their Python dependency management after switching to UV. Supporting UV in the Terraform AWS Lambda module would align with the community's move toward more efficient Python tooling and improve the developer experience for teams building Lambda functions.

@pavitra-infocusp
Copy link

pavitra-infocusp commented Jan 20, 2025

On the similar spirit, I would go further and suggest using esbuild for Nodejs code.

@antonbabenko
Copy link
Member

Instead of providing first-class support for tools like UV, esbuild, and whatever comes next week, I would prefer us to have a standard way to extend this module with some kind of plugin mechanism where usage of such tool is more externalized than it is now (single python script does it all).

Ideally, I think, package.py should be a native Terraform resource within a Terraform provider, but this is harder to implement now.

@pavi2410
Copy link

pavi2410 commented Jan 21, 2025

Definitely agree on the induced module complexity with different runtimes and packaging methods. I have used with this module in the past to deploy multiple Nodejs and Python lambda functions. Configuring the source_paths is difficult in the current state. Even when you finally are able to configure it, you see awful logs from the package.py script (I think it's useful but Terraform log prefixes makes it awful) and in the Terraform plan logs, the execution plan is so confusing with the null archive, build plans and all (Again, I think these are useful but confusing). It's a "No size fits all" kind of problem. Not to mention that Python has to be installed even for Nodejs projects, to run the package.py script.

I have a solution to this - let's create new Terraform modules responsible for creating Runtime-specific deployment zips. The Terraform AWS Lambda module would just be a consumer of the zip file. This approach is similar to setting create_package = false and providing the zip files directly. This would greatly simplify this module interface and implementation. This will allow us break down the Lambda deployment pipeline into logical steps, ie module "DeploymentZipForNodejs" -> module "Lambda".

@pavi2410
Copy link

Please see this Nodejs typescript monorepo example I have created for a project.

https://github.com/pavi2410/aws-lambda-nodejs-typescript-monorepo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants