Lambda refactoring/CRUD ASF provider implementation by dominikschubert · Pull Request #6865 · localstack/localstack · GitHub
Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

Lambda refactoring/CRUD ASF provider implementation#6865

Merged
dominikschubert merged 96 commits into
masterfrom
lambda-crud-newprovider
Oct 7, 2022
Merged

Lambda refactoring/CRUD ASF provider implementation#6865
dominikschubert merged 96 commits into
masterfrom
lambda-crud-newprovider

Conversation

@dominikschubert

@dominikschubert dominikschubert commented Sep 13, 2022

Copy link
Copy Markdown
Member

Goal

Continued implementation for new ASF provider for lambda. The goal of this PR is to make the integration tests in tests.integration.awslambda/test_lambda_api.py|test_lambda.py|test_lambda_common.py pass for the new provider on CircleCI since this was disabled in the preceding test rework.

Changes

  • Implement CRUD of most (except those in Future Work) endpoints
  • Tests of both API and error handling
  • Alias routing with routing config
  • Migration to Stores
  • Proper API error handling
  • Lambda Role Injection
  • Event invocations
  • Zip file storage in S3 buckets / presigned urls for code location

Future Work

  • Event source mappings (currently only barebone crud)
  • Invoke async (deprecated anyway, but should have it)
  • Layer / LayerVersion / LayerPermission CRUD
  • LambdaUrls
  • Event destinations / event retries
  • Timeouts
  • Dry Run
  • Init duration in logs
  • More correct error responses (the todos)

@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 14, 2022 10:56 Inactive
@github-actions

github-actions Bot commented Sep 14, 2022

Copy link
Copy Markdown

@dfangl dfangl temporarily deployed to localstack-ext-tests September 14, 2022 13:19 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 14, 2022 15:02 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 14, 2022 17:23 Inactive
@dominikschubert dominikschubert added the aws:lambda AWS Lambda label Sep 16, 2022
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 22, 2022 11:07 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 22, 2022 12:46 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 22, 2022 12:53 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 22, 2022 13:35 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 22, 2022 17:36 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 22, 2022 18:23 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 23, 2022 07:22 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 23, 2022 07:44 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 23, 2022 08:44 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 23, 2022 11:20 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 26, 2022 19:41 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 26, 2022 19:56 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 27, 2022 12:22 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 27, 2022 12:51 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 27, 2022 12:57 Inactive
@dfangl dfangl temporarily deployed to localstack-ext-tests September 27, 2022 13:02 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 27, 2022 13:32 Inactive
Comment on lines +38 to +46
def __init__(self, *args, **kwargs):
super(ServiceException, self).__init__(*args)

if len(args) >= 1:
self.message = args[0]
else:
self.message = ""
for key, value in kwargs.items():
setattr(self, key, value)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i approve this change :-) coincidentally had this thought here: #6875 (comment)

@bentsku will also make your life easier

@dfangl dfangl temporarily deployed to localstack-ext-tests September 27, 2022 18:00 Inactive
@dominikschubert dominikschubert temporarily deployed to localstack-ext-tests September 28, 2022 07:00 Inactive
@dfangl dfangl force-pushed the lambda-crud-newprovider branch from a01e2ea to 78771e3 Compare October 5, 2022 15:35
@dfangl dfangl marked this pull request as ready for review October 5, 2022 15:36
@dfangl dfangl self-requested a review as a code owner October 5, 2022 15:36

@whummer whummer left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wohooo, fantastic set of changes, 🚀 🚀 extremely excited to see this evolving! Like the approach with frozen dataclasses - immutable state will avoid lots of headaches we've had the past. 👌 Really clean data model and encapsulation of logic for the different aspects of the API, great job.

Also, kudos for the huge depth of testing, insane number of new snapshot tests, which really give us high confidence of parity. 🥳

Added a few minor suggestions and a few questions - mostly nitpicks, nothing major. Looking forward to making this the default soon.. 🙂

Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/api_utils.py
Comment thread localstack/services/awslambda/api_utils.py Outdated
class VersionFunctionConfiguration:
# fields
# name: str
function_arn: str # TODO:?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, what's the required change (TODO) here?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dominikschubert do you remember? I think we wanted to remove it, as the ID including the arn is saved in the FunctionVersion objects?

Comment thread localstack/services/awslambda/invocation/lambda_service.py
Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/invocation/models.py

@thrau thrau left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a pretty massive PR, congrats on these changes, the impeccable testing and the high-quality code. it's easy to read and understand. this PR is i think also a good demonstration of how verbosity can be good thing, and that code golf is not the way we should develop providers.

💯 🏅

my comments are mostly cosmetic changes and one related to IO that we can probably fix later.

Comment thread localstack/services/awslambda/api_utils.py Outdated
Comment thread localstack/services/awslambda/api_utils.py Outdated
Comment thread localstack/services/awslambda/invocation/lambda_models.py
Comment thread localstack/services/awslambda/invocation/runtime_executor.py
Comment on lines +93 to +96
s3_client: "S3Client" = aws_stack.connect_to_service("s3", region_name="us-east-1")
kwargs = {"VersionId": self.s3_object_version} if self.s3_object_version else {}
try:
s3_client.delete_object(Bucket=self.s3_bucket, Key=self.s3_key, **kwargs)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we forcing us-east-1 when creating s3 code objects as well? otherwise delete_object may raise an exception, right?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, all our internal bucket storage is currently in us-east-1. As s3 is global, does not make too many differences, but for the sake of locationconstraints etc, its easier. At some point we want to hide the s3 bucket (its currently enumerable by the user), for example by defining a "LocalStack account". That would also enable us to use s3 storage for other services (e.g. ECR), making persistence and file storage easier.

Comment on lines -47 to +124

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curios: what in your opinion is better in using the list builtin compared to the List from typing?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dominikschubert you have a stronger opinion there. In my opinion its just easier, as you do not need an import.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'm generally pro list/dict etc. I see no reason to use List over list when we don't have to support earlier python versions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my argument for List and Dict would be that it's more consistent when using other imports from typing, like Optional, Type, Generic, etc...

Comment thread localstack/services/awslambda/api_utils.py Outdated
Comment thread localstack/services/awslambda/provider.py
Comment thread localstack/services/awslambda/provider.py Outdated
Comment thread localstack/services/awslambda/provider.py
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

aws:lambda AWS Lambda

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants