References:

• https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#runtimes-lifecycle-shutdown
• https://github.com/aws-samples/graceful-shutdown-with-aws-lambda

When running cargo watch, currently cargo lambda sends SHUTDOWN events to extensions on hot reload when code recompiles. It doesn't seem send any signals through to the function runtime though.

In AWS, if extensions are registered (either internal or external), this will also result in SIGTERM being sent to the function, along with SIGKILL if the runtime doesn't shut down within the relevant timeout.

It would be great to send similar signals in the code paths that are currently sending SHUTDOWN events.

I don't think cargo lambda needs to perfectly match the logic in AWS Lambda orchestration around signal handling (timeouts before sigkill, only sending sigterm if extensions are registered, etc). That seems like a moving target.

Even just unconditionally sending a SIGTERM as part of the shutdown lifecycle would be pretty handy. We could document the lack of guarantee for exact parity.

Context

I'm currently prepping a "graceful shutdown" example for aws-lambda-rust-runtime as well as aws-samples/graceful-shutdown-with-aws-lambda leveraging an internal extension. The existing example assumes an external extension, cloudwatch lambda insights.

Testing in AWS is a bit fiddly because Lambda doesn't offer any guarantees around when a function will actually be spun down. In practice, during off-peak hours, it can be quite a while.

Testing signal handling related function code via cargo lambda watch would be way more convenient.

Current behavior

This doesn't seem to be supported currently. On hot reload, which seems to be the main way to trigger the SHUTDOWN flow, I see:

DEBUG registering extension payload=EventsRequest { events: ["INVOKE"] }
DEBUG extension waiting for next event extension_id=f5e76a8e-9644-4bd5-bc4e-7e3073f844ec
DEBUG watcher action received action=Action { events: [Event { tags: [Source(Internal), ProcessCompletion(Some(ExitSignal(ForceStop)))], metadata: {} }], outcome: OnceCell(Uninit) } signals=[] has_paths=false empty_event=false
ERROR command killed code=ForceStop

I've tried manually sending a signal to the process via kill on the process in my terminal, but it doesn't seem to propagate signals through to the lambda runtime properly. I haven't looked too closely, but it looks like we have some outside signal handling that needs to pass it through.