Add AWS Lambda worker support by eamsden · Pull Request #608 · temporalio/sdk-ruby · GitHub
Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
74 changes: 74 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ Also see:
- [ActiveModel](#activemodel)
- [Converter Hints](#converter-hints)
- [Workers](#workers)
- [AWS Lambda Workers](#aws-lambda-workers)
- [ADOT tracing and metrics](#adot-tracing-and-metrics)
- [Workflows](#workflows)
- [Workflow Definition](#workflow-definition)
- [Running Workflows](#running-workflows)
Expand Down Expand Up @@ -395,6 +397,78 @@ Notes about the above code:
* Workers can have many more options not shown here (e.g. tuners and interceptors).
* The `Temporalio::Worker.run_all` class method is available for running multiple workers concurrently.

#### AWS Lambda Workers

`Temporalio::Contrib::Aws::LambdaWorker` defines a Lambda handler that creates a fresh Temporal client and worker for
each invocation. Set `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, and `TEMPORAL_TASK_QUEUE` in the Lambda environment (or
provide a `temporal.toml` configuration file). The handler resolves configuration during cold start, checking
`TEMPORAL_CONFIG_FILE`, then `$LAMBDA_TASK_ROOT/temporal.toml`, then `./temporal.toml`, before falling back to
environment-only configuration.

```ruby
# lambda_function.rb
require 'temporalio/contrib/aws/lambda_worker'
require_relative 'my_activity'
require_relative 'my_workflow'

VERSION = Temporalio::WorkerDeploymentVersion.new(
deployment_name: 'orders',
build_id: ENV.fetch('TEMPORAL_WORKER_BUILD_ID')
)

OPTIONS = Temporalio::Contrib::Aws::LambdaWorker::Options.new(
workflows: [MyWorkflow],
activities: [MyActivity]
)

TEMPORAL_LAMBDA_WORKER = Temporalio::Contrib::Aws::LambdaWorker.define(VERSION, options: OPTIONS)

def lambda_handler(event:, context:)
TEMPORAL_LAMBDA_WORKER.call(event, context)
end
```

The Lambda worker applies Lambda-oriented polling, concurrency, caching, and graceful-shutdown defaults. It reserves
seven seconds for shutdown by default, uses the Lambda request ID and function ARN as its Temporal identity, and always
enables Worker Versioning with `PINNED` as the default behavior. Set `default_versioning_behavior:` to
`Temporalio::VersioningBehavior::AUTO_UPGRADE` to change that default, or use `UNSPECIFIED` to require an explicit
behavior on every workflow. Use immutable `Options#with` to derive a configuration with client, worker, hook, or plugin
overrides. Eager activities are always disabled.

Increase `shutdown_buffer:` together with `worker_options: { graceful_shutdown_period: ... }` for longer activity
drains or telemetry hooks. Activities must cooperate with cancellation for the worker to finish before Lambda's hard
deadline. Shutdown hooks run in order even after setup or execution fails, and a hook failure does not mask the worker
error or prevent later hooks from running.

See the [deployable Ruby Lambda sample](temporalio/extra/aws_lambda/README.md) for native gem packaging, a workflow
starter, and ADOT collector configuration.

WARNING: Plugins are experimental.

##### ADOT tracing and metrics

The optional `LambdaWorker::OpenTelemetry::Plugin` adds the existing Temporal tracing interceptor, exports Core metrics
to ADOT over OTLP, and flushes the configured tracer provider after each invocation. Configure the application's
OpenTelemetry SDK and exporter before creating the plugin; the Temporal gem does not install or configure them.

```ruby
require 'opentelemetry/sdk'
require 'temporalio/contrib/aws/lambda_worker'

# Configure the application's tracer provider and ADOT exporter before this point.
adot_plugin = Temporalio::Contrib::Aws::LambdaWorker::OpenTelemetry::Plugin.new

TEMPORAL_LAMBDA_WORKER = Temporalio::Contrib::Aws::LambdaWorker.define(
VERSION,
options: OPTIONS.with(plugins: [adot_plugin])
)
```

The plugin uses `OTEL_EXPORTER_OTLP_ENDPOINT`, or `http://localhost:4317` when it is unset. Its metric service name
comes from `OTEL_SERVICE_NAME`, then `AWS_LAMBDA_FUNCTION_NAME`, then `temporal-lambda-worker`. Ruby's OTLP trace exporter
uses HTTP (typically port 4318), while this plugin's Core metrics use gRPC (typically port 4317); configure their
endpoints separately. `metric_periodicity:` defaults to ten seconds and should be shorter than the invocation budget.

### Workflows

#### Workflow Definition
Expand Down
38 changes: 27 additions & 11 deletions temporalio/ext/src/client.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
use std::{collections::HashMap, future::Future, marker::PhantomData, time::Duration};
use std::{
cell::RefCell, collections::HashMap, future::Future, marker::PhantomData, time::Duration,
};

use temporalio_client::{
ClientKeepAliveOptions, ClientTlsOptions, Connection, ConnectionOptions,
Expand Down Expand Up @@ -41,6 +43,7 @@ pub fn init(ruby: &Ruby) -> Result<(), Error> {
class.define_method("async_invoke_rpc", method!(Client::async_invoke_rpc, -1))?;
class.define_method("update_metadata", method!(Client::update_metadata, 1))?;
class.define_method("update_api_key", method!(Client::update_api_key, 1))?;
class.define_method("close", method!(Client::close, 0))?;

let inner_class = class.define_error("RPCFailure", ruby.get_inner(&ROOT_ERR))?;
inner_class.define_method("code", method!(RpcFailure::code, 0))?;
Expand All @@ -56,7 +59,7 @@ pub fn init(ruby: &Ruby) -> Result<(), Error> {
#[derive(TypedData)]
#[magnus(class = "Temporalio::Internal::Bridge::Client", free_immediately)]
pub struct Client {
pub(crate) core: Connection,
core: RefCell<Option<Connection>>,
pub(crate) runtime_handle: RuntimeHandle,
}

Expand All @@ -73,13 +76,13 @@ macro_rules! rpc_call {
($client:ident, $callback:ident, $call:ident, $trait:tt, $service_method:ident, $call_name:ident) => {{
let cancel_token = $call.cancel_token.clone();
if $call.retry {
let mut connection = $client.core.clone();
let mut connection = $client.core()?;
let req = $call.into_request()?;
$crate::client::rpc_resp($client, $callback, cancel_token, async move {
$trait::$call_name(&mut connection, req).await
})
} else {
let connection = $client.core.clone();
let connection = $client.core()?;
let req = $call.into_request()?;
$crate::client::rpc_resp($client, $callback, cancel_token, async move {
connection.$service_method().$call_name(req).await
Expand Down Expand Up @@ -248,7 +251,7 @@ impl Client {
Ok(core) => callback.push(
&ruby,
Client {
core,
core: RefCell::new(Some(core)),
runtime_handle,
},
),
Expand Down Expand Up @@ -313,17 +316,30 @@ impl Client {
pub fn update_metadata(&self, headers: RHash) -> Result<(), Error> {
let ruby = Ruby::get().expect("Ruby not available");
let headers = partition_grpc_headers(&ruby, headers)?;
self.core
.set_headers(headers.headers)
let core = self.core()?;
core.set_headers(headers.headers)
.map_err(|err| error!("Invalid headers: {}", err))?;
self.core
.set_binary_headers(headers.binary_headers)
core.set_binary_headers(headers.binary_headers)
.map_err(|err| error!("Invalid headers: {}", err))?;
Ok(())
}

pub fn update_api_key(&self, api_key: Option<String>) {
self.core.set_api_key(api_key);
pub fn update_api_key(&self, api_key: Option<String>) -> Result<(), Error> {
self.core()?.set_api_key(api_key);
Ok(())
}

pub fn close(&self) -> Result<(), Error> {
self.runtime_handle.fork_check("close client")?;
self.core.borrow_mut().take();
Ok(())
}

pub(crate) fn core(&self) -> Result<Connection, Error> {
self.core
.borrow()
.clone()
.ok_or_else(|| error!("Client is closed"))
}
}

Expand Down
16 changes: 9 additions & 7 deletions temporalio/ext/src/worker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ pub fn init(ruby: &Ruby) -> Result<(), Error> {
)?;
class.define_method("replace_client", method!(Worker::replace_client, 1))?;
class.define_method("initiate_shutdown", method!(Worker::initiate_shutdown, 0))?;
class.define_method("finalized?", method!(Worker::finalized, 0))?;

let inner_class = class.define_class("WorkflowReplayer", ruby.class_object())?;
inner_class.define_singleton_method("new", function!(WorkflowReplayer::new, 2))?;
Expand Down Expand Up @@ -125,12 +126,9 @@ impl Worker {
config.task_types.enable_local_activities || config.task_types.enable_remote_activities;
let workflow = config.task_types.enable_workflows;

let worker = temporalio_sdk_core::init_worker(
&client.runtime_handle.core,
config,
client.core.clone(),
)
.map_err(|err| error!("Failed creating worker: {}", err))?;
let worker =
temporalio_sdk_core::init_worker(&client.runtime_handle.core, config, client.core()?)
.map_err(|err| error!("Failed creating worker: {}", err))?;

Ok(Worker {
core: RefCell::new(Some(Arc::new(worker))),
Expand Down Expand Up @@ -402,7 +400,7 @@ impl Worker {
enter_sync!(self.runtime_handle);
let worker = self.core.borrow().as_ref().unwrap().clone();
worker
.replace_client(client.core.clone())
.replace_client(client.core()?)
.map_err(|err| error!("Failed replacing client: {}", err))
}

Expand All @@ -412,6 +410,10 @@ impl Worker {
worker.initiate_shutdown();
Ok(())
}

pub fn finalized(&self) -> bool {
self.core.borrow().is_none()
}
}

#[derive(TypedData)]
Expand Down
1 change: 1 addition & 0 deletions temporalio/extra/aws_lambda/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
/adot/
18 changes: 18 additions & 0 deletions temporalio/extra/aws_lambda/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
FROM public.ecr.aws/lambda/ruby:3.4 AS worker

ARG SDK_GEM
COPY ${SDK_GEM} /tmp/temporalio.gem
RUN gem install /tmp/temporalio.gem --no-document \
&& gem install opentelemetry-sdk opentelemetry-exporter-otlp --no-document \
&& rm /tmp/temporalio.gem

COPY temporalio/extra/aws_lambda/lambda_function.rb ${LAMBDA_TASK_ROOT}/
COPY temporalio/extra/aws_lambda/telemetry.rb ${LAMBDA_TASK_ROOT}/
COPY temporalio/extra/aws_lambda/otel-collector-config.yaml ${LAMBDA_TASK_ROOT}/

CMD ["lambda_function.lambda_handler"]

FROM worker AS with-adot
COPY temporalio/extra/aws_lambda/adot/ /opt/

FROM worker AS default
7 changes: 7 additions & 0 deletions temporalio/extra/aws_lambda/Dockerfile.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
.git
**/target
**/tmp
**/.bundle
**/doc
**/sdk-core
**/.gem_rbs_collection
102 changes: 102 additions & 0 deletions temporalio/extra/aws_lambda/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# AWS Lambda worker sample

This sample packages the SDK from this checkout in the AWS Ruby 3.4 container image. Each invocation runs a worker
until its shutdown budget is reached. The workflow and activity arguments remain positional; the AWS handler accepts
`event:` and `context:` keywords.

## Build

From `temporalio/`, build a native gem for the Lambda architecture using the repository's pinned protobuf compiler:

```sh
RB_SYS_VERSION=$(bundle exec ruby -e 'require "rb_sys/version"; puts RbSys::VERSION')
PROTOC_VERSION=$(cat ../.protoc-version)
docker build --platform linux/amd64 \
--build-arg BASE_IMAGE="rbsys/aarch64-linux:$RB_SYS_VERSION" \
--build-arg PROTOC_VERSION="$PROTOC_VERSION" \
-f ../.github/docker/rb-sys-protoc.Dockerfile \
-t temporal-ruby-lambda-builder ..
RCD_IMAGE=temporal-ruby-lambda-builder \
bundle exec rb-sys-dock --platform aarch64-linux --ruby-versions 3.4 -- \
'bundle exec ruby extra/aws_lambda/build_gem.rb'
```

From the repository root, build the image using the resulting gem:

```sh
docker build --platform linux/arm64 \
-f temporalio/extra/aws_lambda/Dockerfile \
--build-arg SDK_GEM=temporalio/pkg/temporalio-1.9.0-aarch64-linux.gem \
-t temporal-ruby-lambda .
```

For x86-64, use `x86_64-linux`, `linux/amd64`, and a Lambda function configured for the x86-64 architecture.
Also add `--build-arg GCC_VERSION=10` when building the native gem builder, as in the release workflow.
The Dockerfile installs the optional telemetry gems; tracing is enabled only when `TEMPORAL_LAMBDA_OTEL=1`.

## Configure and deploy

Publish the image to your ECR repository and use it to create a Lambda function with an invocation timeout of at
least 15 seconds. Configure these environment variables:

| Variable | Value |
| --- | --- |
| `TEMPORAL_ADDRESS` | Temporal endpoint, including its port |
| `TEMPORAL_NAMESPACE` | Temporal namespace |
| `TEMPORAL_API_KEY` | API key, if used; TLS is enabled automatically |
| `TEMPORAL_TASK_QUEUE` | Task queue shared with the workflow starter |
| `TEMPORAL_DEPLOYMENT_NAME` | Worker deployment name |
| `TEMPORAL_WORKER_BUILD_ID` | Immutable build identifier, such as the source commit SHA |

The execution role needs the usual Lambda logging permissions. Configure Temporal Cloud's AWS Lambda compute
integration to invoke this function for the deployment version. Follow the
[Serverless Workers deployment guide](https://docs.temporal.io/production-deployment/worker-deployments/serverless-workers/aws-lambda).
The function needs network access to the Temporal endpoint.

Register the version by invoking the function once before starting a workflow pinned to it. To run the sample, set
the same Temporal and deployment environment variables locally, then run from `temporalio/`:

```sh
bundle exec ruby extra/aws_lambda/start_workflow.rb Ruby
```

The result is `Hello, Ruby!`. The event sent to the Lambda function does not contain workflow arguments; the worker
polls Temporal for tasks.

## Optional ADOT telemetry

Container functions bundle extensions in the image. Download the ADOT Collector Lambda layer matching the function
architecture and region, and extract its zip into `temporalio/extra/aws_lambda/adot/`. Choose the collector-only layer
from the [ADOT collector layer ARNs](https://aws-otel.github.io/docs/getting-started/lambda/lambda-go/).
The AWS CLI's `get-layer-version-by-arn` command returns a download URL in `Content.Location`.
Rebuild from the repository root with the optional target:

```sh
docker build --platform linux/arm64 --target with-adot \
-f temporalio/extra/aws_lambda/Dockerfile \
--build-arg SDK_GEM=temporalio/pkg/temporalio-1.9.0-aarch64-linux.gem \
-t temporal-ruby-lambda-adot .
```

Enable `TEMPORAL_LAMBDA_OTEL=1` for this image.
Set `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` and `AWS_REGION`.
The bundled collector configuration exports traces to X-Ray and metrics to CloudWatch Logs through EMF.
The execution role must allow the collector to write X-Ray segments and its CloudWatch log group/streams.

The Ruby trace exporter sends OTLP over HTTP to `localhost:4318/v1/traces`; Core metrics use gRPC on `localhost:4317`.
Override these separately with `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_ENDPOINT`.
ADOT converts OpenTelemetry trace IDs to X-Ray's format. The plugin flushes the tracer provider after the worker drains.
Core metrics export periodically; they have no explicit flush API. This sample exports them every second. Choose
`metric_periodicity:` shorter than the invocation budget when customizing the telemetry plugin.

## Local verification

The SDK tests use a local Temporal server and fake Lambda contexts to verify deadline handling, worker shutdown,
activity retries, and workflow continuation across invocations. From `temporalio/`:

```sh
bundle exec rake test TESTOPTS="--name=/LambdaWorker/"
TEMPORAL_SORBET_RUNTIME_CHECK=1 bundle exec rake test TESTOPTS="--name=/LambdaWorker/"
```

These tests do not create or invoke AWS resources.
13 changes: 13 additions & 0 deletions temporalio/extra/aws_lambda/build_gem.rb
Loading
Loading