GitHub - pedrohakial/aws-serverless-python-template: Production-ready serverless Python API template — FastAPI + Mangum + AWS Lambda + RDS PostgreSQL + SQS + Terraform + GitHub Actions CI/CD · GitHub
Skip to content

pedrohakial/aws-serverless-python-template

Folders and files

Repository files navigation

🚀 AWS Serverless Python Template

CI Deploy Python 3.12 FastAPI Terraform License: MIT Code style: ruff

Production-ready serverless API template — the exact stack I use in financial systems at BW Gestão de Investimentos.

Quick Start · Architecture · API Docs · Deploy · Contributing


✨ Features

Capability Implementation
API Framework FastAPI 0.115 + Mangum (Lambda adapter)
Runtime Python 3.12, async/await throughout
Database RDS PostgreSQL 16 + SQLAlchemy 2.0 async ORM
Async Queue SQS with partial batch failure handling
Storage S3 with presigned URL upload/download
Auth JWT (python-jose) + bcrypt passwords
Infrastructure Terraform (API Gateway v2, Lambda, RDS, SQS, S3)
CI/CD GitHub Actions: lint → test → build → deploy
Containers Docker multi-stage build → ECR → Lambda
Observability Structured JSON logs → CloudWatch Logs Insights
Security Bandit, Trivy, IAM least-privilege, encryption at rest
Testing pytest-asyncio, 80%+ coverage, moto for AWS mocks
Code Quality Ruff (lint+format), Mypy strict, pre-commit hooks

🏗️ Architecture

                              ┌─────────────────────────────────────────────────┐
                              │                    AWS Cloud                    │
                              │                                                 │
   Client Request             │  ┌──────────────┐     ┌───────────────────┐    │
  ──────────────────────────► │  │ API Gateway  │────►│  Lambda (API)     │    │
   HTTPS + JWT Bearer         │  │  (HTTP v2)   │     │  FastAPI + Mangum │    │
                              │  └──────────────┘     └─────────┬─────────┘    │
                              │                                  │              │
                              │          ┌───────────────────────┼──────────┐   │
                              │          │                       │          │   │
                              │          ▼                       ▼          ▼   │
                              │  ┌──────────────┐  ┌─────────────────┐  ┌────┐ │
                              │  │  RDS Postgres │  │   SQS Queue     │  │ S3 │ │
                              │  │  (async ORM) │  │  (async jobs)   │  │    │ │
                              │  └──────────────┘  └────────┬────────┘  └────┘ │
                              │                             │                   │
                              │                             ▼                   │
                              │                   ┌──────────────────┐          │
                              │                   │ Lambda (Worker)  │          │
                              │                   │  SQS processor   │          │
                              │                   └──────────────────┘          │
                              │                                                 │
                              │  ┌────────────────────────────────────────────┐ │
                              │  │          CloudWatch Logs + Alarms           │ │
                              │  │     Structured JSON → Logs Insights         │ │
                              │  └────────────────────────────────────────────┘ │
                              │                                                 │
                              │  ┌────────────────┐  ┌────────────────────────┐ │
                              │  │ Secrets Manager│  │     ECR Registry        │ │
                              │  │ (DB URL, JWT)  │  │  (Lambda container)     │ │
                              │  └────────────────┘  └────────────────────────┘ │
                              └─────────────────────────────────────────────────┘

  CI/CD Pipeline (GitHub Actions)
  ─────────────────────────────────────────────────────────────────────────────►
  push → lint → typecheck → test (80% cov) → build image → ECR → terraform apply

Request Lifecycle

1. Client  ──HTTPS──►  API Gateway (HTTP v2)
2. API GW  ──invoke──►  Lambda (cold start ~400ms / warm ~5ms)
3. Lambda  ──async──►  RDS PostgreSQL (via asyncpg + SQLAlchemy)
4. Lambda  ──send──►   SQS Queue   (background jobs)
5. Lambda  ──presign── S3          (direct upload URLs)
6. Lambda  ──read──►   Secrets Manager (DB URL, JWT secret)
7.                     All logs → CloudWatch → Logs Insights

📁 Project Structure

aws-serverless-python-template/
├── app/
│   ├── main.py                 # FastAPI app + Mangum handler
│   ├── api/
│   │   └── v1/
│   │       ├── router.py       # API v1 router
│   │       └── endpoints/
│   │           ├── auth.py     # Login → JWT
│   │           ├── users.py    # User CRUD
│   │           └── items.py    # Item CRUD + S3 upload URLs
│   ├── core/
│   │   ├── config.py           # Pydantic Settings (env vars)
│   │   ├── security.py         # JWT + bcrypt
│   │   └── logging.py          # Structured JSON logging
│   ├── db/
│   │   ├── base.py             # SQLAlchemy base + mixins
│   │   ├── session.py          # Async engine + session factory
│   │   └── models/
│   │       ├── user.py
│   │       └── item.py
│   ├── services/
│   │   ├── auth_service.py     # get_current_user dependency
│   │   ├── s3_service.py       # Presigned URLs
│   │   └── sqs_service.py      # Message publishing
│   └── workers/
│       └── sqs_handler.py      # SQS batch processor (partial failures)
├── terraform/
│   ├── main.tf                 # S3, SQS, Secrets Manager, outputs
│   ├── lambda.tf               # Lambda functions, IAM, CloudWatch alarms
│   ├── api_gateway.tf          # HTTP API v2 + Lambda integration
│   ├── rds.tf                  # PostgreSQL, subnet group, monitoring
│   └── variables.tf            # All input variables
├── .github/workflows/
│   ├── ci.yml                  # Lint + test + security scan
│   └── deploy.yml              # Build image + Terraform apply
├── tests/
│   ├── conftest.py             # Fixtures (in-memory SQLite for unit tests)
│   ├── unit/
│   │   ├── test_health.py
│   │   ├── test_auth.py
│   │   ├── test_items.py
│   │   └── test_sqs_handler.py
│   └── integration/            # (requires live services)
├── Dockerfile                  # Multi-stage → ECR → Lambda
├── docker-compose.yml          # Local: FastAPI + Postgres + LocalStack
├── Makefile                    # Developer workflow shortcuts
├── pyproject.toml              # Poetry deps + Ruff + Mypy + Pytest config
└── .env.example

⚡ Quick Start

Prerequisites

  • Python 3.12+
  • Poetry
  • Docker + Docker Compose
  • AWS CLI configured (for deploy)
  • Terraform ≥ 1.7 (for infra)

1. Clone & Install

git clone https://github.com/pedrohakial/aws-serverless-python-template.git
cd aws-serverless-python-template

# Install dependencies
make dev-install

# Copy env file
cp .env.example .env

2. Run Locally (Full Stack)

# Start Postgres + LocalStack + API in Docker
make dev-compose

# Or run the API directly (requires local Postgres)
make dev

API will be available at:

3. Run Tests

make test          # Unit tests only (fast, no services needed)
make test-cov      # With coverage report
make check         # lint + typecheck + security

🔌 API Reference

Authentication

# Register
curl -X POST http://localhost:8000/api/v1/users/ \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com","full_name":"Your Name","password":"secret"}'

# Login → get token
TOKEN=$(curl -s -X POST http://localhost:8000/api/v1/auth/login \
  -F "username=you@example.com" -F "password=secret" | jq -r .access_token)

# Use token
curl http://localhost:8000/api/v1/users/me \
  -H "Authorization: Bearer $TOKEN"

Items (CRUD + S3 Upload)

# Create item
curl -X POST http://localhost:8000/api/v1/items/ \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"title":"My File","description":"Upload example"}'

# Get presigned S3 upload URL
curl -X POST "http://localhost:8000/api/v1/items/{item_id}/upload-url?content_type=image/png" \
  -H "Authorization: Bearer $TOKEN"

# Upload directly to S3 (no backend in the loop)
curl -X PUT "$PRESIGNED_URL" \
  -H "Content-Type: image/png" \
  --data-binary @./file.png

🚀 Deployment

Prerequisites

  1. AWS account with appropriate permissions
  2. ECR repository created
  3. VPC with private subnets
  4. GitHub secrets configured:
Secret Description
AWS_DEPLOY_ROLE_ARN IAM role ARN for OIDC-based deployment
AWS_REGION Target AWS region
ECR_REPOSITORY ECR repository name
API_ENDPOINT Deployed API URL (for smoke tests)

Deploy to Staging

# Build and push image
docker build -t my-api .
aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_URI
docker tag my-api:latest $ECR_URI/my-api:latest
docker push $ECR_URI/my-api:latest

# Apply infrastructure
make tf-init
make tf-plan VPC_ID=vpc-xxx SUBNET_IDS='["subnet-a","subnet-b"]' DB_PASSWORD=mypassword
make tf-apply VPC_ID=vpc-xxx SUBNET_IDS='["subnet-a","subnet-b"]' DB_PASSWORD=mypassword

Or just push to main — GitHub Actions handles everything automatically.

Environment Configuration

All config lives in AWS Secrets Manager in production. The Lambda reads SECRET_ARN at startup and populates DATABASE_URL and SECRET_KEY from there. No secrets in environment variables.


🔍 Observability

CloudWatch Logs Insights Queries

# P99 latency
filter @type = "REPORT"
| stats percentile(@duration, 99) as p99,
        avg(@duration) as avg_ms,
        count(*) as invocations
  by bin(5m)

# Error rate
filter level = "error"
| stats count(*) as errors by bin(5m)

# Slow DB queries (> 1s)
filter @message like "slow query"
| stats count(*) as slow_queries by bin(1h)

Alarms Configured

  • Lambda error rate > 5 errors/min (2-minute window)
  • P99 latency > 5 seconds

🧪 Testing Strategy

Unit Tests (SQLite in-memory)     → fast, no services, CI gate
Integration Tests (LocalStack)    → realistic AWS mocking
End-to-end Tests (staging env)    → post-deploy smoke tests

Mock AWS Services

# Using moto to mock S3 in tests
import boto3
from moto import mock_aws

@mock_aws
def test_s3_upload():
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket="test-bucket")
    # ... test your S3Service

🛡️ Security

  • IAM least-privilege: Lambda only accesses its own S3/SQS/Secrets
  • No public RDS: Database only accessible from Lambda's security group
  • Secrets Manager: DB credentials and JWT secret never in env vars
  • Encryption: S3 AES-256, RDS storage encrypted, TLS everywhere
  • Bandit + Trivy: Automated SAST and container vuln scanning in CI

📐 Design Decisions

Why NullPool for Lambda?

Lambda functions don't share state between invocations. Using a connection pool that persists connections causes "too many connections" errors. NullPool creates/destroys a connection per request. For high-traffic workloads, add RDS Proxy to pool connections at the infrastructure level.

Why HTTP API v2 over REST API?

HTTP API is 70% cheaper than REST API, has lower latency, and supports JWT authorizers natively. The only trade-off is no API key management — handled at the app layer with JWT.

Why Mangum instead of a custom handler?

Mangum translates API Gateway v2 payload format 2.0 into ASGI, so FastAPI runs unchanged. No custom request/response mapping code to maintain.

Why container image over zip deployment?

  • Dependencies (asyncpg, cryptography) exceed 50MB unzipped
  • Reproducible builds across environments
  • Layer caching in ECR makes updates fast
  • Consistent with the rest of the infrastructure

🤝 Contributing

# Fork and clone
git checkout -b feat/your-feature

# Install dev dependencies
make dev-install

# Make changes, then:
make check      # lint + types + security
make test-cov   # tests with coverage

# Commit with conventional commits
git commit -m "feat: add webhook endpoint"
git push origin feat/your-feature

📄 License

MIT — see LICENSE


Built with ❤️ — Production patterns from real financial systems.

About

Production-ready serverless Python API template — FastAPI + Mangum + AWS Lambda + RDS PostgreSQL + SQS + Terraform + GitHub Actions CI/CD

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

Contributors