cloudflared-operator/PLAN.md at main · context-labs/cloudflared-operator · GitHub
Skip to content

Latest commit

 

History

History
194 lines (163 loc) · 8.02 KB

PLAN.md

File metadata and controls

194 lines (163 loc) · 8.02 KB

Cloudflare Tunnel Operator — Design & Build Plan

A Kubernetes operator (Go, operator-sdk/kubebuilder) that provisions Cloudflare Tunnels and routes Kubernetes Ingresses through them, in the style of the Tailscale operator.

Module: github.com/context-labs/cloudflared-operator SDK: github.com/cloudflare/cloudflare-go/v7 (Stainless-generated; has the ZeroTrust tunnel, hostname-route and connectivity-directory APIs)

Mental model

CloudflareTunnel CR ──► cfd_tunnel (create/delete, token Secret, cloudflared Deployment)
       ▲
       │ annotation: tunnels.cloudflare.com/tunnel: <name>   (REQUIRED — no default)
Ingress ──► hostname route (zerotrust/routes/hostname)   [always]
        └─► VPC service (connectivity/directory/services) [optional]
        └─► public DNS + ingress rule                     [optional]

There is no default tunnel. The operator manages zero tunnels it wasn't told about; every Ingress must name a CloudflareTunnel CR. Users who want a single tunnel create one CR and point everything at it; users who want isolation create many.

Verified account semantics (Inference.net account, Aug 2026)

Example: redis-cache-azure-development (tunnel ID de5136fc-1f27-483f-b61e-395060a208cf)

  1. Tunnel — cfd_tunnel, token-run cloudflared, 2 replicas × 4 colo connections. GET /cfd_tunnel/{id}/configurations returns config: null — ingress rules are NOT stored on the tunnel for the private-hostname flow.
  2. Private Hostname — POST /accounts/{acc}/zerotrust/routes/hostname maps a hostname → tunnel. E.g. redis-cache-azure-dev.redis-cf-tunnel.svc.cluster.local → de5136fc…. This is what routes traffic into the tunnel. cloudflared resolves the hostname inside the cluster (CoreDNS). SDK: client.ZeroTrust.NetworkHostnameRoutes.
  3. Workers VPC service — POST /accounts/{acc}/connectivity/directory/services with type: tcp, tcp_port, and host: { hostname, resolver_network: { tunnel_id } }. E.g. redis-cache-dev → the hostname route above. Workers reference it with a { type: "vpc_service", service_id } binding (managed in the Worker's own config — out of scope for this operator). SDK: client.Connectivity.Directory.Services.

CRD: CloudflareTunnel (tunnels.cloudflare.com/v1alpha1)

apiVersion: tunnels.cloudflare.com/v1alpha1
kind: CloudflareTunnel
metadata:
  name: redis-cache-azure-development
spec:
  # Cloudflare-side tunnel name. Defaults to metadata.name.
  tunnelName: redis-cache-azure-development
  replicas: 2                      # default 2
  image: cloudflare/cloudflared:2026.7.3
  nodeSelector: {}
  tolerations: []
  affinity: {}
  deleteTunnelOnRemove: true       # default true
status:
  tunnelID: de5136fc-…
  connections: 8
  conditions: [Ready, TokenStored, DeploymentReady]

The controller:

  • get-or-creates the CF tunnel by name, stores status.tunnelID
  • fetches the run token into a Secret (<name>-token) in the CR's namespace (owner references require same-namespace children)
  • owns a cloudflared Deployment (same namespace): cloudflared tunnel --no-autoupdate --metrics 0.0.0.0:2000 run --token $TUNNEL_TOKEN, token from the Secret, readiness/liveness probes on :2000/ready
  • finalizer: refuses deletion while Ingresses in the same namespace still reference the tunnel (TunnelInUse warning event); once clear, deletes the Deployment, Secret, and the CF tunnel if deleteTunnelOnRemove (CF refuses deletion while routes exist)
  • spec.credentialsSecretRef (optional): Secret with CLOUDFLARE_API_TOKEN/CLOUDFLARE_ACCOUNT_ID overriding the operator-wide credentials for everything this tunnel touches, including Ingresses routed through it

Ingress integration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: redis-cache
  namespace: redis-cf-tunnel
  annotations:
    # REQUIRED: name of the CloudflareTunnel CR to route through
    tunnels.cloudflare.com/tunnel: redis-cache-azure-development

    # OPTIONAL: Workers VPC service name (must be an actual name, never boolean).
    # Creates a connectivity-directory service so Workers can reach this backend.
    tunnels.cloudflare.com/vpc-service: redis-cache-dev

    # OPTIONAL: "true" exposes the hostname publicly (proxied DNS record +
    # tunnel ingress rule) instead of private-only.
    tunnels.cloudflare.com/public: "false"

    # OPTIONAL: override the private hostname (default: first rule host,
    # else <ingress>.<namespace>.svc.cluster.local)
    tunnels.cloudflare.com/hostname: redis-cache-azure-dev.redis-cf-tunnel.svc.cluster.local
spec:
  ingressClassName: cloudflare        # operator watches this class
  rules:
  - host: redis-cache-azure-dev.redis-cf-tunnel.svc.cluster.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: { name: redis-cache, port: { number: 6379 } }

Ingress controller behavior:

  • Only reconciles Ingresses with ingressClassName: cloudflare (or the legacy kubernetes.io/ingress.class annotation).
  • Requires tunnels.cloudflare.com/tunnel; if absent or the CR doesn't exist, emit a Warning event and skip.
  • Hostname resolution: tunnels.cloudflare.com/hostname → first rule host → generated <name>.<namespace>.svc.cluster.local.
  • Always: ensure a hostname route hostname → tunnelID exists (idempotent get- or-create by hostname; error if it points at a different tunnel).
  • If vpc-service set: ensure a connectivity-directory service { name, type: tcp, tcp_port: <backend port or vpc-port annotation>, host: { hostname, resolver_network: { tunnel_id } } }. The value must be a real name — "true"/"" are rejected with an event.
  • If public: "true": additionally create a proxied DNS CNAME <hostname> → <tunnelID>.cfargotunnel.com in the zone matching the hostname suffix (zone list from CF API), and add the hostname to the tunnel's remote ingress configuration with a http_status:404 catch-all.
  • Finalizer removes exactly the resources it created (tracked via status annotations: route ID, service ID, DNS record ID).

Operator configuration (v1)

One global API token, via a Secret mounted as env on the manager Deployment:

CLOUDFLARE_API_TOKEN     # token with ZeroTrust tunnels, connectivity directory, DNS edit
CLOUDFLARE_ACCOUNT_ID    # e.g. b0c6d69f10b59b4d5b3a1a6d882d95ba

Cloudflare client layer (internal/cloudflare)

Thin, mockable wrapper around cloudflare-go/v7:

type Client interface {
    EnsureTunnel(ctx, name) (Tunnel, error)          // get-or-create by name
    GetTunnelToken(ctx, id) (string, error)
    DeleteTunnel(ctx, id) error
    ListConnections(ctx, id) (int, error)

    EnsureHostnameRoute(ctx, hostname, tunnelID) (routeID string, error)
    DeleteHostnameRoute(ctx, routeID) error

    EnsureVPCService(ctx, name, hostname, tunnelID string, port int64) (serviceID string, error)
    DeleteVPCService(ctx, serviceID) error

    EnsurePublicDNS(ctx, hostname, tunnelID) (recordID string, error)  // zone lookup by suffix
    DeleteDNSRecord(ctx, zoneID, recordID) error

    EnsureIngressRule(ctx, tunnelID, hostname, service string) error   // merges remote config
    RemoveIngressRule(ctx, tunnelID, hostname) error
}

Build phases

  1. Scaffold — operator-sdk init (kubebuilder layout), Go 1.26, controller-runtime latest.
  2. API types — api/v1alpha1/cloudflaretunnel_types.go + generated CRD/RBAC.
  3. Client layer — as above, unit-tested against httptest fake.
  4. Tunnel controller — tunnel, token Secret, cloudflared Deployment, status/conditions.
  5. Ingress controller — annotations, hostname routes, VPC services, public-DNS option, finalizers, events.
  6. Tests — envtest with mock CF client; table tests for annotation parsing.
  7. Packaging — kustomize, make docker-build deploy, README quickstart.

Out of scope (later)

  • cloudflared autoscaling / metrics
  • OLM bundle / OperatorHub publishing
  • Managing Worker scripts' vpc_service bindings (done in Worker config)