Summary

Rebuilds the change tracking pointer bookkeeping (change_tracking_insert_scrape / diff_get_last_scrape_v7) on GCP Bigtable instead of Postgres. The PG table is pure "latest value per key" bookkeeping — content lives in GCS — so one row per scrape is overkill; the insert path is ~30% of database CPU time. The PG path is removed: Bigtable is the store, no flag.

Design:

• Row key: team_id ‖ sha256(url) ‖ sha256(tag), raw bytes, full 32-byte digests
  • team_id first → team-scoped prefix operations (bulk delete on cleanup) stay a single range
  • url before tag → the dominant write mix (every plain-markdown scrape) carries tag=null, so the high-entropy component occupies the first hashed position
  • injective null-tag encoding (0x00 vs 0x01‖tag) preserves PG's null-vs-empty-tag distinction
• Single cell m:job = job_id, cell timestamp = date_added (µs), column family GC max_versions=1
• Semantics: max-date-wins independent of arrival order — an out-of-order (regressed) write is accepted, invisible to latest-version reads, and physically collected at compaction
• Unconfigured BIGTABLE_* vars make the store throw, landing in the existing non-fatal paths: the insert warns and continues, deriveDiff sets the "Comparing failed" warning — the same degraded posture self-host had before (change tracking remains unavailable in the self-hosting path)

Out-of-band table creation

The app never provisions infrastructure. Before deploying:

gcloud bigtable instances create firecrawl-prod \
  --cluster=firecrawl-prod-c1 --cluster-location=us-east1 \
  --nodes=3 --storage=ssd

gcloud bigtable instances tables create change_tracking \
  --instance=firecrawl-prod \
  --column-families="m:maxversions=1"

(Equivalently: cbt -instance=firecrawl-prod createtable change_tracking families=m:maxversions=1 — families is a positional argument and the GC knob is maxversions, no underscore.)

Env: BIGTABLE_PROJECT_ID, BIGTABLE_INSTANCE_ID, BIGTABLE_APP_PROFILE_ID (optional), BIGTABLE_CHANGE_TRACKING_TABLE (optional, default change_tracking).

Data migration from the PG table is a one-off ops task run alongside the deploy (distinct on (team_id, url, change_tracking_tag) ... order by date_added desc → bulk mutate with cell timestamp = date_added; idempotent, latest-wins makes re-runs safe). After cutover, drop change_tracking_scrapes and its RPCs server-side — that's the 30% CPU recovery.

Tests

• Store snips (TEST_PRODUCTION-gated like the other production snips): write/read round-trip incl. date_added fidelity, newest-wins, regressed-write-shadowed, null/empty/named tag partitioning, team isolation, missing row → null, key layout invariants
• E2E snip: first scrape of a fresh url+tag reports new/previousScrapeAt: null; second scrape reports previousScrapeAt + changed|same — verified locally through the harness (server write → read → GCS fetch all exercised, 8/8)

Dependency notes

• GHSA-8988-4f7v-96qf (@opentelemetry/core < 2.8.0) ships through the bigtable client's OTel 1.x metrics chain; the advisory is only patched on 2.x. The overrides move the tree's OTel trio + Google's monitoring exporter to 2.x, and the client constructs with metricsEnabled: false since its metrics handler is written against OTel 1.x APIs we don't consume. Audit is clean of any new findings (the remaining ip-address/decode-uri-component advisories pre-date this PR and ip-address is allowlisted in audit-ci.jsonc).
• Cell timestamps come back as strings (Long-as-string), so conversions go through Number(), never typeof === "number".