Dogfood Workflow for Intent sync by tannerlinsley · Pull Request #951 · TanStack/tanstack.com · GitHub
Skip to content

Dogfood Workflow for Intent sync#951

Open
tannerlinsley wants to merge 11 commits into
mainfrom
split/workflow-intent-sync-poc
Open

Dogfood Workflow for Intent sync#951
tannerlinsley wants to merge 11 commits into
mainfrom
split/workflow-intent-sync-poc

Conversation

@tannerlinsley

@tannerlinsley tannerlinsley commented May 29, 2026

Copy link
Copy Markdown
Member

Summary

  • Migrate Intent sync orchestration to the TanStack Workflow runtime/store/Netlify sweep path.
  • Add Postgres workflow runtime schema/migration plus admin visibility for recent workflow runs.
  • Add workflow CLI/manual testing docs and focused workflow/runtime tests.

Validation

  • pnpm test
  • node --import tsx --test tests/intent-workflow.test.ts
  • pnpm exec dotenv -e /Users/tannerlinsley/GitHub/tanstack.com/.env.local -- sh -c 'exec node node_modules/.pnpm/cross-env@7.0.3/node_modules/cross-env/src/bin/cross-env.js INTENT_WORKFLOW_DB_TESTS=1 node --import tsx --test tests/intent-workflow.test.ts'
  • pnpm workflow list-workflows

Notes

  • Workflow adapter packages currently publish workspace:* internal deps, so .pnpmfile.cjs rewrites those manifests until packages are republished.
  • drizzle/migrations is normally ignored; this PR force-adds the workflow runtime migration.

Summary by CodeRabbit

  • New Features

    • Added workflow runtime system for managing long-running background tasks with persistence and recovery.
    • Added admin panel view to monitor workflow execution status and history.
  • Refactor

    • Transitioned intent discovery and processing from traditional background jobs to workflow-based execution.
  • Tests

    • Added comprehensive workflow execution tests including retry, idempotency, and failure scenarios.
  • Chores

    • Added workflow runtime dependencies and database schema migration.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 29, 2026

Copy link
Copy Markdown

@tannerlinsley tannerlinsley marked this pull request as ready for review May 29, 2026 21:26

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
src/utils/intent-sync.server.ts (1)

258-259: 💤 Low value

Redundant database write: markPackageVerified duplicates the upsert.

upsertIntentPackage with verified: true already sets both verified and lastSyncedAt on conflict. The subsequent markPackageVerified call performs the same update.

♻️ Suggested fix
   await upsertIntentPackage({ name: packageJson.name, verified: true })
-  await markPackageVerified(packageJson.name)

   const packument = await fetchPackument(packageJson.name)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/intent-sync.server.ts` around lines 258 - 259, The second database
write is redundant: upsertIntentPackage({ name: packageJson.name, verified: true
}) already sets verified and lastSyncedAt on conflict, so remove the subsequent
markPackageVerified(packageJson.name) call (or guard it behind a meaningful
conditional) to avoid duplicate updates; locate the calls to upsertIntentPackage
and markPackageVerified and delete or skip the markPackageVerified invocation
for packageJson.name.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@netlify/functions/workflow-sweep-background.ts`:
- Around line 10-19: The current handler exported via
createNetlifyWorkflowSweepHandler is both a scheduled function (config.schedule
= WORKFLOW_RUNTIME_SWEEP_CRON) and uses the -background deployment model, which
Netlify doesn’t support; split responsibilities: create one scheduled function
(remove the -background suffix from its export/name) that calls or enqueues the
sweep (use createNetlifyWorkflowSweepConfig with schedule =
WORKFLOW_RUNTIME_SWEEP_CRON and export a normal function that triggers the job),
and create a separate background function (keep the -background suffix) that
implements the long-running work using createNetlifyWorkflowSweepHandler with
appropriate maxDurationMs/maxTimers; alternatively, if you prefer a single
function, remove the -background suffix and lower maxDurationMs to the
scheduled-function limit so the scheduled function can run directly. Ensure
references to createNetlifyWorkflowSweepHandler,
createNetlifyWorkflowSweepConfig, and WORKFLOW_RUNTIME_SWEEP_CRON are updated
accordingly.

In `@src/utils/intent-admin.server.ts`:
- Around line 85-99: The current code calls workflowExecutionStore.listRuns
twice with limit: 5 (for INTENT_DISCOVER_WORKFLOW_ID and
INTENT_PROCESS_WORKFLOW_ID), but then merges and slices to 10, which can drop
newer runs; update the per-workflow fetch to request up to the final desired cap
(e.g., limit: 10) so each call to
workflowExecutionStore.listRuns(INTENT_DISCOVER_WORKFLOW_ID,
INTENT_PROCESS_WORKFLOW_ID) can return up to 10 entries before merging, then
keep the existing .flat().sort(...).slice(0, 10) logic to produce the true 10
most-recent runs.

---

Nitpick comments:
In `@src/utils/intent-sync.server.ts`:
- Around line 258-259: The second database write is redundant:
upsertIntentPackage({ name: packageJson.name, verified: true }) already sets
verified and lastSyncedAt on conflict, so remove the subsequent
markPackageVerified(packageJson.name) call (or guard it behind a meaningful
conditional) to avoid duplicate updates; locate the calls to upsertIntentPackage
and markPackageVerified and delete or skip the markPackageVerified invocation
for packageJson.name.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 9b70ed18-5e2a-4881-89ff-49dfec4bd23f

📥 Commits

Reviewing files that changed from the base of the PR and between d99d765 and bb5d0ac.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (19)
  • .pnpmfile.cjs
  • docs/internal/workflow-adapter-research.md
  • docs/internal/workflow-intent-sync.md
  • drizzle/migrations/0000_workflow_run_store.sql
  • netlify/functions/sync-intent-discover-background.ts
  • netlify/functions/sync-intent-process-background.ts
  • netlify/functions/workflow-sweep-background.ts
  • package.json
  • scripts/workflow.ts
  • src/db/schema.ts
  • src/routes/admin/intent.tsx
  • src/utils/intent-admin.functions.ts
  • src/utils/intent-admin.server.ts
  • src/utils/intent-db.server.ts
  • src/utils/intent-sync.server.ts
  • src/utils/intent-workflows.server.ts
  • src/utils/workflow-registrations.server.ts
  • src/utils/workflow-runtime.server.ts
  • tests/intent-workflow.test.ts
💤 Files with no reviewable changes (2)
  • netlify/functions/sync-intent-process-background.ts
  • netlify/functions/sync-intent-discover-background.ts

Comment on lines +10 to +19
export default createNetlifyWorkflowSweepHandler({
runtime: workflowRuntime,
maxDurationMs: 25_000,
maxScheduledRuns: 10,
maxTimers: 10,
})

export const config = createNetlifyWorkflowSweepConfig({
schedule: WORKFLOW_RUNTIME_SWEEP_CRON,
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

Does Netlify support a scheduled function that is also a background function (-background suffix) in 2026?

💡 Result:

As of May 2026, Netlify does not support a single function that is simultaneously a scheduled function and a background function. These are distinct types of serverless functions with different invocation models and limitations [1][2]. Scheduled functions are invoked by Netlify on a regular, cron-like schedule and are intended for automated, periodic tasks [1][3]. They are designed to run for a limited duration (typically 30 seconds) and do not support direct URL invocation or payload processing [1]. Background functions (defined by the -background suffix in the filename) are designed for long-running, asynchronous tasks—supporting execution times of up to 15 minutes—and are typically triggered by client-side HTTP requests [2]. Because these two configurations serve different purposes and utilize different invocation patterns, they cannot be combined into a single file or deployment construct [1][2]. Users needing both scheduling and long-running capabilities typically achieve this by having a scheduled function trigger a separate background function or a workflow process [1][4]. Attempts to apply both configurations to the same function—such as naming a file with a -background suffix while also including a schedule configuration—are not natively supported and will not function as expected [5].

Citations:


Fix -background + config.schedule usage for the workflow sweep handler

Netlify doesn’t support deploying a single function that is both scheduled (config.schedule) and a background function (-background suffix) (as of May 2026). This means the sweep likely won’t run as intended (e.g., the schedule may be ignored or invocation won’t match the background model). Split this into a scheduled function that triggers/enqueues the sweep, and run the long-running work in a separate background function (or remove -background and align maxDurationMs to the scheduled-function limit).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@netlify/functions/workflow-sweep-background.ts` around lines 10 - 19, The
current handler exported via createNetlifyWorkflowSweepHandler is both a
scheduled function (config.schedule = WORKFLOW_RUNTIME_SWEEP_CRON) and uses the
-background deployment model, which Netlify doesn’t support; split
responsibilities: create one scheduled function (remove the -background suffix
from its export/name) that calls or enqueues the sweep (use
createNetlifyWorkflowSweepConfig with schedule = WORKFLOW_RUNTIME_SWEEP_CRON and
export a normal function that triggers the job), and create a separate
background function (keep the -background suffix) that implements the
long-running work using createNetlifyWorkflowSweepHandler with appropriate
maxDurationMs/maxTimers; alternatively, if you prefer a single function, remove
the -background suffix and lower maxDurationMs to the scheduled-function limit
so the scheduled function can run directly. Ensure references to
createNetlifyWorkflowSweepHandler, createNetlifyWorkflowSweepConfig, and
WORKFLOW_RUNTIME_SWEEP_CRON are updated accordingly.

Comment thread src/utils/intent-admin.server.ts Outdated
Comment on lines +85 to +99
const runs = await Promise.all([
workflowExecutionStore.listRuns({
workflowId: INTENT_DISCOVER_WORKFLOW_ID,
limit: 5,
}),
workflowExecutionStore.listRuns({
workflowId: INTENT_PROCESS_WORKFLOW_ID,
limit: 5,
}),
])

return runs
.flat()
.sort((a, b) => b.updatedAt - a.updatedAt)
.slice(0, 10)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Per-workflow cap of 5 can yield an inaccurate "10 most recent" list.

Each query is capped at limit: 5, but the merged result is sliced to 10. If one workflow has more than 5 recent runs that are all newer than the other workflow's runs, those extra recent runs are dropped before sorting, so the top-10 won't reflect the true most-recent runs across both. Fetch up to the final cap from each side.

🔧 Proposed fix
   const runs = await Promise.all([
     workflowExecutionStore.listRuns({
       workflowId: INTENT_DISCOVER_WORKFLOW_ID,
-      limit: 5,
+      limit: 10,
     }),
     workflowExecutionStore.listRuns({
       workflowId: INTENT_PROCESS_WORKFLOW_ID,
-      limit: 5,
+      limit: 10,
     }),
   ])
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const runs = await Promise.all([
workflowExecutionStore.listRuns({
workflowId: INTENT_DISCOVER_WORKFLOW_ID,
limit: 5,
}),
workflowExecutionStore.listRuns({
workflowId: INTENT_PROCESS_WORKFLOW_ID,
limit: 5,
}),
])
return runs
.flat()
.sort((a, b) => b.updatedAt - a.updatedAt)
.slice(0, 10)
const runs = await Promise.all([
workflowExecutionStore.listRuns({
workflowId: INTENT_DISCOVER_WORKFLOW_ID,
limit: 10,
}),
workflowExecutionStore.listRuns({
workflowId: INTENT_PROCESS_WORKFLOW_ID,
limit: 10,
}),
])
return runs
.flat()
.sort((a, b) => b.updatedAt - a.updatedAt)
.slice(0, 10)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/utils/intent-admin.server.ts` around lines 85 - 99, The current code
calls workflowExecutionStore.listRuns twice with limit: 5 (for
INTENT_DISCOVER_WORKFLOW_ID and INTENT_PROCESS_WORKFLOW_ID), but then merges and
slices to 10, which can drop newer runs; update the per-workflow fetch to
request up to the final desired cap (e.g., limit: 10) so each call to
workflowExecutionStore.listRuns(INTENT_DISCOVER_WORKFLOW_ID,
INTENT_PROCESS_WORKFLOW_ID) can return up to 10 entries before merging, then
keep the existing .flat().sort(...).slice(0, 10) logic to produce the true 10
most-recent runs.

@tannerlinsley tannerlinsley force-pushed the split/workflow-intent-sync-poc branch from 224037b to 60021d7 Compare May 29, 2026 21:40

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
netlify/functions/workflow-sweep-background.ts (1)

12-14: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

config.schedule on a -background function is still a deployment-model conflict.

Line 13 schedules this function, but this file is still a background function (workflow-sweep-background.ts). Netlify treats scheduled and background functions as distinct invocation models, so this combo is likely to misfire in production. Split into: (1) scheduled function trigger, (2) background worker.

As of May 2026, does Netlify officially support a single function that is BOTH a scheduled function (export const config.schedule) and a background function (filename ends with -background)?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@netlify/functions/workflow-sweep-background.ts` around lines 12 - 14, The
file workflow-sweep-background.ts is both named as a background worker and
exports config.schedule, which Netlify treats as conflicting invocation models;
fix by splitting responsibilities: remove export const config.schedule from
workflow-sweep-background.ts so it remains a pure background function (keep its
background handler like handler or background-compatible export) and create a
new scheduled function (e.g., workflow-sweep-scheduler.ts) that exports export
const config = { schedule: '*/5 * * * *' } and invokes the background worker
(via Netlify background invocation, HTTP call, or your queue mechanism). Ensure
workflow-sweep-background.ts retains only background logic and the new scheduler
file contains only the schedule config and the trigger call.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@netlify/functions/workflow-sweep-background.ts`:
- Around line 12-14: The file workflow-sweep-background.ts is both named as a
background worker and exports config.schedule, which Netlify treats as
conflicting invocation models; fix by splitting responsibilities: remove export
const config.schedule from workflow-sweep-background.ts so it remains a pure
background function (keep its background handler like handler or
background-compatible export) and create a new scheduled function (e.g.,
workflow-sweep-scheduler.ts) that exports export const config = { schedule: '*/5
* * * *' } and invokes the background worker (via Netlify background invocation,
HTTP call, or your queue mechanism). Ensure workflow-sweep-background.ts retains
only background logic and the new scheduler file contains only the schedule
config and the trigger call.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: a6356d64-065e-4cd3-a41b-2b27a77e013b

📥 Commits

Reviewing files that changed from the base of the PR and between bb5d0ac and 60021d7.

📒 Files selected for processing (3)
  • docs/internal/workflow-intent-sync.md
  • netlify/functions/workflow-sweep-background.ts
  • src/utils/workflow-runtime.server.ts
💤 Files with no reviewable changes (1)
  • src/utils/workflow-runtime.server.ts
✅ Files skipped from review due to trivial changes (1)
  • docs/internal/workflow-intent-sync.md

@tannerlinsley tannerlinsley changed the title [codex] Dogfood Workflow for Intent sync Dogfood Workflow for Intent sync May 29, 2026
@socket-security

socket-security Bot commented May 31, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Added@​tanstack/​workflow-netlify@​0.0.3731008393100
Added@​tanstack/​workflow-store-drizzle-postgres@​0.0.4781008494100
Added@​tanstack/​workflow-runtime@​0.0.2781007992100
Added@​tanstack/​workflow-core@​0.0.37810010093100

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant