fix(retention): switch data retention to be org-level by TheodoreSpeaks · Pull Request #4270 · simstudioai/sim · GitHub
Skip to content

fix(retention): switch data retention to be org-level#4270

Merged
TheodoreSpeaks merged 8 commits intostagingfrom
feat/data-retention-storage
Apr 23, 2026
Merged

fix(retention): switch data retention to be org-level#4270
TheodoreSpeaks merged 8 commits intostagingfrom
feat/data-retention-storage

Conversation

@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator

@TheodoreSpeaks TheodoreSpeaks commented Apr 23, 2026

Summary

Switched data retention setting to be org level. This allows for easier organizational management, since we don't expect users to want this per workspace.

Also fixed bug where mothership ran workflow logs weren't being cleaned up

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation
  • Other: ___________

Testing

  • Tested locally, validated workspaces under org are cleaned up correctly.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 23, 2026

@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator Author

@BugBot review

@cursor
Copy link
Copy Markdown

cursor Bot commented Apr 23, 2026

PR Summary

Medium Risk
Moderate risk due to a schema migration that moves retention configuration from workspace columns to organization.data_retention_settings, plus updated cleanup dispatch logic that could change what gets deleted across all workspaces in an org.

Overview
Switches Enterprise data retention from per-workspace to organization-wide. Retention settings are now stored on organization.data_retention_settings (JSON) and the per-workspace retention columns are dropped via migration.

Adds a new org-scoped API at GET/PUT /api/organizations/[id]/data-retention with member read access and owner/admin write access (including audit logging), and updates the EE settings UI + React Query hooks to read/update org settings instead of workspace settings.

Updates cleanup job dispatching to resolve enterprise retention from the owning organization’s settings, and fixes log cleanup to also purge job_execution_logs in addition to workflow_execution_logs (with updated logging).

Reviewed by Cursor Bugbot for commit 39bb125. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread apps/sim/lib/billing/cleanup-dispatcher.ts Outdated
@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator Author

@BugBot review

Comment thread apps/sim/ee/data-retention/components/data-retention-settings.tsx Outdated
@TheodoreSpeaks
Copy link
Copy Markdown
Collaborator Author

@BugBot review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 78151ad. Configure here.

Comment thread packages/db/migrations/0196_retention_org_level.sql
…n-storage

# Conflicts:
#	apps/sim/app/api/workspaces/[id]/data-retention/route.ts
@TheodoreSpeaks TheodoreSpeaks marked this pull request as ready for review April 23, 2026 06:18
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 23, 2026

Greptile Summary

This PR moves data retention configuration from the workspace level to the organization level, replacing three integer columns on workspace with a single dataRetentionSettings JSON column on organization. It also fixes a bug where jobExecutionLogs (mothership-ran workflow logs) were not being cleaned up alongside workflowExecutionLogs.

  • The SQL migration drops the three per-workspace retention columns without a preceding UPDATE to copy any existing values into organization.data_retention_settings, so any enterprise workspace that already had retention configured will lose those settings silently when the migration runs.

Confidence Score: 4/5

Safe to merge after confirming no production enterprise workspaces have existing per-workspace retention values configured, or after adding a data migration step.

One P1 concern: the migration irreversibly drops three workspace retention columns without migrating existing data. If any enterprise workspace has these set in production, those settings are permanently lost. The rest of the PR is well-structured with correct auth checks, proper cache invalidation, and a clean bug fix for job execution log cleanup.

packages/db/migrations/0196_retention_org_level.sql — needs a data migration step before the DROP COLUMN statements.

Important Files Changed

Filename Overview
packages/db/migrations/0196_retention_org_level.sql Adds data_retention_settings JSON column to organization and drops the three per-workspace retention columns; no data migration step to preserve existing settings.
apps/sim/app/api/organizations/[id]/data-retention/route.ts New org-level GET/PUT endpoint; role check (owner/admin), enterprise plan gate, and audit logging all look correct.
apps/sim/lib/billing/cleanup-dispatcher.ts Dispatch logic migrated from workspace columns to org-level JSON; sql.raw() for JSON key extraction is safe with current constants but uses a fragile pattern.
apps/sim/background/cleanup-logs.ts Adds cleanupJobExecutionLogsTier to clean up mothership-ran workflow logs; logic mirrors existing tier cleanup correctly.
packages/db/schema.ts Adds nullable dataRetentionSettings JSON to organization; removes the three integer retention columns from workspace.
apps/sim/ee/data-retention/components/data-retention-settings.tsx Switched from workspace params + permission context to active organization + session email for role resolution; UI messages updated appropriately.
apps/sim/ee/data-retention/hooks/data-retention.ts Hook renamed from workspace to org variants; fetches/mutations now target /api/organizations/[id]/data-retention; cache invalidation key updated correctly.
apps/sim/app/api/workspaces/[id]/data-retention/route.ts Correctly deleted; replaced by the new org-level endpoint.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[dispatchCleanupJobs] --> B{Plan type?}
    B -->|free/pro/team| C[resolveWorkspaceIdsForPlan]
    B -->|enterprise| D[Query workspaces\nINNER JOIN organization\nINNER JOIN subscription\nWHERE org.dataRetentionSettings->>key IS NOT NULL]
    C --> E[Enqueue cleanup job\nplan payload]
    D --> F[Enqueue cleanup job\nworkspaceId payload]
    E --> G[resolveCleanupScope]
    F --> G
    G -->|plan payload| H[Use CLEANUP_CONFIG defaults]
    G -->|enterprise payload| I[JOIN workspace → organization\nRead settings->>key]
    H --> J[runCleanupLogs]
    I --> J
    J --> K[cleanupTier\nworkflow_execution_logs]
    J --> L[cleanupJobExecutionLogsTier\njob_execution_logs]
Loading

Reviews (1): Last reviewed commit: "Merge remote-tracking branch 'origin/sta..." | Re-trigger Greptile

Comment thread packages/db/migrations/0196_retention_org_level.sql
Comment thread apps/sim/lib/billing/cleanup-dispatcher.ts
@TheodoreSpeaks TheodoreSpeaks merged commit 65972f2 into staging Apr 23, 2026
14 checks passed
@TheodoreSpeaks TheodoreSpeaks deleted the feat/data-retention-storage branch April 23, 2026 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant