fix(db): preserve restore line separators without psql by Ethereal49 · Pull Request #17080 · paperclipai/paperclip · GitHub
Skip to content

fix(db): preserve restore line separators without psql - #17080

Open
Ethereal49 wants to merge 2 commits into
paperclipai:masterfrom
Ethereal49:fix/js-restore-line-splitting
Open

Ethereal49 wants to merge 2 commits into
paperclipai:masterfrom
Ethereal49:fix/js-restore-line-splitting

Conversation

@Ethereal49

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work.
  • Database backups protect Paperclip data during recovery and migration.
  • The JavaScript backup engine writes SQL text that can contain arbitrary Unicode data.
  • A host without psql uses the JavaScript restore fallback.
  • The fallback used node:readline, which treats U+2028, U+2029, and a lone carriage return as line boundaries.
  • These boundaries change JSON strings and text values during restore.
  • This pull request uses a streaming reader that splits only on line feeds.
  • The benefit is a backup that restores the same character data without PostgreSQL client tools.

Linked Issues or Issue Description

What Changed

  • Replace the readline-based fallback reader with a streaming reader that splits only on \n.
  • Preserve lone carriage returns, CRLF values, U+2028, and U+2029 in statement data.
  • Keep the existing gzip handling, statement breakpoints, trailing statement handling, and error cleanup.
  • Add a JavaScript backup-to-restore test that uses a missing psql path and checks jsonb and text values.
  • No documentation change is needed because this keeps the documented fallback behavior.

Verification

  • Before the fix, a JavaScript backup contained raw U+2028 and U+2029. Restore with a missing PAPERCLIP_PSQL_PATH failed with ERROR: invalid input syntax for type json.
  • npx --yes pnpm@9.15.4 exec vitest run packages/db/src/backup-lib.test.ts: 10 tests passed with Vitest 5.0.3.
  • The backup test covers LF, CR, CRLF, U+2028, and U+2029 in jsonb and text values. It also covers the existing COPY-block fallback cases.
  • npx --yes pnpm@9.15.4 --filter @paperclipai/db typecheck: passed.
  • npx --yes pnpm@9.15.4 -r typecheck: stopped at the existing @paperclipai/paperclip-runner Rust gate because cargo is not installed.
  • npx --yes pnpm@9.15.4 build: stopped at the same cargo gate after the database package built successfully.
  • The full pnpm test:run command was interrupted before a final summary. It is NOT_RUN as a complete suite.
  • git diff --check: passed.

Risks

  • Low risk. The change affects only statement splitting in the JavaScript restore fallback.
  • The new reader must preserve memory and stream behavior for large backups. It reads one chunk at a time and keeps only the unfinished line and current statement.
  • CRLF marker lines remain accepted. The reader does not normalize carriage returns inside SQL data.
  • The full repository suite did not complete locally. Pull request CI must provide the remaining test coverage.

I checked ROADMAP.md. This focused bug fix does not duplicate planned core work.

Model Used

DeepSeek V4.1 Flash, through Codex orchestration. Reasoning effort: max. The context window size was not available in this session. The model used issue review, shell execution, local tests, Git, and GitHub tools. Codex scheduled the work and checked the final result.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I have linked the existing public issue with Fixes: #
  • I have not referenced internal/instance-local Paperclip issues or links
  • My branch name describes the change and contains no internal Paperclip ticket id
  • I have run the focused local tests and they pass
  • I have added regression tests
  • I have checked the relevant docs; this change does not alter public commands or API behavior
  • I have documented the risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

@greptile-apps

greptile-apps Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Comment thread packages/db/src/backup-lib.ts Outdated

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JS restore fallback (no psql) breaks rows containing U+2028/U+2029: readline splits statements on them

1 participant