feat(opencode): migrate from JSON files to SQLite database by yukukotani · Pull Request #879 · ccusage/ccusage · GitHub
Skip to content

feat(opencode): migrate from JSON files to SQLite database#879

Closed
yukukotani wants to merge 2 commits into
ccusage:mainfrom
yuku-contrib:feat/opencode-sqlite-migration
Closed

feat(opencode): migrate from JSON files to SQLite database#879
yukukotani wants to merge 2 commits into
ccusage:mainfrom
yuku-contrib:feat/opencode-sqlite-migration

Conversation

@yukukotani

@yukukotani yukukotani commented Mar 7, 2026

Copy link
Copy Markdown
Contributor

Summary

OpenCode has migrated data store from JSON files to SQLite database in v1.2.0.

This PR follows that migration.

Note

  • This PR introduces breaking changes. It will not work with OpenCode under v1.2.0.

Summary by CodeRabbit

  • New Features

    • Data storage migrated from per-message files to a centralized SQLite database for sessions and messages.
  • Documentation

    • Updated docs and README to reflect the new database-backed storage model and token/cost data access.
  • Chores

    • Bumped minimum Node.js requirement to v22.13.0.
    • Removed unused development dependencies.

- Update data loading to read from opencode.db SQLite database instead of individual JSON files
- Parse message data from message table and session metadata from session table
- Update documentation to reflect SQLite storage location and structure
- Bump Node.js minimum version from 20.19.4 to 22.13.0
- Remove unused dependencies: fast-sort, path-type, tinyglobby
- Rewrite tests to use fs-fixture and database seeding
- Support both Bun and Node.js runtime SQLite APIs
@coderabbitai

coderabbitai Bot commented Mar 7, 2026

Copy link
Copy Markdown

@yukukotani yukukotani changed the title feat(opencode): JSONファイルからSQLiteデータベースへの移行 feat(opencode): migrate from JSON files to SQLite database Mar 7, 2026
@yukukotani

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Mar 7, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

},
"engines": {
"node": ">=20.19.4"
"node": ">=22.13.0"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumping this since node:sqlite was introduced in Node v22. We can stick with v20 if adding a dependency is okay.

@yukukotani yukukotani marked this pull request as ready for review March 7, 2026 11:50

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
apps/opencode/package.json (1)

47-64: ⚠️ Potential issue | 🔴 Critical

Missing path-type dependency breaks build.

The path-type package was removed from devDependencies, but data-loader.ts (line 17) still imports isDirectorySync from it:

import { isDirectorySync } from 'path-type';

This is used in getOpenCodePath() (lines 187, 194) and will cause runtime failures.

Either restore the dependency or replace the import with the newly added isFile() helper pattern (using statSync wrapped in Result.try).

🐛 Option A: Restore dependency
 	"devDependencies": {
 		"@ccusage/internal": "workspace:*",
 		"@ccusage/terminal": "workspace:*",
 		"@praha/byethrow": "catalog:runtime",
+		"path-type": "catalog:runtime",
🐛 Option B: Replace with native check in data-loader.ts

Add a helper similar to isFile():

function isDirectory(targetPath: string): boolean {
  const statsResult = getPathStats(targetPath);
  if (Result.isFailure(statsResult)) {
    return false;
  }
  return statsResult.value.isDirectory();
}

Then replace isDirectorySync calls with isDirectory.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@apps/opencode/package.json` around lines 47 - 64, The build breaks because
data-loader.ts still imports isDirectorySync while path-type was removed; either
re-add "path-type" to devDependencies or update data-loader.ts to remove the
import and use the native helper pattern: add an isDirectory function that calls
getPathStats (the same helper used by isFile) and returns false on failure and
stats.isDirectory() on success, then replace all uses of isDirectorySync in
getOpenCodePath with this new isDirectory; ensure you remove the path-type
import and use Result.isFailure/Result.try semantics consistent with existing
getPathStats/isFile logic.
🧹 Nitpick comments (2)
apps/opencode/src/data-loader.ts (2)

343-358: Consider making synchronous functions non-async.

Both loadOpenCodeSessions() and loadOpenCodeMessages() are marked async but contain no await expressions - they only call the synchronous queryOpenCodeDatabase(). This works but adds unnecessary Promise wrapping overhead.

♻️ Optional simplification

If backward compatibility allows, these could return synchronously:

-export async function loadOpenCodeSessions(): Promise<Map<string, LoadedSessionMetadata>> {
+export function loadOpenCodeSessions(): Map<string, LoadedSessionMetadata> {

However, keeping them async maintains API consistency if callers expect Promises.

Also applies to: 364-383

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@apps/opencode/src/data-loader.ts` around lines 343 - 358, Remove unnecessary
async/Promise wrapping for functions that don't use await: change
loadOpenCodeSessions (and likewise loadOpenCodeMessages) to be synchronous by
removing the async keyword and updating their return types from
Promise<Map<string, LoadedSessionMetadata>> (or Promise<...>) to Map<string,
LoadedSessionMetadata> (and the corresponding return for messages). Keep the
existing logic that calls the synchronous queryOpenCodeDatabase() and ensure any
callers are updated to handle the synchronous return (or left as-is if they
already await; adjust callers to stop awaiting if you make this change).

16-17: Move test-only import inside vitest block.

fs-fixture is only used within the if (import.meta.vitest != null) block (lines 495, 640). Importing it at the module level includes it in the production bundle unnecessarily.

♻️ Proposed fix

Move the import inside the test block:

-import { createFixture } from 'fs-fixture';
 import { isDirectorySync } from 'path-type';

Then inside the vitest block:

 if (import.meta.vitest != null) {
 	const { afterEach, describe, expect, it, vi } = import.meta.vitest;
+	const { createFixture } = await import('fs-fixture');

Note: Per coding guidelines, dynamic await import() should be avoided. Instead, consider a conditional require pattern or accept the import for tree-shaking at build time if tsdown handles dead-code elimination for the vitest guard.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@apps/opencode/src/data-loader.ts` around lines 16 - 17, The top-level import
"import { createFixture } from 'fs-fixture';" is test-only and should be moved
inside the vitest guard to avoid including it in production bundles: remove the
module-level import and instead require it inside the existing "if
(import.meta.vitest != null) { ... }" block where createFixture is used (the
blocks referencing createFixture), e.g. const { createFixture } =
require('fs-fixture'); so only test code loads fs-fixture; leave isDirectorySync
at module scope if still needed by runtime code. Ensure all references to
createFixture inside the vitest block use the locally required binding.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@apps/opencode/src/data-loader.ts`:
- Around line 522-523: Replace the outdated modelID string 'claude-sonnet-4-5'
with the current Sonnet model name 'claude-sonnet-4-20250514' in the three
test/data fixtures where modelID is set (the occurrences near the existing
'claude-opus-4-20250514' entry); search for modelID: 'claude-sonnet-4-5' in
data-loader.ts and change each to modelID: 'claude-sonnet-4-20250514' so Sonnet
entries match the current naming convention used for Opus.

---

Outside diff comments:
In `@apps/opencode/package.json`:
- Around line 47-64: The build breaks because data-loader.ts still imports
isDirectorySync while path-type was removed; either re-add "path-type" to
devDependencies or update data-loader.ts to remove the import and use the native
helper pattern: add an isDirectory function that calls getPathStats (the same
helper used by isFile) and returns false on failure and stats.isDirectory() on
success, then replace all uses of isDirectorySync in getOpenCodePath with this
new isDirectory; ensure you remove the path-type import and use
Result.isFailure/Result.try semantics consistent with existing
getPathStats/isFile logic.

---

Nitpick comments:
In `@apps/opencode/src/data-loader.ts`:
- Around line 343-358: Remove unnecessary async/Promise wrapping for functions
that don't use await: change loadOpenCodeSessions (and likewise
loadOpenCodeMessages) to be synchronous by removing the async keyword and
updating their return types from Promise<Map<string, LoadedSessionMetadata>> (or
Promise<...>) to Map<string, LoadedSessionMetadata> (and the corresponding
return for messages). Keep the existing logic that calls the synchronous
queryOpenCodeDatabase() and ensure any callers are updated to handle the
synchronous return (or left as-is if they already await; adjust callers to stop
awaiting if you make this change).
- Around line 16-17: The top-level import "import { createFixture } from
'fs-fixture';" is test-only and should be moved inside the vitest guard to avoid
including it in production bundles: remove the module-level import and instead
require it inside the existing "if (import.meta.vitest != null) { ... }" block
where createFixture is used (the blocks referencing createFixture), e.g. const {
createFixture } = require('fs-fixture'); so only test code loads fs-fixture;
leave isDirectorySync at module scope if still needed by runtime code. Ensure
all references to createFixture inside the vitest block use the locally required
binding.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 629c3e86-42b8-41cd-ae6a-e29dca717bcd

📥 Commits

Reviewing files that changed from the base of the PR and between 0adbb4f and 2afa836.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (5)
  • apps/opencode/CLAUDE.md
  • apps/opencode/README.md
  • apps/opencode/package.json
  • apps/opencode/src/data-loader.ts
  • apps/opencode/tsdown.config.ts

Comment thread apps/opencode/src/data-loader.ts
cristoslc added a commit to cristoslc/ccusage-fork that referenced this pull request Apr 16, 2026
…and role filter

- Add SqliteAdapter abstraction so the DB layer works with both
  better-sqlite3 (Node.js) and bun:sqlite (Bun runtime), matching
  the approach in PR ccusage#887 by unsync
- Add role='assistant' filter in SQL query to exclude user messages
  at the database level, matching the approach in PR ccusage#879 by yuku-contrib
- Add normalizeClaudeModelName() to convert OpenCode's dot-notation
  model names (e.g. claude-opus-4.5) to LiteLLM's dash format
  (claude-opus-4-5) for correct pricing lookup, matching PR ccusage#887
- Mark bun:sqlite as external in tsdown config
- Update test data to include role:'assistant' in DB message fixtures
@cristoslc

Copy link
Copy Markdown
Contributor

Your role='assistant' SQL filter was a good call — we adopted it in #943. We kept the legacy JSON fallback (non-breaking) since some installs may have historical data not backfilled into the DB. Would value your take on the cross-source merge approach.

cristoslc added a commit to cristoslc/ccusage-fork that referenced this pull request May 9, 2026
…and role filter

- Add SqliteAdapter abstraction so the DB layer works with both
  better-sqlite3 (Node.js) and bun:sqlite (Bun runtime), matching
  the approach in PR ccusage#887 by unsync
- Add role='assistant' filter in SQL query to exclude user messages
  at the database level, matching the approach in PR ccusage#879 by yuku-contrib
- Add normalizeClaudeModelName() to convert OpenCode's dot-notation
  model names (e.g. claude-opus-4.5) to LiteLLM's dash format
  (claude-opus-4-5) for correct pricing lookup, matching PR ccusage#887
- Mark bun:sqlite as external in tsdown config
- Update test data to include role:'assistant' in DB message fixtures
@ryoppippi

Copy link
Copy Markdown
Member

@ryoppippi ryoppippi closed this May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants