A GitHub repository source skips empty files, and its repository, token and globs can be changed after it is created by QuintinBotes · Pull Request #1326 · deeplethe/utopia · GitHub
Skip to content

A GitHub repository source skips empty files, and its repository, token and globs can be changed after it is created - #1326

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
QuintinBotes:feat/github-repo-followups
Oct 21, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
QuintinBotes:feat/github-repo-followups

Conversation

@QuintinBotes

Copy link
Copy Markdown
Contributor

Follow-up to #1325, covering two of the three optional points from its review.

Why

In the #1325 review you pointed out that empty files never become documents, so they were fetched again on every sync. You also noted that a github_repo source couldn't be changed once created: the edit dialog showed no repository, token or glob fields, and the glob fields didn't say that a value replaces the defaults.

What changes

  • Empty files are skipped. FileFilter::eligible now requires size > 0. ingest_item creates nothing for empty bytes, so an empty file used to cost a commits request and a blob request every round. The new unit test an_empty_file_is_not_taken uses the real SHA of git's empty blob.
  • The edit dialog covers github_repo. It shows the same repository, token, include and exclude fields as the create dialog, and changing the repository or a glob shows the existing note about what happens to the old documents.
    • Saving replaces the whole config, so githubRepoConfig builds it from the stored config. Keys the form doesn't show, such as max_file_bytes, survive an edit.
    • A cleared glob field drops its key, so the default applies again.
    • A blank token is left out, so the stored one is kept, as keep_secrets expects.
    • The comma-separated parser globList moved to web/src/pages/githubRepoConfig.ts and is shared by both dialogs.
  • Labels. Both glob labels now say a value replaces the default, in en.ts and zh.ts.

Not in this PR: your other point, storing the git SHA so the unchanged check doesn't read each file back on every sync. That one also covers files whose content another document already holds. It needs a column, so it would be a change of its own.

How it was checked

On an M1 Max (macOS), at the PR head on top of dev d62e880:

  • cargo fmt --all --check: passes.
  • cargo clippy --workspace --all-targets -- -D warnings: no issues.
  • cargo test --workspace --no-fail-fast with UTOPIA_DATABASE_URL pointing at a database created and migrated for this branch alone: 1,646 passed, 0 failed, 6 ignored.
  • pnpm build and pnpm test in web/: build passes, 308 tests pass.
  • Both new tests fail without their change. an_empty_file_is_not_taken fails without size > 0. githubRepoConfig.test.ts fails if the stored config isn't carried over, because max_file_bytes is lost.

Before review

  • Every commit is signed off (git commit -s)
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings and cargo test --workspace pass
  • For changes under web/: pnpm build and pnpm test pass
  • UI strings are in both web/src/i18n/en.ts and zh.ts

…en and globs can be changed after it is created

Signed-off-by: Quintin Botes <quintinbotes@outlook.com>
@WaylandYang

WaylandYang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

@WaylandYang
WaylandYang merged commit b0c3c43 into deeplethe:dev Oct 21, 2026
6 checks passed
gedaiu added a commit to re-cinq/utopia that referenced this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants