Read Snowflake result partitions using the SQL API contract by Maya-Kid · Pull Request #1062 · deeplethe/utopia · GitHub
Skip to content

Read Snowflake result partitions using the SQL API contract - #1062

Merged
WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:codex/snowflake-result-partitions
Oct 3, 2026
Merged

WaylandYang merged 2 commits into
deeplethe:devfrom
Maya-Kid:codex/snowflake-result-partitions

Conversation

@Maya-Kid

@Maya-Kid Maya-Kid commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Snowflake SQL API partition metadata contains row counts and sizes, not URLs. Any real multi-partition query therefore failed with partition without a url, including schema ingestion. The previous fixture inadvertently modeled the same nonexistent field.

Fetch subsequent partitions using the statement handle and ?partition=N, starting at 1 because partition 0 is returned inline. Reuse the initial column types for metadata-free partition responses and keep the existing completeness checks. Correct the fixtures to match the official response contract, including gzip compression.

The corrected schema regression failed before the fix. Tests also cover asynchronous execution, typed/null rows across partitions, authentication, missing handles, HTTP errors, and missing rows.

Validation: format check, workspace clippy with warnings denied, guarded workspace tests with dedicated PostgreSQL 16/pgvector and PDF extraction required, and workspace build passed. Frozen pnpm 10.2.1 install, 228 frontend tests, and production build also passed. Cloud behavior is validated against documented HTTP fixtures; no live Snowflake account was used.

…ete queries

Signed-off-by: dada-yan <BinjunYann@gmail.com>

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Maya-Kid. This matches the SQL API: partitionInfo carries row counts and sizes, not URLs, and later partitions are read by statement handle with ?partition=N, without metadata, so the first response's column types are reused. The earlier fixture modelled a field the API does not have. Not verified against a live account, as you say; #242 stays open for that. Landing it.

@WaylandYang
WaylandYang merged commit 3bbdd7b into deeplethe:dev Oct 3, 2026
6 checks passed
@WaylandYang WaylandYang mentioned this pull request Oct 3, 2026
@Maya-Kid
Maya-Kid deleted the codex/snowflake-result-partitions branch October 15, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants