Add SQLite Input/Output format by vlerdman · Pull Request #49966 · ClickHouse/ClickHouse · GitHub
Skip to content

Add SQLite Input/Output format#49966

Closed
vlerdman wants to merge 31 commits into
ClickHouse:masterfrom
vlerdman:sqlite_format
Closed

Add SQLite Input/Output format#49966
vlerdman wants to merge 31 commits into
ClickHouse:masterfrom
vlerdman:sqlite_format

Conversation

@vlerdman

@vlerdman vlerdman commented May 17, 2023

Copy link
Copy Markdown

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add supports of SQLite input/output format (reading and saving tables from/to SQLite3 .db files).

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

@alexey-milovidov alexey-milovidov added the can be tested Allows running workflows for external contributors label May 18, 2023
@robot-clickhouse-ci-2 robot-clickhouse-ci-2 added the pr-feature Pull request with new product feature label May 18, 2023
@robot-clickhouse-ci-2

robot-clickhouse-ci-2 commented May 18, 2023

Copy link
Copy Markdown
Contributor

@al13n321 al13n321 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FastTest fails because it builds without sqlite. Wrap everything in #if USE_SQLITE like in StorageSQLite.h et al.

StyleCheck fails for various reasons, e.g. tabs, see output on github. If it's not clear what's wrong on a line, play with the regex in utils/check-style/check-style.

Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp Outdated
Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.h Outdated
Comment thread src/Processors/Formats/Impl/SQLiteInputVFS.h Outdated
Comment thread src/Processors/Formats/Impl/SQLiteInputVFS.h Outdated
Comment thread src/Processors/Formats/Impl/SQLiteOutputFormat.cpp Outdated
Comment thread src/Processors/Formats/Impl/SQLiteOutputFormat.cpp Outdated
Comment thread src/Processors/Formats/Impl/SQLiteOutputFormat.cpp Outdated
}
*/

std::string select_query = fmt::format("SELECT * FROM {};", table_name);

@al13n321 al13n321 Aug 9, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can reuse SQLiteSource (or its parts) here instead of querying and deserializing by hand. It does useful things:

  • It selects only the requested columns.
  • It applies filters from WHERE when possible. (Currently this is not possible with input formats, but Parquet filter pushdown #52951 makes it possible.)
  • It reads results in binary form rather than parsing from text.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done yet, leaving this open. SQLiteSource is built around a real SQLite connection opened from a file path (it can select only the requested columns, push down WHERE, and read values in binary form), whereas this input format reads an arbitrary ReadBuffer through a custom read-only VFS, so the two don't share a connection abstraction today. As a smaller correctness step, NULLs are now handled via sqlite3_column_type/sqlite3_column_bytes rather than text length. Reusing SQLiteSource (column projection, predicate push-down, binary reads) would be a good follow-up.

@alexey-milovidov alexey-milovidov marked this pull request as ready for review September 18, 2023 07:49
@alexey-milovidov

Copy link
Copy Markdown
Member

This is not ready, please continue.

@nikitamikhaylov nikitamikhaylov self-assigned this May 13, 2024
@clickhouse-gh

clickhouse-gh Bot commented Jun 25, 2024

Copy link
Copy Markdown
Contributor

Dear @nikitamikhaylov, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

@nikitamikhaylov nikitamikhaylov added the comp-formats Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.). label Dec 12, 2025
alexey-milovidov and others added 4 commits June 3, 2026 18:39
# Conflicts:
#	src/Core/Settings.h
#	src/Core/SettingsChangesHistory.h
#	src/Formats/FormatFactory.cpp
#	src/Formats/registerFormats.cpp
Adapt the format classes to the current code base after merging master:
the `IOutputFormat` and `IRowInputFormat` constructors now take a
`SharedHeader`, `IOutputFormat::flush` is no longer virtual, and format
settings moved to `FormatFactorySettings.h`.

Address review feedback from the original review:

- Move the read-only VFS implementation out of `SQLiteInputVFS.h` and into
  `SQLiteInputVFS.cpp`, with the global state in an anonymous namespace and
  the registration guarded by `std::call_once`, so the VFS is registered
  exactly once even under concurrent reads.
- Pass the `arrow::io::RandomAccessFile` pointer to the VFS as a portable
  lowercase hex string via `getHexUIntLowercase`/`unhexUInt`, instead of
  relying on platform-dependent pointer formatting with `operator<<`.
- In `SQLiteOutputFormat`, throw a clear exception when the output is not a
  regular file: only writing to files is supported, and `fstat` is used to
  rule out pipes, sockets and terminals.
- Build the rows in `SQLiteOutputFormat` with a prepared statement and
  parameter binding instead of concatenating SQL text, which avoids quoting
  and escaping bugs, and wrap the inserts in a single transaction.
- Handle SQL `NULL` on input by checking `sqlite3_column_type` and reading
  the exact byte length with `sqlite3_column_bytes`.

Add a stateless test that round-trips a database through the output and
input formats, verifies the produced file is a real SQLite database, reads
a database created by the `sqlite3` CLI by table name, and checks that
writing to a non-file is rejected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ClickHouse type names such as `Nullable(UInt64)` are not valid SQLite type
names, so creating the `result` table with them failed with a syntax error.
SQLite is dynamically typed, so the columns are now declared without a type:
this gives them no affinity and the values bound as text are stored as is,
which round-trips faithfully.

Also fix the `sqlite3` CLI quoting in the stateless test and add the
reference file. Verified end to end with a freshly built binary:
write via the output format, read back via the input format (including
NULLs), cross-check with the `sqlite3` CLI, read by table name, and reject
writing to a non-file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… in the test

Extend `04309_sqlite_format_input_output` with two cases:

* Reading a table whose name contains a double quote, verifying the name is
  correctly quoted as an SQLite identifier.
* Providing a structure with the wrong number of columns, verifying the input
  format rejects it with `INCORRECT_NUMBER_OF_COLUMNS`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Pushed f355996d8cf: merged latest master, fixed the Build (arm_tidy) failure, and addressed the open review threads.

Build (arm_tidy) fix

The job failed on a single clang-tidy error in SQLiteOutputFormat.cpp:

error: uninitialized record type: 'file_stat' [cppcoreguidelines-pro-type-member-init]

Initialized the local as struct stat file_stat{}.

Review threads addressed

  • Read errors must not become a clean EOFreadRow now treats only SQLITE_DONE as the end of the result set; any other sqlite3_step status (corrupt/truncated database, VFS I/O error, SQLITE_BUSY) throws SQLITE_ENGINE_ERROR.
  • Column-count mismatchreadPrefix rejects a structure whose width differs from sqlite3_column_count with INCORRECT_NUMBER_OF_COLUMNS.
  • Unchecked sqlite3_bind_* — every bind return value is checked; a non-SQLITE_OK status throws immediately.
  • Unescaped table name — the table name is quoted as an SQLite identifier; the quoting helper moved to SQLiteUtils (quoteSQLiteIdentifier) and is shared with the output format.
  • Docs frontmatter — left as is: format pages under docs/en/interfaces/formats/ use a different frontmatter convention (no sidebar_label/sidebar_position), and SQLite.md matches its siblings.

The SQLiteSource reuse suggestion is left open as a follow-up, per the earlier note.

04309_sqlite_format_input_output was extended with cases for identifier escaping and column-count validation, and passes locally. Built clickhouse clean on the local build.

Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp Outdated
Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp Outdated
pull Bot pushed a commit to sowelswl/ClickHouse that referenced this pull request Jun 7, 2026
The test asserts that ProfileEvents['UserThrottlerSleepMicroseconds'] is
greater than half of the target query duration. The throttler only sleeps
when the natural read rate exceeds the configured throttle limit; when the
natural rate drops close to the limit the token bucket never goes negative
and the throttler skips its sleep path (see `Throttler::throttle` in
`src/Common/Throttler.cpp`).

PR ClickHouse#103586 added `no-random-settings` to remove S3-prefetch settings as one
source of slow natural rate, but contention on the `Stateless tests
(amd_tsan, parallel, 2/2)` runner still occasionally drives the natural S3
read rate close to or below the previous 1 MB/s limit. Post-merge CIDB:
`pull_request_number != 0 AND check_start_time > 2026-06-05` shows 4
sightings (PRs ClickHouse#106222, ClickHouse#102039, ClickHouse#49966, ClickHouse#106184) all on amd_tsan parallel
2/2 with the `read 1 1 0` shape (duration ok, bytes ok, sleep below
threshold). All four PRs already include the `no-random-settings` tag, so
the remaining flakiness is contention-driven, not random-settings driven.

Drop the throttle limit and dataset 5x so the throttle is well below any
plausible natural rate (200 KB/s vs ~0.9 MB/s observed worst case = 4.5x
safety margin). Test wall-clock stays at ~8s. Local 10/10 runs against an
S3 disk produced sleep_us between 16.8s and 17.3s, comfortably above the
3.5s threshold.

Closes: ClickHouse#103422
Related: ClickHouse#103586
Related: ClickHouse#106184

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
alexey-milovidov and others added 3 commits June 9, 2026 09:45
The input format used `arrow::io::RandomAccessFile` (via `asArrowFile`)
purely as a seekable byte source for the read-only SQLite VFS. This coupled
the `SQLite` format to Arrow, which is disabled on some platforms, so the
build failed on `amd_freebsd`, `riscv64` and `loongarch64` with
`'arrow/io/interfaces.h' file not found`.

Replace the Arrow dependency with ClickHouse's own `SeekableReadBuffer`: the
VFS now reads through `seek`/`read` and reports the size cached in a small
`SQLiteReadSource`. When the input buffer is seekable and its size is known,
the database is read with true random access; otherwise (a pipe, a stream of
unknown length) it is loaded into a `ReadBufferFromOwnString`.

Also fix two NULL-handling issues in `readRow`:
- A SQLite `TEXT` value equal to the textual null marker (e.g. `NULL` or
  `ᴺᵁᴸᴸ`) in a `Nullable` column was mistaken for SQL `NULL`, because
  `deserializeWholeText` on a `Nullable` serialization treats the marker as
  null. Known-non-null values are now deserialized through the nested
  (non-nullable) serialization.
- SQL `NULL` in a non-nullable column silently became the type default,
  ignoring `input_format_null_as_default`. It now throws `INCORRECT_DATA`
  when the setting is disabled, matching other input formats.

The stateless test was renumbered to `04326` (the `04309` prefix collided
with tests merged from `master`) and extended with both NULL cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`03251_insert_sparse_all_formats` and `02187_async_inserts_all_formats`
iterate over every format that is both input and output and round-trip data
through an HTTP socket / async-insert pipe. The `SQLite` format cannot
participate: its output requires a regular, seekable file (writing to a pipe
or socket throws `NOT_IMPLEMENTED`) and reading requires random access, so it
cannot stream. Add `SQLite` to the existing exclusion lists, next to the
other formats that cannot do a generic streaming round-trip.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Pushed 9de7b0108b8 and bb45d2f5712: merged latest master and fixed the remaining CI failures.

Build on amd_freebsd, riscv64, loongarch64 ('arrow/io/interfaces.h' file not found)

The input format used arrow::io::RandomAccessFile (via asArrowFile) only as a seekable byte source for the read-only SQLite VFS, which coupled the SQLite format to Arrow — disabled on those platforms. Removed the Arrow dependency entirely and replaced it with ClickHouse's own SeekableReadBuffer: the VFS now reads via seek/read and reports the size cached in a small SQLiteReadSource. A seekable input with a known size is read with true random access; a non-seekable input (pipe, stream of unknown length) is loaded into a ReadBufferFromOwnString. The output format was already Arrow-free.

03251_insert_sparse_all_formats and 02187_async_inserts_all_formats

Both iterate over every input+output format and round-trip through an HTTP socket / async-insert pipe. SQLite cannot stream (its output needs a regular seekable file, its input needs random access), so it is now in the exclusion lists next to the other non-streamable formats.

Review threads

  • Text value equal to the null marker — known-non-null values are now deserialized through the nested (non-nullable) serialization, so a SQLite TEXT NULL/ᴺᵁᴸᴸ in a Nullable column is no longer read as SQL NULL.
  • input_format_null_as_default — SQL NULL in a non-nullable column now throws INCORRECT_DATA when the setting is disabled, instead of silently inserting the default.

The SQLiteSource-reuse suggestion remains a deferred follow-up.

The stateless test was renumbered 0430904326 (the 04309 prefix collided with tests merged from master) and extended with both NULL cases. Built clickhouse clean on ARM and the test passes locally.

Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp Outdated
Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp
Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp Outdated
alexey-milovidov and others added 2 commits June 11, 2026 06:59
The style check flagged a `catch (...)` in `memRead` that silently swallowed
exceptions. Save the exception into `SQLiteReadSource` instead, so the caller
of the SQLite API can rethrow the original error (e.g. a network read failure)
rather than reporting a generic SQLite I/O error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Rethrow the exception saved by the VFS read callback in the format's error
  paths, so the original error is reported instead of a generic SQLite one.
- Handle `LowCardinality(Nullable)` columns: use
  `isNullableOrLowCardinalityNullable` instead of `isNullable`, insert SQL NULL
  as NULL, and deserialize known-non-NULL values with the non-nullable nested
  serialization (a text value equal to the textual null marker is not mistaken
  for SQL NULL).
- When `input_format_null_as_default` consumes a NULL for a non-nullable
  column, mark the column as not read in `RowReadExtension`, so that
  `AddingDefaultsTransform` computes the table DEFAULT expression for it.
- Honor `input_format_allow_seeks` (`FormatSettings::seekable_read`): when
  disabled, buffer the whole database in memory instead of seeking.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Pushed two commits:

  • c77e6bf9f72 fixes the failed Style check (catch_all): the catch (...) in the VFS read callback now saves the exception into SQLiteReadSource instead of silently swallowing it, and the input format rethrows the original error (e.g. a network read failure) instead of a generic SQLite I/O error.
  • 4f13b311daf addresses the three remaining review comments: LowCardinality(Nullable) support, marking input_format_null_as_default columns as not read so AddingDefaultsTransform applies table DEFAULT expressions, and honoring input_format_allow_seeks. All with new test cases in 04326_sqlite_format_input_output.

The extended test passes against a local build.

Comment thread src/Processors/Formats/Impl/SQLiteInputFormat.cpp
alexey-milovidov and others added 2 commits June 12, 2026 20:13
`sqlite3_open_v2` can leave the database handle non-null even when it
returns an error, and that handle still has to be released with
`sqlite3_close_v2`. Previously the pointer was wrapped into the owning
`db` only after the status check, so every failed open leaked the
connection object; malformed input files could hit this path
repeatedly. Wrap the handle into `db` immediately after the call so the
connection is released on every path. The same fix is applied in both
`SQLiteInputFormat` and `SQLiteOutputFormat`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Merged master to resolve the conflict (the branch is now MERGEABLE), and addressed the connection-leak review: sqlite3_open_v2 can leave the handle non-null on error, so it is now wrapped into the owning db (with the sqlite3_close_v2 deleter) immediately after the call in both SQLiteInputFormat and SQLiteOutputFormat (6518357e6d4). The three SQLite format translation units compile cleanly against current master.

@al13n321, your earlier CHANGES_REQUESTED was about FastTest failing in a build without SQLite. Everything is now wrapped in #if USE_SQLITE (sources, headers, and the registrations in registerFormats.cpp) and CI is green — could you take another look so the review can be cleared? The SQLiteSource reuse suggestion is tracked as a follow-up.

Comment thread src/Processors/Formats/Impl/SQLiteOutputFormat.cpp
alexey-milovidov and others added 3 commits June 13, 2026 23:54
`SQLiteOutputFormat::writePrefix` is not just a prefix: it opens the SQLite
connection, creates the `result` table, starts the transaction and prepares the
insert statement. Because the format was registered without
`markFormatHasNoAppendSupport`, `StorageFile` treated an existing non-empty
`.db` file as appendable and called `doNotWritePrefix` before writing, so the
next `consume` ran with `db`/`insert_stmt` unset and a second
`INSERT INTO FUNCTION file(..., SQLite)` against the same file failed in a
confusing way.

Mark `SQLite` as not append-supported, like `Parquet`/`ORC`, so `StorageFile`
produces the standard `CANNOT_APPEND_TO_FILE` error. Add a regression to
`04326_sqlite_format_input_output` with two inserts into the same file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alexey-milovidov

Copy link
Copy Markdown
Member

Pushed c95faf3a4da:

  • Resolved the conflict with master. Merged the latest master into the branch; the PR is MERGEABLE again. The only net changes versus master are the SQLite format files plus the SQLite exclusions in 02187_async_inserts_all_formats / 03251_insert_sparse_all_formats.
  • Disallow appends to an existing .db file (f8769328c59, resolves the review on SQLiteOutputFormat). The whole SQLite database (table, transaction, prepared statement) is set up in writePrefix, so an append that skips the prefix would leave the connection unset. SQLite is now registered with markFormatHasNoAppendSupport, so StorageFile produces the standard CANNOT_APPEND_TO_FILE error, the same as Parquet/ORC. Added a regression to 04326_sqlite_format_input_output (two inserts into the same file); the full test passes against a fresh ARM build.
  • The previous Build (arm_tidy) failure was unrelated — a cppcoreguidelines-init-variables error in src/Storages/TimeSeries/PrometheusQueryToSQL/applyHistogramQuantile.cpp (an uninitialized out_of_range_value), which has since been fixed on master and is now included via the merge.

The SQLiteSource reuse suggestion remains tracked as a follow-up.

@clickhouse-gh

clickhouse-gh Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

LLVM Coverage Report

Metric Baseline Current Δ
Lines 85.10% 85.10% +0.00%
Functions 92.30% 92.40% +0.10%
Branches 77.30% 77.40% +0.10%

Changed lines: Changed C/C++ lines covered by tests: 231/313 (73.80%) | Lost baseline coverage: none · Uncovered code

Full report · Diff report

@alexey-milovidov

Copy link
Copy Markdown
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors comp-formats Input/output formats (CSV/JSON/Parquet/ORC/Arrow/Protobuf/etc.). pr-feature Pull request with new product feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants