iframe-proxy

lnkuiper · 2025-12-09T15:21:37Z

Fixes https://github.com/duckdblabs/duckdb-internal/issues/6807#issuecomment-3631452424

Just some basic functionality, but it could be improved using stats in the future.

…httpfs into grow_httpfs_read_buffer

carlopi

This looks great.

I have not tested this locally yet. Would it make sense to add a test? Say something like:

statement ok
CALL enable_logging('HTTP');

--- or from elsewhere in the internet
statement ok
FROM read_json('https://raw.githubusercontent.com/duckdb/duckdb/main/data/...');

query I
SELECT count(*) FROM duckdb_logs_parsed('HTTP');
----
7?

carlopi · 2025-12-10T08:39:55Z

carlopi · 2025-12-10T15:06:42Z

Both to have this enabled on nightly, and to more widely test it

Both to have this enabled on nightly, and to more widely test it in the context of main repo CI.

Issue duckdb/duckdb#20136: Secondary IGNORE NULLS (duckdb/duckdb#20153) Bumping httpfs to include duckdb/duckdb-httpfs#174 (duckdb/duckdb#20145)

Issue duckdb/duckdb#20136: Secondary IGNORE NULLS (duckdb/duckdb#20153) Bumping httpfs to include duckdb/duckdb-httpfs#174 (duckdb/duckdb#20145) Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

Both to have this enabled on nightly, and to more widely test it

* Use separated tarballs for duckdb and py-duckdb. The tarball of duckdb does not contain py-duckdb. Changelog: 1.4.4: What's Changed Cast Fix: Correctly handle negative exponent with a number with a decimal in VARCHAR -> INTEGER cast by @Mytherin in #20098 [Storage] Fix NULL filter check for constant segments by @Tishj in #20103 fixing staged upload for install.duckdb.org - hopefully by @hannes in #20104 Remove undefined loop by @artjomPlaunov in #20105 [Test] Adjust concurrent attach-detach test to expect write-write conflict by @taniabogatsch in #20108 Fix ALIAS function in filter pushdown to preserve column aliases(#20008) by @henry8128 in #20106 Headers missing from http logs by @samansmink in #20030 Bumping httpfs to include duckdb/duckdb-httpfs#174 by @carlopi in #20145 Issue #20136: Secondary IGNORE NULLS by @hawkfish in #20153 Fix minio nightly tests by @c-herrewijn in #20132 Fix use-after-free in mode aggregate Combine function by @victor-ab in #20146 Only pushdown varchar if the arrow type is not a string view by @evertlammerts in #20165 Quote filters in adbc_get_objects by @evertlammerts in #20172 Issue #20156: Streaming Window Unions by @hawkfish in #20191 MergeInto: correctly clean-up buffer and handle non trivial GetData by @carlopi in #20163 Fix extentension-ci-tools to latest release version in the v1.4-andium branch by @carlopi in #20227 [Chore] Clean up CI nightly run to prevent time outs by @taniabogatsch in #20150 Bump Julia to v1.4.3 by @maiadegraaf in #20248 Internal #6881: 2025c Time Zones by @hawkfish in #20258 Split statements by semicolon by @Dtenwolde in #20174 [chore] Remove numeric_cast.hpp import and other tidy stuff by @taniabogatsch in #20278 [chore] Reduce latency of VARCHAR index creation test by @taniabogatsch in #20277 Reuse correct table-level metadata during checkpoints by @ywelsch in #20267 Fix view resolution not being stable if the referenced table lives in a different schema by @Tishj in #20260 ConstantOrNullFunction input validity mask overwrite bugfix by @artjomPlaunov in #20283 dbgen: use TaskExecutor framework by @Mytherin in #20284 fix(adbc): return error when setting an empty sql query by @gishor in #20071 Bump iceberg, and add wasm platforms! by @carlopi in #20205 _extension_distribution: Pin to v1.4-andium branch of extension-ci-tools by @carlopi in #20294 Optimize prepared statement parameter lookups by @EtgarDev in #20252 Update VectorType in ComputePartitionIndices by @lnkuiper in #20343 Add some defensive programming in RadixPartitionedHashTable::Combine and RadixPartitionedHashTable::Finalize by @lnkuiper in #20342 Increase reserved size for paths in SetPathsInternal by @Flogex in #20340 Use UTF-16 console output in Windows shell (1.4) by @staticlibs in #20339 Force repartitioning in RadixPartitionedHashTable::Combine by @lnkuiper in #20357 Fixup comparison to wrong iterator on abdc's driver by @carlopi in #20360 [chore] dsdgen generation signed integer overflow fix by @taniabogatsch in #20279 Give preference do variables set in config by @pdet in #20396 Add QueryContext also to Write(void *buffer, idx_t nr_bytes), so that BytesWritten are updated also there by @carlopi in #20393 Fix Issue #20233: fix function chain in qualify by @ArNine in #20302 Backport client data cleanup by @taniabogatsch in #20403 Add v1.4.4 to Storage Version by @maiadegraaf in #20404 Parquet Reader: Ignore invalid UTF8 in string stats, instead of throwing an error by @Mytherin in #20405 Don't add a semicolon to final query when splitting statements by @Dtenwolde in #20401 Fixup BRANCHES_TO_BE_CACHED, vars are not available on PRs, so env it is by @carlopi in #20411 [Fix] Defensive infinite loop guard and UTF-8 check in C API by @taniabogatsch in #20348 Add ducklake, httpfs and iceberg tests so they are run in CI by @carlopi in #20319 [Stats] date_trunc stat propagation fix by @Tishj in #20421 Refactor allowed path sanitize by @hannes in #20346 Issue #20413: ASOF SEMI/ANTI Bindings by @hawkfish in #20433 Fix #20410: fix for RIGHT SEMI/ANTI - cannot fully label chain as found if there are non-equality predicates present in the join condition by @Mytherin in #20435 [Fix] Misaligned size in ART prefix count by @taniabogatsch in #20455 Wrap ccache in own action by @carlopi in #20466 [Chore] Add ORDER BY to AS OF test by @taniabogatsch in #20465 Fix segfault in hive partitioning with NULL values by @Schwarf in #20468 Nightly test encryption fixes by @ccfelius in #20461 [C API] Fix error data creation by @taniabogatsch in #20451 Skip concurrent_encrypted_attach due to race in autoloading httpfs by @carlopi in #20471 CI fixup: Comparisons needs to be done via strings (since input via workflow_call is a string) by @carlopi in #20464 Add unity_catalog to extensions built by duckdb/duckdb by @carlopi in #20445 Invert the setup of ccache and cleanup_runner by @carlopi in #20429 bump spatial for andium by @Maxxen in #20479 Reclaim disk space on MacOS runners by @lnkuiper in #20493 Adding secure clear functions by @ccfelius in #20285 rewrite unaligned scan by @artjomPlaunov in #20474 Issue #20470: TIMESTAMPTZ to DATE by @hawkfish in #20498 Reset cached dictionaries in TRY expression by @Maxxen in #20452 [Copy] Fix #20324 partition_by option binding by @Tishj in #20509 Bump multiple extensions by @maiadegraaf in #20504 update azure ref for v1.4-andium by @benfleis in #20506 [Fix] Directly retrieve the logical column index during MERGE INTO binding by @taniabogatsch in #20503 Sanitize pragmas by @samansmink in #20514 Bump httpfs, ducklake and postgres by @carlopi in #20518 Revert vortex bump for v1.4.4 by @taniabogatsch in #20527 Bump VSS by @taniabogatsch in #20542 Bump Excel by @maiadegraaf in #20540 Fix Issue #20233: Fix function chain in having and merge to v1.4 by @ArNine in #20532 bump iceberg by @Tmonster in #20549 Bump Iceberg v1.4 by @Tmonster in #20604 Fix KeyValueSecretReader init by @NiclasHaderer in #20620 Fail fast extensions by @carlopi in #20605 1.4.3: What's Changed implement proper equals for table functions by @Maxxen in #19722 Fix behavior for HAVING clause without a GROUP BY by @Tishj in #19739 Remove httpfs patches and bump by @carlopi in #19763 Correctly use a lock when accessing the EncryptionKeyManager by @Mytherin in #19772 Bump iceberg to now default available extension by @carlopi in #19764 Extract all column bindings of json each function by @Tmonster in #19766 constraint violation bug fix by @artjomPlaunov in #19776 Avoid binding macro arguments for untyped macros by @lnkuiper in #19779 It should probably be ${EXTENSION_CONFIG_BUILD} instead of EXTENSION_CONFIG_BUILD by @hannes in #19694 Fix #19575: Fix illegal utf8 character by @flashmouse in #19699 chore: bump vortex to 0.56.0 by @0ax1 in #19815 PositionalScan: handle also HAVE_MORE_OUTPUT + empty chunk via iteration by @carlopi in #19824 AttachDatabase: first set whether file is remote, then do the rest by @carlopi in #19826 Add CacheBehavior::AUTOMATIC to DBInstanceCache that automatically does the right thing™ by @Mytherin in #19841 Reuse metadata even in the presence of deletes by @ywelsch in #19823 DuckIndexScanState::TableScanFunc, split into 2 explicit phases by @carlopi in #19838 Bump httpfs and iceberg by @carlopi in #19859 Keep cte_root alive while binding materialized CTEs in MERGE INTO children by @Mytherin in #19863 CI Cleanup (#19840) by @yan-alex in #19857 Fix #19517: preserve relation name for table-qualified star LIKE expression by @henry8128 in #19887 avoid underflow/"inf" loop while reporting unittest summary (backport… by @benfleis in #19900 Fixes incorrect handing of APPROX_QUANTILE TIME by @Damon07 in #19891 Add v1.4.3 to Storage Version by @maiadegraaf in #19907 clean up tmp files while building extensions by @c-herrewijn in #19908 Bump: ducklake, mysql_scanner by @maiadegraaf in #19910 remove large limit optimization whenever there is a filter by @guillesd in #19911 free disk space in Upload Extensions job by @c-herrewijn in #19912 [Compression] Prevent overriding COMPRESSION_EMPTY with COMPRESSION_CONSTANT by @Tishj in #19913 Make make tidy-check-diff compare against base branch, instead of always comparing against origin/main by @Mytherin in #19917 More testing for appender and attach-detach by @taniabogatsch in #19708 Script to release a extension manually by @samansmink in #19881 Assert that result types match with column data collection when fetching data by @pdet in #19750 No longer auto-enable profiling on DEBUG mode by @maiadegraaf in #19931 Fix optimizer incorrectly remove ORDER BY clause from aggregates by @tianjq16 in #19925 Backport revert append fixes by @Mytherin in #19941 Bump: spatial by @Maxxen in #19943 Issue #19916: WASM Time Zones by @hawkfish in #19918 Fix correlated column binding in ConstantBinder by @d-justen in #19945 [chore] Increase slow threshold by @taniabogatsch in #19965 [Parquet] Fix prepared copy option parameter by @Tishj in #19966 [TestConfig] Fix verify_fetch_row config, reduce duplication in skip_tests by @Tishj in #19967 remove sha from artifacts by @c-herrewijn in #19957 Unbound index binding with context by @artjomPlaunov in #19953 [Fix] Bug in FetchRow after update on indexed table with dict_fsst compression by @taniabogatsch in #19970 Null assertion on denormalized_table argument by @Dtenwolde in #19947 [Art][Wal]Unbound index allocations by @artjomPlaunov in #19901 No sampling over 281TB by @Tmonster in #19978 Take in consideration if the sniffer used unstrictness while selecting candidates by @pdet in #20005 [CSV] Avoid throwing unnecessary errors on strict mode by @pdet in #20007 [Compression] Fix an issue with NULL updates to a column compressed with DICT_FSST by @Tishj in #20009 Bump: delta, ducklake, iceberg by @maiadegraaf in #20012 add unity_catalog, update delta by @samansmink in #20019 Bump spatial by @staticlibs in #20020 Retag #19821 to v1.4 - Use PLAIN_DICTIONARY for Parquet version 1 by @pdet in #20024 Fix INSERT OR REPLACE BY NAME with partial columns(#19845) by @henry8128 in #19989 Bump MySQL scanner by @staticlibs in #20025 Windows must sample less by @Tmonster in #20018 Issue #20015: Streaming Window Sequence by @hawkfish in #20027 bump iceberg by @Tmonster in #20032 Fix #20014: correctly use numeric_limits::min for NumericLimits::Min so that stats are initialized to -infinity for floating points by @Mytherin in #20039 Enable windows_arm64 arch for main extensions by @staticlibs in #20004 Fix mark join decorrelation by @kryonix in #20033 propagate gate status in Node4::DeleteChild by @artjomPlaunov in #20044 Bump: httpfs by @samansmink in #20036 Bump ducklake by @pdet in #20054 bump azure to v1.4.3 by @benfleis in #20057 Bump extensions by @samansmink in #20055 Fix unnecessary dependent join rewrite by @kryonix in #20048 Version tag for local files for robust external file cache validation by @lnkuiper in #20058 backport runner cleanup action by @hannes in #20085 1.4.2: What's Changed bump duckdb-azure ref for 1.4.1 by @benfleis in #19275 Fix regex optimization to remove flags when converting to contains by @mlafeldt in #19290 feat: Fix length of sort keys by @krlmlr in #19260 Pass lambda bindings to next binder by @Dtenwolde in #19144 bump Julia to v1.4.1 by @c-herrewijn in #19306 Use cross-compilation for static libs on OSX by @hannes in #19304 Allow multiple read-only attaches to the same database across database instances by @Mytherin in #19319 hugeint_t fixes by @carlopi in #19318 Use CMAKE__COMPILER_LAUNCHER by @evertlammerts in #19326 Add settings field to test config by @Mytherin in #19330 Keep track of which database managers have which databases attached in the DatabaseFilePathManager by @Mytherin in #19338 Bugfixes by @lnkuiper in #19329 skip several ci jobs on prs that only bump extensions by @samansmink in #19249 [C API] bind_value out of range fix by @taniabogatsch in #19348 Destroy TaskNotifier prior to calling FinishTask by @Mytherin in #19373 Remove zip bombs by @Mytherin in #19380 Avoid calling shared_from_this() and instead use the passed in ClientContext in buffered data by @Mytherin in #19379 add test tag support [vfs integration tests p1] by @benfleis in #19331 Throw if non-VARCHAR key is passed to json_object by @lnkuiper in #19365 BUGFIX: Silent failure to write row groups with large lists by @J-Meyers in #19376 Fixes for CTE (de)serialization compatibility with older versions by @Mytherin in #19393 Fix #18139: correctly initialize flush size in MemoryStream, and re-use writer states by @Mytherin in #19398 Fixup Kalman fiter: bound it's (0, scale_factor), not (0, 1) by @carlopi in #19395 Fix StringDecompress for hugeint_t values by @ywelsch in #19403 Fix #19211: make INSERT OR IGNORE correctly handle multiple constraints by @Mytherin in #19409 Issue #19386: ICU TZDEFAULT by @hawkfish in #19413 Fix issue in MetadataManager triggered when doing concurrent reads while checkpointing, and rework concurrent attach / detach test by @Mytherin in #19424 ADBC fix: escape schema, table and column identifiers by @evertlammerts in #19407 v1.4: Only serialize CTE nodes when MATERIALIZED is specified by @Mytherin in #19420 Add forwards compatibility tests to CI by @Mytherin in #19432 Internal #6168: Unsupported Correlated Binds by @hawkfish in #19431 Don't pull up filters through DISTINCT ON and make enum casts in Parquet safe (bugfixes) by @lnkuiper in #19406 Get table bindings for all operators under a logical get if the logical get function is an unnest by @Tmonster in #19467 WASM #1897: UTC Offset Support by @hawkfish in #19464 Add tests for nested lambda bindings by @Dtenwolde in #19453 [Compression] Fix issue in ZSTD decompression related to mis-interpreted segment offsets by @Tishj in #19475 vfs integration tests p2 - update (env) vars and tests by @benfleis in #19428 CI Fix: clear benchmark cache between runs by @Mytherin in #19508 Make DatabaseInstance::log_manager a unique_ptr by @Flogex in #19471 Support non-standard NULL in Parquet again by @Mytherin in #19523 Bump: inet by @samansmink in #19526 Bump multiple extensions by @samansmink in #19522 add upcoming patch release to internal versions by @samansmink in #19525 Add test that either 'latest' or 'vX.Y.Z' are supported STORAGE_VERSIONs by @carlopi in #19527 Moving staging to cf and uploading to install bucket by @hannes in #19539 Creating separate OSX cli binaries for each arch by @hannes in #19538 [Dev] Disable the use of ZSTD if the block_manager is the InMemoryBlockManager by @Tishj in #19543 Follow up to staging move by @hannes in #19551 Bugfixes: Parquet JSON+DELTA_LENGTH_BYTE_ARRAY and sorting iterator by @lnkuiper in #19556 [ported from main] Fix bug initializing std::vector for column names by @evertlammerts in #19555 Disable jemalloc on BSD by @lnkuiper in #19560 Fix race condition between Append and Scan by @Captain32 in #19571 Release relevant tests to still be run on all builds by @carlopi in #19559 fix inconsistent behavior in remote read_file/blob, and prevent union… by @Maxxen in #19531 [v1.4-andium] Add Profiler output to logger interface by @carlopi in #19572 Fix edge case in uncompressed validity scan with offset and fix off-by-one in ArrayColumnData::Select by @Maxxen in #19567 Skip compiling remote optimizer test when TSAN Is enabled by @Mytherin in #19590 Wal index deletes by @artjomPlaunov in #19477 add vortex external extension by @samansmink in #19580 Avoid eagerly resolving the next on-disk pointer in the MetadataReader, as that pointer might not always be valid by @Mytherin in #19588 [DevEx] Improve error message when FROM clause is omitted by @Tishj in #18995 bump iceberg by @Tmonster in #19618 Improve error message around compression type deprecation/availability checks by @Tishj in #19619 Increase cast-cost of old-style implicit cast to string by @Maxxen in #19621 Try to prevent overshooting of FILE_SIZE_BYTES by pre-emptively increasing bytes written in Parquet writer by @lnkuiper in #19622 Bump: spatial by @Maxxen in #19620 Detect invalid merge into action and throw exception by @Mytherin in #19636 Fix #19455: correctly extract root table in merge into when running ajoin that contains single-sided predicates that are transformed into filters by @Mytherin in #19637 Remove FlushAll from DETACH by @lnkuiper in #19644 Bump MySQL scanner by @staticlibs in #19643 Enable running all extensions tests as part of the build step by @carlopi in #19631 Always remember extra_metadata_blocks when checkpointing by @ywelsch in #19639 duckdb_logs_parsed to do case-insensitive matching by @carlopi in #19669 Categorize ParseLogMessage as CAN_THROW_RUNTIME_ERROR LVM by @carlopi in #19668 Log total probe matches in hash join by @lnkuiper in #19683 Fix InsertRelation on attached database by @evertlammerts in #19583 Add request timing to HTTP log by @samansmink in #19691 Add missing query location tohttpfs by @samansmink in #19715 Fix #19355: correctly resolve subquery in MERGE INTO action condition by @Mytherin in #19720 Fix #19700: correctly sort output selection vector in nested selection operations by @Mytherin in #19718 Bump httpfs and resume testing on Windows by @carlopi in #19714 [Dev] Fix assertion failure for empty ColumnData serialization by @Tishj in #19713 Bump the Postgres scanner extension by @Mytherin in #19730 Add explicit Initialize(HTTPParam&) method to HTTPClient by @carlopi in #19723 Logs to be case-insensitive also at enable_logging callsite by @carlopi in #19734 Fix CVE GHSA-vmp8-hg63-v2hp crypto issues by @samansmink in #19716 1.4.1: What's Changed Fix attach to right DB when using DuckLake by @pdet in #19011 set default value of MAIN_BRANCH_VERSIONING to false by @c-herrewijn in #19014 ComplexJSON: parse all valid JSON correctly by @Mytherin in #19024 Issue #19016: ICU Offset Parsing by @hawkfish in #19029 Throw if we detect a quoted new line with the null padding set in parallel mode by @pdet in #19012 Bump iceberg & ducklake by @carlopi in #19037 Build Fix: unordered_map<enum class is not supported in all compilers, use map< instead by @Mytherin in #19046 Disable emitting versioned libraries by default by @Mytherin in #19047 Re-add aliased settings to duckdb_settings() view, and some fixes for aliased settings by @Mytherin in #19050 Fix threading issues in metadata manager, and expand concurrent attach / detach fuzz test by @Mytherin in #19054 Correctly re-align all child column segments of the ColumnData on Deserialize, and add logging to checkpoints by @Mytherin in #19055 [unittest] Fixes so that '{BASE_TEST_NAME}' can be used within --on-new-connection by @carlopi in #19056 add a bunch of expected error messages to old macro tests and fix iss… by @lnkuiper in #19042 Always execute cast and try_cast if they are not invertible by @DinosL in #19010 Switching core extension upload to dedicated credentials by @hannes in #19061 Include BeginQuery in latency metric by @taniabogatsch in #19064 [Dev] Bit of code cleanup in (parquet) ColumnWriter by @Tishj in #19063 Add config: one_schema_per_test.json by @carlopi in #19059 Change bucket name for core extensions by @hannes in #19083 Moved test data into testing dir by @NiclasHaderer in #19102 Bump httpfs by @carlopi in #19104 Fix example syntax in variant_typeof() function by @krlmlr in #18977 Avoid throwing on unset extension setting by @Mytherin in #19117 Fix internal issue 5975 by @lnkuiper in #19101 Properly initialize StringStats in Parquet reader by @lnkuiper in #19139 Remove HTTPFS tests and setup scripts by @Mytherin in #19140 Validate JSON in Parquet reader by @lnkuiper in #19143 Fix bug in merge into when condition is in parenthesis by @pdet in #19137 Allow implicit casts from JSON[] to JSON again by @lnkuiper in #19141 [ci] Change logic for saving caches: Github variable that decides what gets cached by @carlopi in #19150 Fix handling of quotes in ToString() of search_path in current_setting by @Mytherin in #19162 Delay throwing NotImplementedException in ExpressionBinder by @lnkuiper in #19153 Issue #18303: AsOf NLJ Nulls by @hawkfish in #19173 HTTPUtil: response might be null, perform check by @carlopi in #19179 Handle malformed schema index in Parquet reader by @Mytherin in #19191 ATTACH IF NOT EXISTS: avoid looping waiting for DETACH to finish, wait only for an ATTACH operation to finish by @Mytherin in #19193 Implement duckdb_connection_count table function by @taniabogatsch in #19187 Disable ALP for non-default block sizes by @taniabogatsch in #19197 Check for unresolved parameters when binding CREATE MACRO ... AS TABLE by @lnkuiper in #19196 https://duckdb-blobs.s3.amazonaws.com -> https://blobs.duckdb.org by @carlopi in #19206 [chore] Attempt at restoring workflow for MinGW Static libs by @carlopi in #19205 Simple no default region return 301 response by @Tmonster in #19087 [Fix] Correctly reset the gate status during ART merging by @taniabogatsch in #19204 build spatial extension for mingw by @c-herrewijn in #19207 Fixup templated version of TryGetSecretKeyOrSetting by @carlopi in #19218 Bump: delta by @samansmink in #19220 Autoloading helper file system: allow either autoloading or proper errors in more file operations by @carlopi in #19198 Eargerly destroy sort buffers in Window by @lnkuiper in #19224 [Transaction] Delete and drop of a table can now happen in the same transaction without error by @Tishj in #18918 PRAGMA's MissingEntry: Suggest CALL might be an option by @carlopi in #18815 Bump: aws, ducklake, iceberg by @samansmink in #19228 Issue 18603 by @Tmonster in #19227 Bump DuckLake to latest of V1.4 by @pdet in #19237 Bump mysql and sqlite by @staticlibs in #19240 Don't write parquet-native GEOMETRY by default, add option to control GeoParquet version by @Maxxen in #19244 When executing a relation, generate a query to set if it is not a query relation by @Mytherin in #19234 add support for writing geoparquet with v2 metadata too by @Maxxen in #19246 Bump: iceberg by @samansmink in #19250 Bump: avro, httpfs by @samansmink in #19248 bump duckdb-azure ref for 1.4.1 by @benfleis in #19275 1.4.0: What's Changed Python package devexp improvements by @evertlammerts in #17483 change exception type to not be an internal exception by @samansmink in #17551 Remove redundant code path in the ConflictManager by @taniabogatsch in #17562 Add support for ToSqlString for union types by @wmTJc9IK0Q in #17513 Update function descriptions and examples by @c-herrewijn in #17132 Move query profiler's EndQuery after commit/rollback by @taniabogatsch in #17595 fix extension troubleshooting link by @simon0191 in #17616 C API tidying by @taniabogatsch in #17623 bump DuckDB_jll to v1.3.0 by @c-herrewijn in #17677 Add rowsort in generate_series test #43 by @jeewonhh in #17675 [C API] Expose duckdb_scalar_function_bind_get_extra_info by @taniabogatsch in #17666 Enable profiling output for all operator types by @taniabogatsch in #17665 Output hashes in unittest and fix order by @niykko in #17664 New Sorting Implementation by @lnkuiper in #17584 Merge v1.3-ossivalis into main by @Mytherin in #17690 Issue #17040: FILL Window Function by @hawkfish in #17686 ClientBufferManager wrapper to access the client context in the buffer manager by @taniabogatsch in #17699 Revert "set default for MAIN_BRANCH_VERSIONING to false" by @carlopi in #17708 Sorting followup by @lnkuiper in #17717 Correctly setting the delim offset by @Damon07 in #17716 fix linux extension ci by @samansmink in #17720 Aggregation performance by @lnkuiper in #17718 Fix windows-2025 build errors by @adsharma in #17726 [SQLLogicTester] Introduce reset label <query label> in the tester by @Tishj in #17729 Adding additional authenticated data for encryption by @ccfelius in #17508 csv_scanner: correct code comment by @Djfe in #17735 Deprecate windows-2019 runners by @hannes in #17745 re-add httpfs apply_patches by @samansmink in #17755 Rename decorator from test_nulls to null_test_parameters by @Mytherin in #17760 [CAPI] Expose ErrorData by @taniabogatsch in #17722 Expose file_size_bytes and footer_size in parquet_file_metadata by @gijshendriksen in #17750 Pass ExtensionLoader when loading extensions, change extension entry function by @Maxxen in #17772 Support glibc 2.28 environments by @James-Gilbert- in #17776 Mark Upper/LowerComparisonType as const by @JelteF in #17773 [Indexes] Buffer-managed indexes part 1: segment handles by @taniabogatsch in #17758 [Julia] api docs improvements by @tqml in #15645 Ensure we use the same layout in RadixPartitionedHashTable and GroupedAggregateHashTable by @lnkuiper in #17790 [Profiling] Propagate the ClientContext into file handle writes by @taniabogatsch in #17754 Fix propagatesNullValues for case expr by @suibianwanwank in #17796 Add qualified parameter to Python GetTableNames API by @evertlammerts in #17797 Merge v1.3 into main by @Mytherin in #17806 Pushdown pivot filter by @flashmouse in #17801 Replace string for const data ptr in encryption api by @ccfelius in #17825 Merge130 by @carlopi in #17833 fix: escape using_columns on JoinRef::ToString by @akoshchiy in #17839 Fix ICE with Windows ARM64 by @staticlibs in #17844 Merge v1.3 into main by @Mytherin in #17851 Add duckdb_type column to parquet_schema by @Mytherin in #17852 Internal #4991: Remove Epoch_MS(MS) by @hawkfish in #17816 #17853 Enable flexible page sizes and update Android NDK to r27 in workflow. by @aprock in #17854 [Indexes] Buffer-managed indexes part 2: segment handle for base nodes by @taniabogatsch in #17828 Function Serialization: adapt to removal of overloads by explicitly casting if argument types have changed by @Mytherin in #17864 julia: add missing methods from C-API by @tqml in #17733 Issue #17153: Window Order Columns by @hawkfish in #17835 Issue #17040: FILL Secondary Sorts by @hawkfish in #17821 Add STRUCT to MAP cast function by @evertlammerts in #17799 Issue #17849: Test FILL Duplicates by @hawkfish in #17869 Add GenAI policy by @szarnyasg in #17882 Update function descriptions and examples for list, array, lambda functions by @c-herrewijn in #17886 Issue #17861: FILL Argument Types by @hawkfish in #17888 Reword GenAI policy by @szarnyasg in #17895 Use an arena linked list for the physical operator children by @taniabogatsch in #17748 Make CTE Materialization the Default Instead of Inlining by @kryonix in #17459 Merge v1.3 into main by @Mytherin in #17897 Leverage VectorType in ColumnDataCollection by @lnkuiper in #17881 Fix empty BP block when writing parquet by @platypii in #17929 fix use after free in adbc on invalid stmt by @ruslandoga in #17927 Do not dispatch JDBC/ODBC jobs in release CI runs by @staticlibs in #17937 Block based encryption by @ccfelius in #17275 Unittester failures summary by @hmeriann in #16833 Add v1.3-ossivalis to Cross version workflow by @hmeriann in #17906 [CI Nightly Fix] Skip logging test if not standard block size by @taniabogatsch in #17957 Visual Studio 17 (2022) fixes by @edouarda in #17948 [Nested] Add struct_position and struct_contains functions by @maiadegraaf in #17819 Enable building spatial and encodings extensions by @staticlibs in #17960 [Nested] Optimize structs in LIST_by @hmeriann in #17949 Issue #5144: AsOf Join Threshold by @hawkfish in #17979 [Fix] Reset profiling info before preparing a query by @taniabogatsch in #17940 Flag to disable database invalidation by @taniabogatsch in #17938 Issue #5123: mpdet in #17947 c-api to copy vector with selection by @abramk in #17870 Fix #18007: correctly execute expressions with pivot operator by @Mytherin in #18020 [Chore] Minor conflict manager refactoring by @taniabogatsch in #18015 Remove LinuAML format for EXPLAIN statements by @qsliu2017 in #17572 Unittest: Add skip_compiled option that can be used to skip built-in C++ tests by @Mytherin in #18034 Add ppc64le spin-wait instruction by @mgiessing in #17837 Merge ossivalis into main by @taniabogatsch in #18036 Remove match-case statements from polars_io.py by @evertlammerts in #18052 Avoid adding commands read from a file to the shell history by @Mytherin in #18057 Adding WAL encryption by @ccfelius in #17955 Encryption: adding -key for the command line by @ccfelius in #17950 fix star expr exclude error by @jayhan94 in #18063 Add support for class-based expression iteration by @Mytherin in #18070 Use timestamp_t instead of time_t for file last modified time by @lnkuiper in #18037 Unittester: add on_new_connection + on_load + skip_tests options by @carlopi in #18042 Fix some scaling issues by @lnkuiper in #17985 Issue #18071: Temporal inf -inf by @hawkfish in #18083 Switch to Optional for type hints in polars lazy dataframe function by @evertlammerts in #18078 Unittest: Configure skip error messages by @carlopi in #18087 Avoid running DraftPR.yml until timeout if token is missing by @carlopi in #18090 Add start/end offset percentage options to Python test runner by @Flogex in #18091 [CSV Reader] Prohibit options delim and sep in same read_csv call by @ackxolotl in #18096 Fix correlated subquery unnest fail by @flashmouse in #18092 [CI] don't run jobs on draft PRs by @hmeriann in #18016 TPC-DS: Use BIGINT fields by @szarnyasg in #18098 Don't throw InternalException in Sort::Sink by @lnkuiper in #18105 ci: build duckdb against the latest emscripten by @cpcloud in #18110 [chore] Merge v1.3-ossivalis on main by @carlopi in #18109 Update description of 'arrow_lossless_conversion' by @szarnyasg in #18046 Internal #3273: Window Task Generation by @hawkfish in #18113 set ::error:: annotations for test runners by @hmeriann in #18072 Improve sort key comparison performance by @lnkuiper in #18131 Add support for MERGE INTO by @Mytherin in #18135 Detect when updates have no effect, and skip performing the actual updates if we encounter these nop updates by @Mytherin in #18144 Add support for AdbcConnectionGetObjects(table_type) by @kou in #18066 Issue #17683: TIME_NS Compilation by @hawkfish in #18053 Implement replace_type function by @lnkuiper in #18077 Bump spatial again: include re-linking for handling global objects in Wasm by @carlopi in #18170 Resolve some small build issues by @madscientist in #18162 fix typo by @felixhummel in #18165 Avoid realloc in CSV writer by @lnkuiper in #18174 fix bug with allowed_paths by @samansmink in #18176 Reduce lock contention for the instance cache by @NiclasHaderer in #18079 Check if GetLastSegment is not nullptr in ColumnData::RevertAppend by @lnkuiper in #18171 [Profiling] Move the client context into more write functions by @taniabogatsch in #17875 Bump Julia to v1.3.2 by @hmeriann in #18185 Merge v1.3-ossivalis into main by @carlopi in #18188 Parquet reader logging by @lnkuiper in #18172 Add VS2019 compat flag to Python wheel build by @staticlibs in #18198 [Parquet][Dev] Update the vendored parquet.thrift to 3ce0760 by @Tishj in #18195 Two-rowID-leaf support in the conflict manager and general refactoring by @taniabogatsch in #18194 More internal-linkage by @Maxxen in #18177 Temporary file encryption by @Mytherin in #18208 Adding temporary file encryption by @ccfelius in #18013 Skip logging test for smaller block sizes by @taniabogatsch in #18201 ci(pyodide): enable WASM exceptions on the latest pyodide build by @cpcloud in #18173 Allow explicit compression for user types by @lnkuiper in #18219 Get type of encoded SortKey from TupleDataLayout by @lnkuiper in #18218 Improve Parquet reader NULL statistics and compress all-NULL columns using CompressedMaterialization by @lnkuiper in #18217 Internal #5264: NLJ Not Distinct by @hawkfish in #18216 Bug#18163 Fix STDDEV_SAMP undeterminism by @minaracic in #18210 [Parquet] Add read support for the VARIANT LogicalType by @Tishj in #18187 Track DataChunk memory usage in various places by @lnkuiper in #18191 Better NULL handling in TupleDataLayout by @lnkuiper in #18069 Dictionary functions by @lnkuiper in #18127 Add support for geoarrow encoded geometries in geoparquet files. by @cfis in #17942 Improve descriptions of thresholds vars affecting join algorithm selection by @TheHillBright in #17377 Connect relations that are not in a subgraph, but are still part of the new relation set by @Tmonster in #18182 [Fix] Don't write empty (partial) blocks for FSST dictionary compression by @taniabogatsch in #18203 Slightly higher memory limit for test by @lnkuiper in #18235 Re-add string -> hugeint compressed materialization function by @lnkuiper in #18234 [Fix] Database path conflict resolution by @taniabogatsch in #18247 Remove require block size from a batch of tests by @taniabogatsch in #18242 Add nightly builds for out-of-tree python extension by @evertlammerts in #18239 Backport DB invalidation flag to ossivalis by @taniabogatsch in #18244 Bump vcpkg-duckdb-ports and test extensions on Windows 10 default stdlib by @carlopi in #18205 Add type safety to FlatVector::GetData<T>, ConstantVector::GetData<T> and UnifiedVectorFormat::GetData<T> by @Mytherin in #18256 [Fix] Adjust test for smaller block sizes by @taniabogatsch in #18255 Fix integer overflow in sequence vector by @xuke-hat in #18245 fixes for some minor llvm 20 complaints by @hannes in #18257 update run_extension_medata_tests.sh by @hmeriann in #17976 Bunch of loosely connected test/CI fixes by @carlopi in #18254 disable WebAssembly duckdb-wasm builds job in NightlyTests triggered by 'workflow_dispatch' event by @hmeriann in #18129 Allow for static libs from extension dependencies to be bundled by @abramk in #18226 Fix dictionary-related assertions by @lnkuiper in #18260 Fixes for gcc 15 by @hannes in #18261 Reduce copy in Vector::Reinterpret by @xuke-hat in #18264 [Parquet] Add read support for the VARIANT LogicalType (with shredded encoding) by @Tishj in #18224 Expanded autocomplete suggestions by @Dtenwolde in #18243 Support HUGEINT in printf and format by @xuke-hat in #13277 Move aarch64 / arm64 to native github runner by @evertlammerts in #18269 Bump vcpkg-duckdb-ports to solve OSX linking by @carlopi in #18268 Add support for RETURNING to MERGE INTO by @Mytherin in #18271 Use set for row ID scanning during index scans by @taniabogatsch in #18274 Use DuckDB cast infrastructure in fmt for new uhugeint/hugeint code by @Mytherin in #18275 [Fix] Adjust test to run with different block sizes by @taniabogatsch in #18277 Use FromEpochSeconds instead of FromTimeT in FileSystem::GetLastModifiedTime by @lnkuiper in #18281 Add target for installing Python deps. by @xevix in #18285 backport 'Unit Tester Configuration' pt2 by @c-herrewijn in #18282 backport 'Unit Tester Configuration' by @c-herrewijn in #18263 Fixup Main.yml for v1.3-ossivalis post #18282 by @carlopi in #18289 SHOW TABLES FROM <qualified_name> by @xevix in #18179 [Unittester] Add autoloading option by @carlopi in #18290 Merge ossivalis into main by @taniabogatsch in #18272 resolve hidden merge conflict with duplicate db name in json configs by @c-herrewijn in #18292 Bump vcpkg-duckdb-ports, now fixing also mingw by @carlopi in #18300 [Fix] Missing block when renaming fields by @taniabogatsch in #18293 [Arrow] Fix unused static function warning by @Tishj in #18278 feat: Parquet extension add row_group_compressed_size by @mapleFU in #18294 [Parquet][Write] Fix timestamp sec writes to parquet by @Tishj in #18273 bump httpfs by @Tmonster in #18258 [Clang Tidy] Fix missing includes in patas_scan.hpp by @Tishj in #18276 New Arrow C-API by @pdet in #18246 Skip test/sql/copy/s3/url_encode.test due to httpfs update by @carlopi in #18317 Make storage-version a test parameter by @Mytherin in #18324 Backport #18254 by @carlopi in #18306 feat: making Parquet write RowGroup.total_compressed_size by @mapleFU in #18307 add the from-table-function as parameter to copy-from-bind by @peterboncz in #18004 Python external dispatch param fixes by @evertlammerts in #18343 Aarch64 backport by @evertlammerts in #18345 Fix debug error in join order optimizer by @Tmonster in #18344 [Fix] Block verification for add and drop field info by @taniabogatsch in #18302 download Real Nest data in quiet mode by @hmeriann in #18346 Fix condition indexes in join filter pushdown by @Damon07 in #18341 [unittest] - fix doubled error headers on Unexpected failure by @hmeriann in #18314 Extend PEG parser grammar by @Dtenwolde in #18221 [C API] Expose expressions and use them in scalar function binding by @taniabogatsch in #18142 Restore OSX tests, moving them to single --autoloading available step by @carlopi in #18335 Add support for checkpointing in-memory tables by @Mytherin in #18348 Revert "[unittest] - fix doubled error headers on Unexpected failure" by @Mytherin in #18355 Python external dispatch param fixes by @evertlammerts in #18359 Re-enable url-encode test by @Tmonster in #18360 Enable stack traces on linux for bundled libraries by @taniabogatsch in #18363 Split up out-of-tree extensions into separate files, and allow out-of-tree extensions to be built using BUILD_EXTENSIONS={ext_name} by @Mytherin in #18357 Pass AttachOptions to attach method, and turn StorageExtensionInfo into an optional_ptr by @Mytherin in #18368 Merge v1.3-ossivalis into main by @carlopi in #18364 More robustness around deprecated extension settings by @carlopi in #18353 Add missing ninja to workflow file by @Dtenwolde in #18373 bump httpfs by @Tmonster in #18380 Re-enable but deprecate CORE_EXTENSIONS in CMakeLists.txt by @evertlammerts in #18377 Uncomment skipped decimal REE tests by @amoeba in #18372 add option 'block_size' to test configs by @c-herrewijn in #18347 [chore] Fixup side-effects from 8cf9ed4 by @carlopi in #18385 Bump httpfs by @taniabogatsch in #18388 Re-use table metadata when table is not altered during checkpoint by @Mytherin in #18390 Approx database count system function by @taniabogatsch in #18392 Re-use metadata of unaltered row groups when checkpointing a table by @Mytherin in #18395 Correct and consistent integer arithmetic error messages by @soerenwolfers in #18393 Record whether or not cross products are implicit or not, and use this for converting queries back to SQL by @Mytherin in #18394 CI: Fix Discussion mirroring by @szarnyasg in #18397 Store extra metadata blocks in RowGroupPointer, and only flush dirty Metadata blocks by @Mytherin in #18398 Internal #3273: Window Hashed Sort by @hawkfish in #18337 Wrap runner.ExecuteFile, otherwise cleanup is not properly performed by @carlopi in #18400 [BUGFIX] Update delim offset for RHS of DELIM JOIN when correlated column is in RHS of Cross product by @Tmonster in #18375 CI: Add separate job for discussion mirroring by @szarnyasg in #18407 [ Python SQLLogic Tester ] Add MERGE_INTO statement to duckdb python by @hmeriann in #18402 Remove incorrect assertion by @Mytherin in #18404 Internal #5294: TIME_NS C API by @hawkfish in #18215 Add DuckLake back in by @Mytherin in #18405 Add support for table_constraints of AdbcConnectionGetObjects() by @kou in #18181 Merge v1.3-ossivalis in main by @carlopi in #18401 feat: remove anything following ? in database name by @rustyconover in #18417 Correctly fetch only base column data in ColumnData::FetchUpdateData by @Mytherin in #18423 Refactor extension CI to use extension-ci-tools by @samansmink in #18361 Internal #5367: SortedAggregateFunction Sort Update by @hawkfish in #18408 Internal #5368: WindowNaiveAggregator Sort Update by @hawkfish in #18409 [Fix] Block size nightly by @taniabogatsch in #18425 [Chore] Tidy test configs by @taniabogatsch in #18426 Include pyodide build configuration by @rgbkrk in #18183 Parquet: add row-group ordinal during writing encryption by @mapleFU in #18433 [Fix] Reset segment memory when initialising new Prefix by @taniabogatsch in #18441 Update pyodide build to 0.28.0 by @rgbkrk in #18446 Add support for "template" types by @Maxxen in #18410 Internal #5384: WindowDistinctAggregator Sort Update by @hawkfish in #18442 [Chore] Improve skipped tests in test config and add verify_fetch_row config by @taniabogatsch in #18436 Buffer index appends during WAL replay by @taniabogatsch in #18313 Add support for generic settings, and move many settings over to generic settings by @Mytherin in #18447 Internal #5385: WindowMergeSortTree Sort Update by @hawkfish in #18461 Bump postgres to latest main by @Mytherin in #18464 Merge ossivalis by @taniabogatsch in #18456 Internal #5366: WindowDeltaScanner by @hawkfish in #18468 SUM and + Operator for Varints by @pdet in #18424 [Fix] Rework transaction logic in commit, rollback and checkpoint paths by @taniabogatsch in #18474 re-nable extensions in invokeci by @samansmink in #18476 Internal #5384: Window Sorting Polish by @hawkfish in #18484 Unify ON CONFLICT and MERGE INTO by @Mytherin in #18480 More insights around dict_fsst compression failure by @taniabogatsch in #18475 Change ctrl-a/ctrl-e to move to start/end of line, not buffer by @tpot in #18490 add delta linux back to ci by @samansmink in #18491 Fix accidental internal exception in type transformation by @hannes in #18492 [Profiling] Add client context into read functions by @maiadegraaf in #18438 julia: config improvements by @aplavin in #17585 fix: add missing space in AttachInfo::ToString() by @rustyconover in #18500 Merge ossivalis by @taniabogatsch in #18502 Change UNICODE to UTF8 by @sheldonrobinson in #17586 Fix: Remove overly strict assertion on empty string value by @Dtenwolde in #18504 Fix several bugs/fuzzer issues by @Mytherin in #18503 Allow expressions to be used in ATTACH / COPY options by @Mytherin in #18515 Remove immediate_transaction_mode from DB config options by @jeewonhh in #18516 Temporarily excluding Build Pyodide wheel for Python 3.11 because it fails to build WASM wheels by @hmeriann in #18508 ParserException for Pragma with named parameters by @Dtenwolde in #18506 Add verify fetch row config to Main.yml by @taniabogatsch in #18478 Adding WITH ORDINALITY to DuckDB by @niykko in #16581 When tracking evicted_data_per_tag, track actual size on disk after temp file compression by @Mytherin in #18521 Fix: Write the salt together with the HT offset when determining the value for key comparison by @gropaul in #18374 Fix incorrect character encoding in GetLastErrorAsString on Windows by @soutong in #18431 Dynamically determine dictionary size limit in Parquet writer (if unset) by @lnkuiper in #18356 Internal #16560: Numeric TRUNC Precision by @hawkfish in #18511 Consistently detect JSON schema indepent of number of threads by @lnkuiper in #18522 ALP test: skip TPC-DS 67 - it is not consistent with floating point numbers by @Mytherin in #18528 [Varint] Negation, Subtraction and Over/under-flow checking by @pdet in #18477 fix: support both field orders for variant struct by @samansmink in #18532 Add CAPI to retrieve client context for table functions by @VGSML in #18520 Add StatementVerifier for EXPLAIN by @lnkuiper in #18529 Use global index, not local id when creating filters in MultiFileColumnMapper by @lnkuiper in #18537 Add support for explicit clean-up routine in test config, and exit multi-statement execution when an error is encountered by @Mytherin in #18539 fix: improve handling variant nulls and nested types by @samansmink in #18538 Allow overriding openssl version for FIPS compliance by @abramk in #18499 Unittester: Add the --sort-style parameter that allows a fallback comparison where results are sorted according to a given sort-style by @Mytherin in #18542 Restore missing test/configs/small_block_size.json file by @hmeriann in #18507 [Fix] Follow-up PR to only delete unique row IDs by @taniabogatsch in #18545 [ART] Node::Free refactoring by @taniabogatsch in #18544 Implement special-case VARCHAR to JSON[] casts and vice versa by @lnkuiper in #18541 Check if heap_block_ids is empty before getting start/end when destroying chunks in TupleDataCollection by @lnkuiper in #18556 optimize/parquet: generate movable types for parquet by @mapleFU in #18510 [easy] [no-op] Minor optimization on iterator lookup by @dentiny in #15349 Fixing compilation with -std=cpp23 by @hannes in #18557 Add compile option standalone-debug for clang by @flashmouse in #17433 Rename the Varint type to Bignum by @pdet in #18547 [Indexes] Buffer-managed indexes part 3: segment handle for Node48 and Node256 by @taniabogatsch in #18567 fix: add formatting to explain row counts by @rustyconover in #18566 [CSV Sniffer] Fixing bug of not properly setting skipped rows from sniffer by @pdet in #18555 [Fix] Tidy check ossivalis by @taniabogatsch in #18583 [Fix] Adjust shrink threshold back to original count > SHRINK_THRESHOLD by @taniabogatsch in #18582 Flip left/right delim join based on cardinalities by @lnkuiper in #18552 fix: use thousands separator and decimal for row counts induckbox output format by @rustyconover in #18564 Force LIST/ARRAY child vectors on a Parquet single page by @lnkuiper in #18578 String dictionary hash cache by @lnkuiper in #18580 fix: libduckdb.so missing soversion by @strophy in #18305 Pushdown filters on coalesced outer join keys compared for equality under the join condition by @matteobilardi in #18169 Adds a function for updating and adding values in a struct by @teaguesterling in #15533 fix hidden merge conflict by @hannes in #18589 Increment storage version to enable DICT_FSST in benchmark file by @lnkuiper in #18588 [Fix] Hidden test failure in test_struct_update.test by @taniabogatsch in #18598 correctly setting log transaction id in ThreadContext by @xuke-hat in #18536 Backport renaming a config name small_block_size.json to block_size_16kB in NightlyTests by @hmeriann in #18581 Update README.md by @matthew-wright07 in #18614 [Test] Fix test case and a benchmark by @hmeriann in #18610 [CI] Don't zip and upload Code Coverage tests results when Code Coverage got cancelled by @hmeriann in #18607 [Profiling] Add client context into more read functions by @maiadegraaf in #18514 bump httpfs by @Tmonster in #18591 Fix serialization backwards compatability for varargs functions by @Maxxen in #18596 Issue #18631: Streaming Windowed Quantile by @hawkfish in #18636 parquet/parquet_multi_file_info.cpp: fix move from stack by @carlopi in #18634 Adjust filter pushdown to latest polars release by @pdet in #18624 Re-add hugeint to __internal_compress_string by @jeewonhh in #18622 Add Field IDS to multi file reader for positional deletes by @Tmonster in #18617 [CSV Sniffer] Fix type detection issue with union and empty columns by @pdet in #18606 [ART] ART::Erase refactoring by @taniabogatsch in #18595 wrap httplib ::max() call in WIN_32 check by @Tmonster in #18590 Add enable verification config run by @taniabogatsch in #18467 feat: add ETA to progress bar in DuckDB CLI by @rustyconover in #18575 Add "Hash Zero" verification CI run by @lnkuiper in #18623 Make more configs into generic settings by @jeewonhh in #18592 bump avro to v1.4 by @Tmonster in #18434 bump spatial (on main) by @Maxxen in #18197 Change arrow() to export record batch reader by @pdet in #18642 [Fix] Prevent logger deadlock by @taniabogatsch in #18637 Remove PRAGMA enable_verification by @taniabogatsch in #18645 Add 1.4 release codename by @hannes in #18652 Python test runner: Fix result check for COPY ... RETURN_STATS queries by @Flogex in #18625 Merge ossivalis into main by @taniabogatsch in #18644 CLI: Make ETA more of an estimate, and support large_row_rendering for footers by @Mytherin in #18656 Remove more PRAGMA enable_verification by @taniabogatsch in #18664 [CI] skip building encodings extension in InvokeCI by @hmeriann in #18655 Python test runner: Fix hash comparison error output by @Flogex in #18626 [Dev] Add script to create patch from changes in an extension repository by @Tishj in #18620 Correctly set weights in reservoir sample when switch to slow sampling by @xuke-hat in #18563 Internal #5366: Window Interrupt Arguments by @hawkfish in #18651 Remove PRAGMA enable_verification in more tests by @taniabogatsch in #18670 [ Python SQLLogic Tester ] Add MERGE_INTO to statement.type enum in result.py by @hmeriann in #18675 Load pandas in import cache before binding by @evertlammerts in #18658 Internal #5662: IEJoin Test Plans by @hawkfish in #18680 Correctly allocate uncompressed string data in ZSTD for many giant strings by @Mytherin in #18678 Grab lock and double-check that column is not loaded in MoveToCollection by @Mytherin in #18677 fix error message related to wrong memory unit by @LiranBri in #18671 [CI] Temporarily skip triggering R Package Windows (Extensions) job by @hmeriann in #18628 Fix the issue where delta_for isn't used in bitpacking when for is unavailable by @xuke-hat in #18616 Add date_trunc() simplification rules by @rcurtin in #18457 Expected errors 2053 by @hmeriann in #14213 Internal #5366: Window State Arguments by @hawkfish in #18676 Add WAL test config run by @taniabogatsch in #18683 Using a different workflow to release the python package by @evertlammerts in #18685 Make sure parse errors are wrapped in ErrorData by @evertlammerts in #18682 [Python SQLLogicTest] Add test/sql/pragma/profiling/test_profiling_all.test to the SKIPPED_TESTS set by @hmeriann in #18689 Issue #18457: DateTrunc Simplification Warnings by @hawkfish in #18687 Expected errors 2053 by @hmeriann in #18695 Hold row group lock for entire call of MoveToCollection by @Mytherin in #18694 Unplug python (in ossivalis) by @evertlammerts in #18699 Correctly handle collations for IN (subquery) by @Mytherin in #18698 Move attached databases from a CatalogSet to a dedicated map of shared pointers by @Mytherin in #18693 Make ART construction iterative via ARTBuilder by @taniabogatsch in #18702 [Fix] Correctly handle table and index chunks in WAL replay buffering by @taniabogatsch in #18700 Python-style positional/named arguments for macro's by @lnkuiper in #18684 Internal #3273: Hashed Sort States by @hawkfish in #18690 Add Option to Allocate Using an Arena in string_t by @maiadegraaf in #17992 Fix issue with materialized CTE optimization in flatten_dependent_join by @kryonix in #18714 [Profiling] Add Profiling to Read Function by @maiadegraaf in #18661 Correctly throw an error when too few columns are supplied in MERGE INTO INSERT by @Mytherin in #18715 Improved grammar generation script by @NiclasHaderer in #18716 #Fix 18558: add row_group scan fast path by @flashmouse in #18686 Added support for blob<->uuid conversions by @dioptre in #18027 Minor fixes for other catalogs - mostly checking IsDuckTable() for unsupported operations by @Mytherin in #18720 Fix PIVOT in multiple statements by @evertlammerts in #18729 Internal #5669: Loop Join Thresholds by @hawkfish in #18733 feat: enhance .tables command with schema disambiguation and filtering by @shivampr in #18641 Add (CSV) file logger by @samansmink in #17692 Use 1-based indexing for SQL-based JSON array extraction by @lnkuiper in #18735 [unittest] SkipLoggingSameError() to make unittester report one failure per case by @hmeriann in #18270 fix timetravel for default tables by @samansmink in #18240 [C API] Function to set a copy callback for bind data by @taniabogatsch in #18739 Secrets: if serialization_type is not specified, assume it's a key value secret by @Mytherin in #18743 Merge ossivalis into main by @taniabogatsch in #18719 Use correct type for pushing collations in subqueries by @Mytherin in #18744 Add OS X notarization for DuckDB CLI and libduckdb.dylib by @hannes in #18747 Add missing expected errors to the test cases by @hmeriann in #18746 removed placeholder client directories for node and jdbc, its been > 1 yr by @hannes in #18757 Append using a SQL query, instead of directly appending to a base table, and support user-provided queries through the QueryAppender by @Mytherin in #18738 Backport #18374 to v1.3-ossivalis by @lnkuiper in #18752 Add leak suppressions to nightly runs by @Mytherin in #18748 Remove separate WAL encryption flag by @Mytherin in #18750 Fixing lazy polars execution on query result by @pdet in #18749 [Profiling] Add Profiling to Write Function by @maiadegraaf in #18724 Extensions.yml should also check converted_to_draft by @carlopi in #18754 Minor logging fixes and more benchmarking by @samansmink in #18755 Add missing expected errors to the test cases (next chunk) by @hmeriann in #18753 Refactor read_blob and read_text to use MultiFileFunction. by @xevix in #18706 Add support for auto-globbing within a directory: if no matches are found for a specific path, we retry with /**/*.[ext] appended by @Mytherin in #18760 Fix radix partitioning with more than 10 bits by @ctsk in #18761 Fix index resolution when querying table with index via view by @mach-kernel in #18319 Fix Path Typo in Extension's CMake Warning Message by @beryllw in #18766 Make duckdb_log return a TIMESTAMP_TZ by @Mytherin in #18768 Revert "Use 1-based indexing for SQL-based JSON array extraction" by @Mytherin in #18758 [CI] Adjust test configs post logger PR by @taniabogatsch in #18771 [Test Fix] Forward output to file by @taniabogatsch in #18772 Propagate DUCKDB_*_VERSION in extensions and tests by @Y-- in #18774 Add file_size_bytes (de-)serialization by @lnkuiper in #18775 Use microsecond resolution for printing the current timestamp by @Mytherin in #18776 Improve error messages for merge / vector reference by @Mytherin in #18777 Move row id logic to separate RowIdColumnData class instead of inlining it into the RowGroup by @Mytherin in #18780 Treat ENABLE_EXTENSION_AUTOINSTALL as the BOOL that it is by @evertlammerts in #18778 Add memory_limit parameter to benchmark_runner/test_runner.py by @lnkuiper in #18790 fix: improve speed of GetValue() for STRUCT type by @rustyconover in #18785 Internal #3273: Parallel Window Masks by @hawkfish in #18731 Task Scheduler: track exact task count, and re-signal on dequeue failure if there are tasks left by @Mytherin in #18792 fix: coalesce query progress updates to reduce terminal writes by @rustyconover in #18672 Support expressions as COPY file target by @Mytherin in #18795 Remove everything python-package related by @evertlammerts in #18789 Improve autocomplete suggestions by @Dtenwolde in #18773 bump httpfs so it includes curl option by @Tmonster in #18691 Issue #18767: Ignore Timestamp Offsets by @hawkfish in #18794 Fixup progress_bar: avoid converting doubles into int32_t unchecked by @carlopi in #18800 [chore] Fixup tidy-check on src/logging/log_manager.cpp by passing const & by @carlopi in #18801 Internal #3273: Hashed Sort Callbacks by @hawkfish in #18796 Typed macro parameters by @lnkuiper in #18786 Fix some unindented interactions between EMPTY_RESULT_PULLUP and MATERIALIZED CTEs by @kryonix in #18805 Add support for non-aggregate window functions by @Maxxen in #18788 Expected errors 2053 by @hmeriann in #18810 Test runner: Expand '{UUID}' into a random UUID by @carlopi in #18809 Provide failing file name in Parquet reader error messages by @Mytherin in #18814 [CI] install libcurl4-openssl-dev with apt-get by @hmeriann in #18811 fix: Add COLLATE NOCASE support to strpos function by @shivampr in #18819 Add callback to get a list of copy options, use this to provide suggestions and to erase options from import that are only used during exporting by @Mytherin in #18812 For BC reasons - keep VARINT as alias for BIGNUM by @Mytherin in #18821 [Fix] Bug in fixed-size buffer when throwing out-of-memory by @taniabogatsch in https://github.com/duckdb/duckdb/pull/18769 Re-add accidentally removed check if copy_from is supported by @Mytherin in https://github.com/duckdb/duckdb/pull/18824 Fix format-fix runs on Linux by @staticlibs in https://github.com/duckdb/duckdb/pull/18827 Extensions.yml: Pass down save_cache to inner workflows by @carlopi in https://github.com/duckdb/duckdb/pull/18828 Fix: Preserve database configuration flags for tab completion in DuckDB shell by @rustyconover in https://github.com/duckdb/duckdb/pull/18482 Ensure a WAL file matches the DB file and checkpoint iteration by @taniabogatsch in https://github.com/duckdb/duckdb/pull/18823 fix: sanitize input for enable_logging by @samansmink in https://github.com/duckdb/duckdb/pull/18830 fix: silence warnings about signed/unsigned conversions. by @rustyconover in https://github.com/duckdb/duckdb/pull/18835 Avoid expensive checkpoints and write amplification by appending row groups, and limiting vacuum operations for the last number of row groups by @Mytherin in https://github.com/duckdb/duckdb/pull/18829 Fix/run function in transaction by @Evannnnnnnn in https://github.com/duckdb/duckdb/pull/18741 add appender to concurrent test by @c-herrewijn in https://github.com/duckdb/duckdb/pull/18721 Add support for reading/writing native parquet geometry types by @Maxxen in https://github.com/duckdb/duckdb/pull/18832 Don't notify Py pkg when override git describe is set by @evertlammerts in https://github.com/duckdb/duckdb/pull/18843 Avoid printing '99 hours', given in most cases that means estimate is… by @carlopi in https://github.com/duckdb/duckdb/pull/18839 Add the VARIANT LogicalType by @Tishj in https://github.com/duckdb/duckdb/pull/18609 Document storage version flag in CLI + minor rendering fix by @Mytherin in https://github.com/duckdb/duckdb/pull/18841 Ignore null verification for statistics on structs by @d-justen in https://github.com/duckdb/duckdb/pull/18813 Add OnBeginExtensionLoad callback by @Mytherin in https://github.com/duckdb/duckdb/pull/18842 Bump MySQL/Postgres/SQLite by @Mytherin in https://github.com/duckdb/duckdb/pull/18848 Merge ossivalis into main by @taniabogatsch in https://github.com/duckdb/duckdb/pull/18844 Add test_env to unit tester by @pdet in https://github.com/duckdb/duckdb/pull/18847 WAL <> DB File Match Fixes by @Mytherin in https://github.com/duckdb/duckdb/pull/18849 Make ATTACH OR REPLACE atomic, keep list of used databases in MetaTransaction by @Mytherin in https://github.com/duckdb/duckdb/pull/18850 Fix NULL path for json_each/json_tree by @lnkuiper in https://github.com/duckdb/duckdb/pull/18852 No more wal_encryption flag by @jeewonhh in https://github.com/duckdb/duckdb/pull/18851 Bump Ducklake by @pdet in https://github.com/duckdb/duckdb/pull/18825 Add more encryption modes CTR and CBC by @hannes in https://github.com/duckdb/duckdb/pull/18619 Centralize attached database paths in a DatabaseFilePathManager which is shared across databases created through the same DBInstanceCache by @Mytherin in https://github.com/duckdb/duckdb/pull/18857 Hold segment lock during GetColumnSegmentInfo by @Mytherin in https://github.com/duckdb/duckdb/pull/18859 update duckdb azure extension ref for 1.4.0 by @benfleis in https://github.com/duckdb/duckdb/pull/18868 Expected errors 2053 by @hmeriann in https://github.com/duckdb/duckdb/pull/18864 Add a FORCE_DEBUG flag to force -DDEBUG, similar to FORCE_ASSERT by @Mytherin in https://github.com/duckdb/duckdb/pull/18872 Bump & remove patches for delta, avro, excel, encodings, fts by @samansmink in https://github.com/duckdb/duckdb/pull/18869 [minor] Incompatible DB error message: add newline by @carlopi in https://github.com/duckdb/duckdb/pull/18861 Bump mbedtls to v3.6.4 by @Mytherin in https://github.com/duckdb/duckdb/pull/18871 Storage fuzzing + several fixes by @Mytherin in https://github.com/duckdb/duckdb/pull/18876 Update ducdkb iceberg hash by @Tmonster in https://github.com/duckdb/duckdb/pull/18873 [Test] Small fixes to concurrent attach/detach test by @taniabogatsch in https://github.com/duckdb/duckdb/pull/18862 Internal #5796: Window Progress by @hawkfish in https://github.com/duckdb/duckdb/pull/18860 Add COPY (...) TO ... (FORMAT BLOB) by @Maxxen in https://github.com/duckdb/duckdb/pull/18840 Update spatial+vss+sqlsmith in preparation for v1.4 by @Maxxen in https://github.com/duckdb/duckdb/pull/18882 Avoid automatically checkpointing if the database instance has been invalidated by @Mytherin in https://github.com/duckdb/duckdb/pull/18881 Add COPY (FORMAT BLOB) to Andium too :^) by @Maxxen in https://github.com/duckdb/duckdb/pull/18884 [C API] Result schema of prepared statements by @hrl20 in https://github.com/duckdb/duckdb/pull/18779 Json: no reinterpret<size_t*> by @carlopi in https://github.com/duckdb/duckdb/pull/18886 [Dev] Fix footgun in string_t::SetSizeAndFinalize by @Tishj in https://github.com/duckdb/duckdb/pull/18885 [chore] Bump config test/configs/compressed_in_memory.json to new format by @carlopi in https://github.com/duckdb/duckdb/pull/18888 bump aws and iceberg by @Tmonster in https://github.com/duckdb/duckdb/pull/18889 Add rowsort to upsert_default.test by @jeewonhh in https://github.com/duckdb/duckdb/pull/18890 fixing auto-specifying ciphers and remove double storage by @hannes in https://github.com/duckdb/duckdb/pull/18891 Expected errors 2053 by @hmeriann in https://github.com/duckdb/duckdb/pull/18892 Keep base data scan state alive in ColumnData::Update call by @Mytherin in https://github.com/duckdb/duckdb/pull/18893 Add callback for when an extension fails to load, and also log this by @Mytherin in https://github.com/duckdb/duckdb/pull/18894 Encryption now encoded as a bit, centralizing in set/getter by @carlopi in https://github.com/duckdb/duckdb/pull/18897 Bump httpfs to v1.4-andium branch by @carlopi in https://github.com/duckdb/duckdb/pull/18898 fix: refine query ETA display and Kalman filter stability by @rustyconover in https://github.com/duckdb/duckdb/pull/18880 Bump inet & aws by @samansmink in https://github.com/duckdb/duckdb/pull/18899 [chore] Fix amalgamation build in progress_bar by @carlopi in https://github.com/duckdb/duckdb/pull/18910 Cannot create table from variant yet by @Mytherin in https://github.com/duckdb/duckdb/pull/18912 In VerifyZeroReaders, get the header size from the buffer we are replacing instead of from the block manager by @Mytherin in https://github.com/duckdb/duckdb/pull/18909 Fix #18152: avoid auto-detecting hive partitioning with COPY .. FROM by @Mytherin in https://github.com/duckdb/duckdb/pull/18911 CLI: Correctly move to start of line by @Mytherin in https://github.com/duckdb/duckdb/pull/18920 Strip question mark parameters from default temporary directory by @Mytherin in https://github.com/duckdb/duckdb/pull/18915 Move Hash Zero CI run to nightly by @Mytherin in https://github.com/duckdb/duckdb/pull/18925 Bump Iceberg by @Tmonster in https…

Laurens Kuiper added 4 commits December 9, 2025 16:10

logarithmically grow read buffer

b34797c

logarithmically grow read buffer

99b2575

Merge branch 'grow_httpfs_read_buffer' of github.com:lnkuiper/duckdb-…

68b17bc

…httpfs into grow_httpfs_read_buffer

negate

897f9c5

carlopi reviewed Dec 9, 2025

View reviewed changes

Comment thread src/httpfs.cpp Outdated

carlopi self-requested a review December 9, 2025 15:39

avoid allocating the read buffer if we know it won't be used

6d9494b

carlopi reviewed Dec 10, 2025

View reviewed changes

add test

9c57a4a

carlopi merged commit 7bd8c9c into duckdb:v1.4-andium Dec 10, 2025
34 of 35 checks passed

carlopi added a commit to carlopi/duckdb that referenced this pull request Dec 11, 2025

Bumping httpfs to include duckdb/duckdb-httpfs#174

bd9f3dd

Both to have this enabled on nightly, and to more widely test it

Mytherin added a commit to duckdb/duckdb that referenced this pull request Dec 12, 2025

Bumping httpfs to include duckdb/duckdb-httpfs#174 (#20145)

2df4935

Both to have this enabled on nightly, and to more widely test it in the context of main repo CI.

github-actions Bot mentioned this pull request Dec 31, 2025

vendor: Update vendored sources to duckdb/duckdb@42fbf1093eba4cbe4a000b0f888bdf4f012adc3b duckdb/duckdb-r#1838

Merged

artjomPlaunov pushed a commit to artjomPlaunov/duckdb that referenced this pull request Jan 9, 2026

Bumping httpfs to include duckdb/duckdb-httpfs#174

c3dde55

Both to have this enabled on nightly, and to more widely test it

dentiny mentioned this pull request Feb 5, 2026

GEOMETRY Rework: Part 7 - More Coordinate Reference System Support duckdb/duckdb#20721

Merged

lnkuiper deleted the grow_httpfs_read_buffer branch June 29, 2026 14:30

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Logarithmically grow read buffer for sequential reads - #174

Logarithmically grow read buffer for sequential reads#174
carlopi merged 6 commits into
duckdb:v1.4-andiumfrom
lnkuiper:grow_httpfs_read_buffer

lnkuiper commented Dec 9, 2025

Uh oh!

Uh oh!

carlopi left a comment

Uh oh!

carlopi commented Dec 10, 2025 •

edited

Loading

Uh oh!

carlopi commented Dec 10, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

lnkuiper commented Dec 9, 2025

Uh oh!

Uh oh!

carlopi left a comment

Choose a reason for hiding this comment

Uh oh!

carlopi commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carlopi commented Dec 10, 2025

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

carlopi commented Dec 10, 2025 •

edited

Loading