Date: 2026-03-09 19:41:35 +0100

Fix DuckFuzz duckdb/duckdb#4325 (duckdb/duckdb#21215)
Internal duckdb/duckdb#7691: 2026a Time Zones (duckdb/duckdb#21204)
Add framework to test two DuckDB versions (duckdb/duckdb#21184)

Date: 2026-03-09 19:41:35 +0100

Fix DuckFuzz duckdb/duckdb#4325 (duckdb/duckdb#21215)
Internal duckdb/duckdb#7691: 2026a Time Zones (duckdb/duckdb#21204)
Add framework to test two DuckDB versions (duckdb/duckdb#21184)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

Date: 2026-03-09 19:41:35 +0100

Fix DuckFuzz duckdb/duckdb#4325 (duckdb/duckdb#21215)
Internal duckdb/duckdb#7691: 2026a Time Zones (duckdb/duckdb#21204)
Add framework to test two DuckDB versions (duckdb/duckdb#21184)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

Date: 2026-03-09 19:41:35 +0100

Fix DuckFuzz duckdb/duckdb#4325 (duckdb/duckdb#21215)
Internal duckdb/duckdb#7691: 2026a Time Zones (duckdb/duckdb#21204)
Add framework to test two DuckDB versions (duckdb/duckdb#21184)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

Date: 2026-03-09 19:41:35 +0100

Fix DuckFuzz duckdb/duckdb#4325 (duckdb/duckdb#21215)
Internal duckdb/duckdb#7691: 2026a Time Zones (duckdb/duckdb#21204)
Add framework to test two DuckDB versions (duckdb/duckdb#21184)

Co-authored-by: krlmlr <krlmlr@users.noreply.github.com>

# Overview

This pull request introduces a Python-based backward compatibility (BWC)
test runner for DuckDB, along with CI integration and cache management
tooling.

The high-level idea for this framework is the following:

Given two DuckDB versions, `Vold` and `Vnew` (with `Vold < Vnew`):

For each test suite of DuckDB @ `Vold`:
* **Step 1 - Serialize plans & results (old version):**
  * For each query in the test suite:
    * Serialize the execution plan with DuckDB @ `Vold`
    * Execute the query and serialize the result with DuckDB @ `Vold`
* **Step 2 - Execute serialized plans (new version):**
  * For each serialized plan from step 1:
    * Deserialize and execute the plan with DuckDB @ `Vnew`
    * Serialize the result with DuckDB @ `Vnew`
* **Step 3 - Compare results:**
* Compare the results obtained in step 2 with the baseline results from
step 1

# Implementation

## Components

```
test/bwc/
├── runner.py                 # Main test orchestrator
├── update_cache.py           # Update cache from CI artifacts
├── export_cache.py           # Export cache to compressed archives
└── utils/
    ├── duckdb_cli.py         # DuckDB CLI process interface
    ├── duckdb_installer.py   # CLI download, test spec cloning, extension install
    ├── test_files_parser.py  # SQL logic test parsing & serialization
    ├── test_report.py        # Test result reporting (writes to DuckDB database)
    ├── query_exclusions.py   # Query filtering (skip / execute-from-SQL rules)
    └── logger.py             # Logging setup (CI-aware)
```

* `runner.py` orchestrates BWC tests: installs CLIs and extensions,
loads test specs, runs tests sequentially or in parallel, and writes
results to a DuckDB report database.
* `duckdb_cli.py` implements a class for launching and interacting with
DuckDB CLI processes (crash recovery, timeout handling, output parsing).
* `duckdb_installer.py` automates CLI installation, downloads test specs
from GitHub, and manages extension installation for CLIs.
* `test_files_parser.py` parses SQL logic test files using
`duckdb-sqllogictest-python` and converts them into tagged query files
with serialized plans.
* `query_exclusions.py` defines patterns for queries that should be
skipped entirely (e.g., transactions, pragma verification) or executed
from SQL instead of serialized plans (e.g., DDL, DML, prepared
statements).

The implementation relies on a dedicated DuckDB extension, `test-utils`,
implemented [here](https://github.com/duckdb/bwc-test-utils).
The latest version was compiled for all DuckDB versions starting at
v1.1.0, and published at `ext.motherduck.com`'s repo. It is also
possible to use a custom-built extension (cf. below for building).

The implementation also relies on the `duckdb-sqllogictest-python`
package (repo
[here](https://github.com/duckdb/duckdb-sqllogictest-python)), as well
as the most recent DuckDB Python package.

## Skip configuration

Known failing tests are tracked in JSON configuration files under
`test/configs/`:
- `serialization_bwc_base.json` - tests skipped across all versions
- `serialization_bwc_v{version}.json` - version-specific skips

## CI

The GitHub Actions workflow (`.github/workflows/SerializationBWC.yml`)
runs on pushes to `main` and on pull requests:

1. **Build job:** Compiles DuckDB with the `test-utils`, `httpfs`, and
`inet` extensions.
2. **Test job (matrix):** Runs BWC tests in parallel across all
supported old versions (v1.1.0 through v1.4.4), then exports a cache
archive for each version.
3. **Summary job:** Aggregates results and reports overall
success/failure.

## Cache management

To speed up repeated runs, the framework caches serialized plans and
old-version results:
- `export_cache.py` packages cached files (`.sql`, `.plan.bin`,
`.result.bin`, fixtures) into compressed archives.
- `update_cache.py` downloads cache artifacts from a CI run and updates
the `test-utils` repository.

# Usage

## Runner parameters

```
python3 test/bwc/runner.py [OPTIONS]
```

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `--old_duckdb_version` | string | _(all supported)_ | Test against a
specific old DuckDB version (e.g. `v1.2.0`). If not set, tests all
supported versions. |
| `--new_cli_path` | string | `build/release/duckdb` | Path to the new
(current) DuckDB CLI binary. |
| `--test_pattern` | string | _(none)_ | Run only tests whose relative
path contains the given pattern. Forces sequential execution. |
| `--test_file` | string | _(none)_ | Path to a single test file to run.
|
| `--run_sequentially` | boolean | `False` | Run tests sequentially
instead of in parallel. |
| `--max_workers` | integer | `24` | Maximum number of worker threads
for parallel execution. |
| `--stop_on_failure` | boolean | `False` | Stop execution after the
first failure. |
| `--cleanup_bwc_directory` | boolean | `False` | Clean up the BWC
runtime directory. Use with `--dry_run=False` to actually delete files.
|
| `--dry_run` | boolean | `True` | When cleaning up, only log files that
would be deleted without actually deleting them. |

### Examples

Run against all supported versions (parallel, 24 workers):
```shell
python3 test/bwc/runner.py
```

Run against a specific old version:
```shell
python3 test/bwc/runner.py --old_duckdb_version=v1.2.0
```

Run a subset of tests matching a pattern:
```shell
python3 test/bwc/runner.py --old_duckdb_version=v1.2.0 --test_pattern=window
```

Use a custom-built CLI:
```shell
python3 test/bwc/runner.py --new_cli_path=./build/debug/duckdb --old_duckdb_version=v1.2.0
```

## Cache tools

Export cache after a run:
```shell
python3 test/bwc/export_cache.py --version v1.2.0 --output-dir ./cache
```

Update test-utils cache from a CI run:
```shell
python3 test/bwc/update_cache.py --run-id <run_id> --test-utils-dir ../test-utils
```

## Compile extension for current DuckDB version

Create `.github/config/extensions/test-utils.cmake` in the `duckdb`
repository with the following content:

```cmake
duckdb_extension_load(test_utils
  GIT_URL https://github.com/motherduckdb/test-utils
  GIT_TAG f90429ac336fef250d60ae27bd87813a53da4e7f
  # Or, for local use:
  # SOURCE_DIR "${EXTENSION_CONFIG_BASE_DIR}/../../../../test-utils"
)

include("${EXTENSION_CONFIG_BASE_DIR}/../in_tree_extensions.cmake")
```

Then compile:

```shell
make release EXTENSION_CONFIGS=.github/config/extensions/test-utils.cmake
```

## Runtime directory structure

All intermediary files are stored under `duckdb_unittest_tempdir/bwc/`
relative to the repository root:

```
duckdb_unittest_tempdir/bwc/
├── specs/{version}/
│   ├── test/                        # Downloaded test files (cloned from duckdb repo at tag)
│   └── data/                        # Downloaded data fixtures
├── runtime/{version}/
│   └── {test_category}/{test_name}/
│       ├── {test_name}.sql              # Generated query file with bwc_tag annotations
│       ├── {test_name}.plan.bin         # Serialized execution plans (cacheable)
│       ├── {test_name}.{version}.result.bin  # Old version results (cacheable)
│       ├── {test_name}.new.result.bin   # New version results
│       ├── output/                      # Test output directory (working dir for CLI)
│       ├── fixtures/                    # Generated fixture databases (cacheable)
│       ├── data -> ../../specs/{version}/data   # Symlink
│       └── test -> ../../specs/{version}/test   # Symlink
├── reports/
│   └── test_report_{versions}_{timestamp}.duckdb  # Result database
└── tests_summary_{version}.txt          # Per-version summary (used by CI)
```

A new temporary folder is created for each run. The runner does not
remove it after the run, so one can replay eventual failures. The runner
also writes a `skip` file and adds an entry for each successful test
suite; subsequent runs will skip that suite.

Upstream-Commit: duckdb/duckdb@4c00426