This readme explains what types of extensions there are in DuckDB and how to build them.
DuckDB extensions are libraries containing additional DuckDB functionality separate from the main codebase. These extensions can provide added functionality to DuckDB that can/should not live in DuckDB main code for various reasons. DuckDB extensions can be built in two ways. Firstly, they can be statically linked into DuckDBs executables (duckdb cli, unittest binary, benchmark runner binary, etc). Doing so will automatically make them available when using these binaries. Secondly, DuckDB has an extension loading mechanism to dynamically load extension binaries.
DuckDB Extensions can de divided into different types: In-tree extensions and out-of-tree extensions. These types refer to where the extensions live and who maintains them.
In-tree extensions are extensions that live in the main DuckDB repository. These extensions are considered fundamental to DuckDB and/or tie into to DuckDB so deeply that changes to DuckDB are expected to regularly break them. We aim to keep the amount of in-tree extensions to a minimum and strive to move extensions out-of-tree where possible.
Out-of-tree extensions live in separate repositories outside the main DuckDB repository. The reasons for moving extensions out-of-tree can vary. Firstly, moving extensions out of the main DuckDB code-base keeps the core DuckDB code smaller and less complex. Secondly, keeping extensions out-of-tree can be useful for licensing reasons.
There are two main types of OOTEs. Firstly, there are the DuckDB Managed OOTEs. These are distributed through the main
DuckDB CI. These extensions are signed using DuckDBs signing key and are maintained by the DuckDB team. Some examples are
the sqlite_scanner and postgres_scanner extensions. The DuckDB Managed OOTEs are distributed automatically with every
release of DuckDB. For the current list of extensions in this category check out .github/config/out_of_tree_extensions.cmake
Secondly, there are External OOTEs. Extensions in this category are not tied to the DuckDB CI, but instead their CI/CD runs in their own repository. The maintainer of the external OOTE repo is responsible for testing, distribution and making sure that an up-to-date version of the extension is available. Depending on who maintains the extension, these extensions may or may not be signed.
Under the hood, all types of extensions are built the same way, which is using the DuckDB's root CMakeLists.txt file as root CMake file
and passing the extensions that should be build to it. DuckDB has various methods to configure which extensions to build.
Additionally, we can configure for each extension how we want to build it: for example, whether to only
build the loadable extension, or also link the extension in the DuckDB binaries. There's different ways to load extensions
in DuckDB with various
The simplest way to specify which extensions to load is using the DUCKDB_EXTENSIONS variable. To specify which extensions
to build when making duckdb set the extensions variable to a ; separated list of extensions names. For example:
DUCKDB_EXTENSIONS='json;icu' makeMake builds that select extensions automatically use the synchronized extension build path. This includes
DUCKDB_EXTENSIONS, BUILD_EXTENSIONS, CORE_EXTENSIONS, the BUILD_<extension> shorthands,
BUILD_ALL_*, and EXTENSION_CONFIGS. Set DUCKDB_NEW_EXTENSION_BUILD=0 to temporarily use the legacy
build path instead.
The DUCKDB_EXTENSIONS variable is passed to a CMake variable BUILD_EXTENSIONS which can also be invoked directly:
cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_EXTENSIONS='parquet;icu;tpch;tpcds;fts;json'With the synchronized extension build path, the sync script and CMake use the same configuration
precedence: explicit EXTENSION_CONFIGS files first, then named BUILD_EXTENSIONS configs.
The first declaration of each extension wins, including SOURCE_DIR and DONT_BUILD
declarations. An extension project can therefore supply its own repository URLs and
GIT_TAG hashes without changing DuckDB's checked-in defaults:
EXTENSION_CONFIGS=/path/to/project/extension_config.cmake \
BUILD_EXTENSIONS='httpfs;avro;aws' make reldebugEXTENSION_CONFIGS is a semicolon-separated list; earlier files take precedence over later
files. Make passes it to CMake as DUCKDB_EXTENSION_CONFIGS.
To own the entire directory of named configs, set EXTENSION_CONFIG_BASE_DIR. For example,
the following reads httpfs.cmake and avro.cmake from the project's directory:
EXTENSION_CONFIG_BASE_DIR=/path/to/project/extension-configs \
BUILD_EXTENSIONS='httpfs;avro' make reldebugThe default is DuckDB's .github/config/extensions. A custom directory replaces that
lookup location; missing files do not fall back to DuckDB's pinned revisions. Both sync
and CMake resolve relative config paths from the DuckDB source directory. Absolute paths
are recommended for builds driven by another repository.
The sync script also accepts --extension-configs and --extension-config-base-dir.
CMake accepts -DEXTENSION_CONFIG_BASE_DIR=... or the environment variable. Build wrappers
must pass the same settings to both steps.
Direct CMake builds do not run the synchronization step automatically. Call the sync script first and set
DUCKDB_NEW_EXTENSION_BUILD=1, or use the Make wrapper for builds containing out-of-tree extensions.
Sync reads literal duckdb_extension_load declarations and follows absolute includes and
includes using ${CMAKE_CURRENT_LIST_DIR} or ${EXTENSION_CONFIG_BASE_DIR}. It does not
evaluate general CMake variables or conditionals; keep repository URLs and commit hashes
literal in configs used by sync. This controls extension Git revisions, not vcpkg registry
baselines.
Another way to specify building an extension is with the BUILD_<extension name> variables defined in the root
Makefile in this repository. For example, to build the JSON extension, simply run BUILD_JSON=1 make. These Makevars
should be added manually for each extension and are simply syntactic sugar around the DUCKDB_EXTENSIONS variable.
To have more control over how in-tree extensions are built, extension config files should be used. These config files are simply CMake files that are included by DuckDB's CMake build. There are 4 different places that will be searched for config files:
- The base configuration
extension/extension_config.cmake. The extensions specified here will be built every time DuckDB is built. This configuration is always loaded. - (Optional) The client specific extensions specification in
tools/*/duckdb_extension_config.cmake. These config specify which extensions are built and linked into each client. - (Optional) The local configuration file
extension/extension_config_local.cmakeThis is where you would specify extensions you need included in your local/custom/dev build of DuckDB. This file is gitignored and to be created by the developer. - (Optional) Additional configuration files passed to the
DUCKDB_EXTENSION_CONFIGSparameter. This can be used to point DuckDB to config files stored anywhere on the machine.
DuckDB will load these config files in reverse order and ignore subsequent calls to load an extension with the
same name. This allows overriding the base configuration of an extension by providing a different configuration
in the local config. For example, the parquet extension is built and statically linked into DuckDB because of these
lines in extension/extension_config.cmake:
duckdb_extension_load(parquet)
duckdb_extension_statically_link(parquet)Now say we want to build DuckDB with our custom parquet extension. We can achieve this creating the
extension/extension_config_local.cmake file and adding:
duckdb_extension_load(parquet
SOURCE_DIR /path/to/my/custom/parquet
)duckdb_extension_load only decides what is built. What the DuckDB targets (the shell, unittest, the library) link
statically is decided separately: by default the built extensions that the configs list with
duckdb_extension_statically_link(<extension_name>) (plus the ones named in BUILD_EXTENSIONS when building with
LINK_CORE_EXTENSIONS=1), or exactly STATICALLY_LINK_EXTENSIONS when that is set
(e.g. STATICALLY_LINK_EXTENSIONS='core_functions;parquet;json;icu;tpch' make). An extension that is built but not
linked is still available: automatic installs come from the build's own extension repository, so it is installed and
loaded the first time a query needs it.
The duckdb_extension_load function is used in the configuration files to specify how an extension should
be loaded. There are 3 different ways this can be done. For some examples, check out .github/config/*.cmake. These are
the configurations used in DuckDBs CI to select which extensions are built.
The simplest way to load an extension is just passing the extension name. This will automatically try to load the extension.
duckdb_extension_load(<extension_name>)This configuration of duckdb_extension_load will search the ./extension and ./extension_external directories for
extensions and attempt to load them if possible. Note that the extension_external directory does not exist but should
be created and populated with the out-of-tree extensions that should be built. Extensions based on the
extension-template should work out of the box using this automatic
loading when placed in the extension_external directory.
When extensions are located in a path or their project structure is different from that the
extension-template, the SOURCE_DIR and INCLUDE_DIR variables can
be used to tell DuckDB how to load the extension:
duckdb_extension_load(<extension_name>
SOURCE_DIR <absolute_path_to_extension_root>
(INCLUDE_DIR <absolute_path_to_extension_header>)
)Directly installing extensions from GitHub repositories is also supported. This will download the extension to the current cmake build directory and build it from there:
duckdb_extension_load(postgres_scanner
GIT_URL https://github.com/duckdb/postgres_scanner
GIT_TAG cd043b49cdc9e0d3752535b8333c9433e1007a48
)Because the sometimes you may want to override extensions set by other configurations, explicitly disabling extensions is
also possible using the DONT_BUILD flag. This will disable the extension from being built all together. For example, to build DuckDB without the parquet extension which is enabled by default, in extension/extension_config_local.cmake specify:
duckdb_extension_load(parquet DONT_BUILD)Note that this can also be done from the Makefile:
DUCKDB_EXTENSIONS='tpch;json' SKIP_EXTENSIONS=parquet makeresults in:
...
-- Building extension 'tpch' from '/Users/sam/Development/duckdb/extensions'
-- Building extension 'json' from '/Users/sam/Development/duckdb/extensions'
-- Extensions linked into DuckDB: tpch, json
-- Extensions explicitly skipped: parquet
...An extension can report its version through the EXT_VERSION_<NAME_UPPERCASE> macro, which DuckDB defines to
the extension version configured at build time:
std::string ParquetExtension::Version() const {
#ifdef EXT_VERSION_PARQUET
return EXT_VERSION_PARQUET;
#else
return "";
#endif
}By default the macro is defined for every source file of the extension, including any third party code it vendors. Since the version defaults to a git commit hash - DuckDB's for in-tree extensions, the extension's own otherwise - that puts a value which changes on every commit on all of their compiler command lines, forcing a recompile of the whole extension for a string that a single file reads.
An extension avoids that by declaring which of its sources actually read the macro, in its own CMakeLists.txt:
project(ParquetExtension)
# Before any subdirectory or target, which would otherwise inherit the directory-wide definition
set_extension_version_sources(parquet parquet_extension.cpp)Only the listed sources then get EXT_VERSION_<NAME>, and a version change recompiles those rather than
everything. It has to be called before the extension declares any targets or adds any subdirectories, because
those inherit the directory-wide definition at the point they are created. Out-of-tree extensions can use it
too; their version is their own commit hash, so it changes on every commit in their repository.
DuckDB extensions can use VCPKG to manage their dependencies. Check out the Extension Template for an example on how to set up vcpkg in extensions.
To build duckdb with multiple extensions that all use vcpkg, some extra steps are required. This is due to the fact that each extension will specify their own vcpkg.json manifest for their dependencies, but vcpkg allows only a single manifest. The workaround here is to merge the dependencies from the manifests of all extensions being built. This repo contains a script to do automatically perform this merge.
For example, lets say we want to create a DuckDB binary which has two extensions statically linked that each use vcpkg. The first step is to add the two extensions
to extension/extension_config_local.cmake:
duckdb_extension_load(extension_1
GIT_URL https://github.com/example/extension_1
GIT_TAG some_git_hash
)
duckdb_extension_load(extension_2
GIT_URL https://github.com/example/extension_2
GIT_TAG some_git_hash
)Now to merge the vcpkg.json manifests from these two extension run:
make extension_configurationThis will create a merged manifest in ./build/extension_configuration/vcpkg.json.
Next, run:
USE_MERGED_VCPKG_MANIFEST=1 VCPKG_TOOLCHAIN_PATH="/path/to/your/vcpkg/installation" makewhich will use the merged manifest to install all required dependencies, build extension_1 and extension_2, build DuckDB,
and finally link both extensions into DuckDB.
Outside of CMake, a program links one archive per extension it wants, compiles a generated static extension loader, and puts the engine archive last:
LINK_EXTENSIONS="parquet;json" make static_extension_loader # writes build/release/static_extension_loader.c
cc -I src/include main.c build/release/static_extension_loader.c libparquet_extension.a libjson_extension.a libduckdb_static.a -lstdc++Each extension archive carries a describe function, duckdb_extension_<name>_describe, that fills in a
duckdb_extension_descriptor (see duckdb_static_extension.h). The loader is plain C: it defines
duckdb_register_static_extensions(), which passes every describe function to duckdb_register_static_extension, and that also
pulls the extensions out of their archives. Call it before opening a database, from any language that can call C. A
C++ program that wants this to happen before main compiles extension/loader/static_extension_autoregister.cpp next to
the loader, into the program itself rather than into an archive, or the linker drops it. Without LINK_EXTENSIONS
the loader registers every extension archive in the build. A program can also skip the loader and call
duckdb_register_static_extension on the describe functions it declares itself. An archive whose describe function is
not registered contributes nothing, and a registered one whose archive is missing fails the link. CMake targets get all of this from
link_extension_libraries.
A loadable extension is the same archive linked as a shared library with its entry point exported, next to the engine archive when it is built to carry its own copy of DuckDB:
c++ -shared -o parquet.duckdb_extension libparquet_extension.a libduckdb_static.a -Wl,-exported_symbol,_parquet_duckdb_cpp_initThe describe function is never taken there, since nothing registers it.