Add support for UV dependency management (#2601) · apache/iceberg-python@75d78c1 · GitHub
Skip to content

Commit 75d78c1

Browse files
Drew Gallardokevinjqliu
Drew Gallardo
and
kevinjqliu
authored
Add support for UV dependency management (#2601)
Closes #2553 # Rationale for this change During the release process cleanup, I also had prototyped a migration from Poetry to UV. I used the https://github.com/mkniewallner/migrate-to-uv package to migrate to UV and ran a release workflow and some others like doc serve for validation. This PR also migrates the Makefile to use UV. I'll still need to upgrade the docs to reflect the UV contribution workflow. I think it would be a good start to push up as a prototype, and have others checkout and give it a try. ## Build System Changes ### Build Backend UV doesn't have a Poetry-equivalent build backend with build hooks, so I've switched to __setuptools__ with native Cython support for the avro changes. ``` [build-system] requires = ["setuptools>=80", "wheel", "Cython>=3.0.0"] build-backend = "setuptools.build_meta" ``` ### Packaging for release __PEP 639 License Compliance__ By going to the test above the license is now properly included in the release metadata ```toml [project] license = "Apache-2.0" # SPDX identifier license-files = ["LICEN[CS]E*", "NOTICE*"] # Automatic inclusion ``` ### Build Verification __Release builds verified:__ ```bash uv build --sdist # Source distribution with correct metadata uv build --wheel # Wheel with compiled Cython extensions check-wheel-contents dist/ # Quality checks pass ``` __Wheel vs Poetry:__ The migration addresses a packaging issue where Poetry was building non-standard "fat" wheels. Binary comparison using diff revealed significant differences in wheel structure between Poetry and UV builds. ``` diff -rq uv-wheel poetry-wheel Only in poetry-wheel/pyiceberg/avro: decoder_basic.c <-- built into .so so not needed :) Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-310-darwin.so Only in poetry-wheel/pyiceberg/avro: decoder_fast.cpython-311-darwin.so Only in poetry-wheel: NOTICE Files uv-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so and poetry-wheel/pyiceberg/avro/decoder_fast.cpython-312-darwin.so differ ``` Poetry's wheels tagged for a specific Python version (e.g., `cp312-cp312`) contained compiled extensions for multiple Python versions simultaneously. The comparison shows that a Python 3.12 wheel included `decoder_fast.cpython-310-darwin.so` and `decoder_fast.cpython-311-darwin.so` alongside the correct binary. These wheels also included source files and placed NOTICE at the wheel root instead of `dist-info/licenses/`. The new build process with `cibuildwheel` creates standard PyPI-compliant wheels where each wheel contains only its corresponding Python version's binary. This reduces wheel size while ensuring proper PEP 639 license file placement in `dist-info/licenses/`. Test builds are deployed also can be checked against the repo above. ## Are these changes tested? Yes - [x] `make install` Environment setup and dependency installation - [x] `make test` - Unit tests - [x] `make test-integration` - Integration tests with all extras - [x] `make lint` - Linting and code quality checks - [x] `make doc-install && make doc-serve` - Documentation ## Are there any user-facing changes? No user facing changes. Contributors will need to use UV instead of Poetry for development, but all make targets remain the same. So not really? --------- Co-authored-by: Kevin Liu <kevinjqliu@users.noreply.github.com>
1 parent 35a5f2d commit 75d78c1

16 files changed

Lines changed: 5560 additions & 7138 deletions

‎.github/dependabot.yml‎

Lines changed: 1 addition & 1 deletion

‎.github/workflows/nightly-pypi-build.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,13 @@ jobs:
3939
with:
4040
python-version: 3.12
4141

42-
- name: Install Poetry
43-
run: make install-poetry
42+
- name: Install UV
43+
uses: astral-sh/setup-uv@v6
4444

4545
- name: Set version
4646
id: set-version
4747
run: |
48-
CURRENT_VERSION=$(poetry version --short)
48+
CURRENT_VERSION=$(uv version --short)
4949
TIMESTAMP=$(date +%Y%m%d%H%M%S)
5050
echo "VERSION=${CURRENT_VERSION}.dev${TIMESTAMP}" >> "$GITHUB_OUTPUT"
5151

‎.github/workflows/pypi-build-artifacts.yml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -46,18 +46,18 @@ jobs:
4646
3.11
4747
3.12
4848
49-
- name: Install poetry
50-
run: make install-poetry
49+
- name: Install UV
50+
uses: astral-sh/setup-uv@v6
5151

5252
- name: Set version with RC
5353
env:
5454
VERSION: ${{ inputs.VERSION }}
55-
run: python -m poetry version "${{ env.VERSION }}"
55+
run: uv version "${{ env.VERSION }}"
5656

5757
# Publish the source distribution with the version that's in
5858
# the repository, otherwise the tests will fail
5959
- name: Compile source distribution
60-
run: python3 -m poetry build --format=sdist
60+
run: uv build --sdist
6161
if: startsWith(matrix.os, 'ubuntu')
6262

6363
- name: Build wheels
@@ -77,6 +77,7 @@ jobs:
7777
# Skip free-threaded (PEP 703) builds until we evaluate decoder_fast support
7878
CIBW_SKIP: "cp3*t-*"
7979

80+
8081
- name: Add source distribution
8182
if: startsWith(matrix.os, 'ubuntu')
8283
run: ls -lah dist/* && cp dist/* wheelhouse/

‎.github/workflows/python-ci-docs.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,8 @@ jobs:
3939
- uses: actions/setup-python@v6
4040
with:
4141
python-version: 3.12
42-
- name: Install poetry
43-
run: make install-poetry
42+
- name: Install UV
43+
uses: astral-sh/setup-uv@v6
4444
- name: Install
4545
run: make docs-install
4646
- name: Build docs

‎.github/workflows/python-ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,8 @@ jobs:
5454
- uses: actions/setup-python@v6
5555
with:
5656
python-version: ${{ matrix.python }}
57-
- name: Install poetry
58-
run: make install-poetry
57+
- name: Install UV
58+
uses: astral-sh/setup-uv@v6
5959
- name: Install system dependencies
6060
run: sudo apt-get update && sudo apt-get install -y libkrb5-dev # for kerberos
6161
- name: Install

‎.github/workflows/python-release-docs.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,8 @@ jobs:
3434
- uses: actions/setup-python@v6
3535
with:
3636
python-version: ${{ matrix.python }}
37-
- name: Install poetry
38-
run: make install-poetry
37+
- name: Install UV
38+
uses: astral-sh/setup-uv@v6
3939
- name: Install docs
4040
run: make docs-install
4141
- name: Build docs

‎.github/workflows/python-release.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -97,20 +97,20 @@ jobs:
9797
with:
9898
python-version: 3.12
9999

100-
- name: Install Poetry
101-
run: make install-poetry
100+
- name: Install UV
101+
uses: astral-sh/setup-uv@v6
102102

103103
- name: Validate current pyiceberg version
104104
env:
105105
VERSION: ${{ needs.validate-inputs.outputs.VERSION }}
106106
run: |
107-
# Extract the current version from Poetry
108-
current_pyiceberg_version=$(poetry version --short)
109-
echo "Detected Poetry version: $current_pyiceberg_version"
107+
# Extract the current version from UV
108+
current_pyiceberg_version=$(uv version --short)
109+
echo "Detected UV version: $current_pyiceberg_version"
110110
111-
# Compare the input version with the Poetry version
111+
# Compare the input version with the UV version
112112
if [[ "$VERSION" != "$current_pyiceberg_version" ]]; then
113-
echo "Error: Input version ($VERSION) does not match the Poetry version ($current_pyiceberg_version)"
113+
echo "Error: Input version ($VERSION) does not match the UV version ($current_pyiceberg_version)"
114114
exit 1
115115
fi
116116

‎.github/workflows/svn-build-artifacts.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,13 @@ jobs:
4646
3.11
4747
3.12
4848
49-
- name: Install poetry
50-
run: make install-poetry
49+
- name: Install UV
50+
uses: astral-sh/setup-uv@v6
5151

5252
# Publish the source distribution with the version that's in
5353
# the repository, otherwise the tests will fail
5454
- name: Compile source distribution
55-
run: python3 -m poetry build --format=sdist
55+
run: uv build --sdist
5656
if: startsWith(matrix.os, 'ubuntu')
5757

5858
- name: Build wheels

‎MANIFEST.in‎

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,4 +15,22 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717

18-
graft src
18+
# Include Cython source files for building from source
19+
recursive-include pyiceberg *.pyx *.c
20+
21+
# Exclude generated Cython C file
22+
exclude pyiceberg/avro/decoder_fast.c
23+
24+
# Include test files in sdist
25+
recursive-include tests *
26+
27+
# Include development files
28+
include Makefile
29+
recursive-include dev *
30+
31+
# Exclude build artifacts
32+
global-exclude */__pycache__/*
33+
prune .venv
34+
prune build
35+
prune dist
36+
prune .pytest_cache

‎Makefile‎

Lines changed: 31 additions & 31 deletions

0 commit comments

Comments
 (0)