{{ message }}
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
feat(storage): merge main into async branch#1741
Closed
googlyrahman wants to merge 57 commits into
Closed
Conversation
…upload (googleapis#1654) feat: send entire object checksum in the final api call of resumable upload fixes b/461994245 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
feat: Support urllib3 >= 2.6.0 **Context**: * This library implements a custom decoders ( `_GzipDecoder` , `_BrotliDecoder` ) which inherit from `urllib3.response.ContentDecoder` * Interface of `urllib3.response.ContentDecoder` was changed in [2.6.0](https://urllib3.readthedocs.io/en/stable/changelog.html#id1) to fix security vulnerability for highly compressed data reads. (Decompression bombs) Hence we need to change our interfaces as well. **Changes** * Add `max_length` param on decompress method, provide default value of -1 (same as urllib3's decompress) * Provide backwards compatibility ( ie urllib3 <= 2.5.0)
🤖 I have created a release *beep* *boop* --- ## [3.7.0](googleapis/python-storage@v3.6.0...v3.7.0) (2025-12-09) ### Features * Auto enable mTLS when supported certificates are detected ([googleapis#1637](googleapis#1637)) ([4e91c54](googleapis@4e91c54)) * Send entire object checksum in the final api call of resumable upload ([googleapis#1654](googleapis#1654)) ([ddce7e5](googleapis@ddce7e5)) * Support urllib3 >= 2.6.0 ([googleapis#1658](googleapis#1658)) ([57405e9](googleapis@57405e9)) ### Bug Fixes * **bucket:** Move blob fails when the new blob name contains characters that need to be url encoded ([googleapis#1605](googleapis#1605)) ([ec470a2](googleapis@ec470a2)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com>
chore: add gcs-fs as CODEOWNERS
fix: close write object stream always. otherwise Task will remain for long time until GC kills it and it'll throw this `"Task was destroyed but it is pending!"`
…oogleapis#1636) fix(experimental): no state lookup while opening bidi-write stream
This PR adds support for Python 3.14 to the library. Key changes include: - Updating `setup.py` to include the Python 3.14 classifier. - Updating `testing/constraints-3.14.txt` to specify `grpcio >= 1.75.1`. - Updating `noxfile.py` to include 3.14 sessions and set default to 3.14. - Updating `.github/sync-repo-settings.yaml` to include 3.14 in required checks. Tests were run locally using the `python-multi` Docker image for Python 3.9 and 3.14. Some unit tests failed due to environment issues within the container, but these are not expected to occur in the CI environment. --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
…nt (googleapis#1668) skip failing samples due to public access prevention enforcement. More Details on b/469643064
chore: add system test for cloud path
chore: add system 3.9 tests as required
feat: expose persisted size in MRD (MultiRangeReader)
fix: add system test for opening with read_handle
chore: Prevent OS Login key accumulation in e2e tests
feat: compute chunk wise checksum for bidi_writes and send it via BidiWriteObjectRequest As a part of this change, also did a small refactoring * Moved the precondition check to __utils.py_ file
chore: Add Sys test large obj
feat: make flush size configurable
chore: skip kms tests until b/470276398 --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…nning zonal system tests (googleapis#1691) chore: optimization to reduce number of open TCP connections while running zonal system tests 1. Increase `ulimit -n 10000` before ssh'ing into the VM where system tests for zonal buckets are running. 2. Delete `mrd` and `writer` instance and trigger `gc.collect()` ( this alone should suffice but increasing doing the above optimization to avoid future issues.
feat: implement `append_from_file`
chore:delete topic after creation part of b/470069573
…is#1694) Revert "skip notification tests until b/470069573 is fixed"
- Add bidi stream retry manager. It will the main class that would be responsible for retries. - Integrate MRD with bidi stream retry manager and reads resumption strategy - Added conformance tests for all error scenarios and smart resumption for bidi reads - Refactored MRD's `download_ranges()` method, which will now use retry manager's `execute()` method to send the requests - Modified the logic to parse and handle the `BidiReadObjectRedirectedError`. - Modified cloud build configuration to also pull the latest changes of the current PR. - Formatted few more files using black formatter
…h generation (googleapis#1698) chore(tests): add test for reading unfinalized appendable objects with generation
Adding writes resumption strategy which will be used for error handling of bidi writes operation.
…#1699) Earlier the last chunk was being flushed while calling the close() method. Now it will be done inside the append method itself.
Bump the current version in `.librarian/state.yaml` to `3.7.0` to cater for googleapis#1621 which was released on December 9 2025. This will address the issue in googleapis#1701 where librarian sets the version to `3.7.0` which has already been released. The reason that this manual PR is needed is because the release on December 9th used a different tool `release-please` which does not update the `.librarian/state.yaml` file.
) PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v0.7.0 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91 <details><summary>google-cloud-storage: 3.8.0</summary> ## [3.8.0](googleapis/python-storage@v3.7.0...v3.8.0) (2026-01-13) ### Features * expose persisted size in mrd (googleapis#1671) ([0e2961b](googleapis@0e2961be)) * implement &googleapis#34;append_from_file&googleapis#34; (googleapis#1686) ([1333c95](googleapis@1333c956)) * compute chunk wise checksum for bidi_writes (googleapis#1675) ([139390c](googleapis@139390cb)) * flush the last chunk in append method (googleapis#1699) ([89bfe7a](googleapis@89bfe7a5)) * add write resumption strategy (googleapis#1663) ([a57ea0e](googleapis@a57ea0ec)) * add bidi stream retry manager. (googleapis#1632) ([d90f0ee](googleapis@d90f0ee0)) * make flush size configurable (googleapis#1677) ([f7095fa](googleapis@f7095faf)) ### Bug Fixes * no state lookup while opening bidi-write stream ([2d5a7b1](googleapis@2d5a7b16)) * no state lookup while opening bidi-write stream (googleapis#1636) ([2d5a7b1](googleapis@2d5a7b16)) * close write object stream always (googleapis#1661) ([4a609a4](googleapis@4a609a4b)) * add system test for opening with read_handle (googleapis#1672) ([6dc711d](googleapis@6dc711da)) </details> --------- Co-authored-by: Victor Chudnovsky <vchudnov@google.com>
…quests in async streams. Gracefully close streams. (googleapis#1700) fix(experimental): implement requests_done method to signal end of requests in async streams. Gracefully close streams.
…ng existing objects and add `is_stream_open` support (googleapis#1709) feat(ZonalBuckets): add support for `generation=0` to prevent overwriting existing objects feat(ZonalBuckets): add `is_stream_open` property to AsyncAppendableObjectWriter for stream status check --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
…oogleapis#1705) feat(samples): add samples for appendable objects writes and reads --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
When `writer.close()` is called without setting finalize_on_close flag, we need to get two responses: 1) to get the persisted_size 2) eof response That's why added a check if the first response is not eof, then again receive the response from the stream.
chore: Add README for running zonal buckets samples
This method can be used to fetch the metadata of an object using the async grpc API.
) PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v1.0.2-0.20251119154421-36c3e21ad3ac Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:8e2c32496077054105bd06c54a59d6a6694287bc053588e24debe6da6920ad91 <details><summary>google-cloud-storage: 3.9.0</summary> ## [3.9.0](googleapis/python-storage@v3.8.0...v3.9.0) (2026-02-02) ### Features * update generation for MRD (googleapis#1730) ([08bc708](googleapis@08bc7082)) * add get_object method for async grpc client (googleapis#1735) ([0e5ec29](googleapis@0e5ec29b)) * Add micro-benchmarks for reads comparing standard (regional) vs rapid (zonal) buckets. (googleapis#1697) ([1917649](googleapis@1917649f)) * Add support for opening via `write_handle` and fix `write_handle` type (googleapis#1715) ([2bc15fa](googleapis@2bc15fa5)) * add samples for appendable objects writes and reads ([2e1a1eb](googleapis@2e1a1eb5)) * add samples for appendable objects writes and reads (googleapis#1705) ([2e1a1eb](googleapis@2e1a1eb5)) * add context manager to mrd (googleapis#1724) ([5ac2808](googleapis@5ac2808a)) * Move Zonal Buckets features of `_experimental` (googleapis#1728) ([74c9ecc](googleapis@74c9ecc5)) * add default user agent for grpc (googleapis#1726) ([7b31946](googleapis@7b319469)) * expose finalized_time in blob.py applicable for GET_OBJECT in ZB (googleapis#1719) ([8e21a7f](googleapis@8e21a7fe)) * expose `DELETE_OBJECT` in `AsyncGrpcClient` (googleapis#1718) ([c8dd7a0](googleapis@c8dd7a0b)) * send `user_agent` to grpc channel (googleapis#1712) ([cdb2486](googleapis@cdb2486b)) * integrate writes strategy and appendable object writer (googleapis#1695) ([dbd162b](googleapis@dbd162b3)) * Add micro-benchmarks for writes comparing standard (regional) vs rapid (zonal) buckets. (googleapis#1707) ([dbe9d8b](googleapis@dbe9d8b8)) * add support for `generation=0` to avoid overwriting existing objects and add `is_stream_open` support (googleapis#1709) ([ea0f5bf](googleapis@ea0f5bf8)) * add support for `generation=0` to prevent overwriting existing objects ([ea0f5bf](googleapis@ea0f5bf8)) * add `is_stream_open` property to AsyncAppendableObjectWriter for stream status check ([ea0f5bf](googleapis@ea0f5bf8)) ### Bug Fixes * receive eof while closing reads stream (googleapis#1733) ([2ef6339](googleapis@2ef63396)) * update write handle on every recv() (googleapis#1716) ([5d9fafe](googleapis@5d9fafe1)) * implement requests_done method to signal end of requests in async streams. Gracefully close streams. (googleapis#1700) ([6c16079](googleapis@6c160794)) * implement requests_done method to signal end of requests in async streams. Gracefully close streams. ([6c16079](googleapis@6c160794)) * instance grpc client once per process in benchmarks (googleapis#1725) ([721ea2d](googleapis@721ea2dd)) * Fix formatting in setup.py dependencies list (googleapis#1713) ([cc4831d](googleapis@cc4831d7)) * Change contructors of MRD and AAOW AsyncGrpcClient.grpc_client to AsyncGrpcClient (googleapis#1727) ([e730bf5](googleapis@e730bf50)) </details>
Automated: Migrate {target_path} from gsutil to gcloud storage
This CL is part of the on going effort to migrate from the legacy
`gsutil` tool to the new and improved `gcloud storage` command-line
interface.
`gcloud storage` is the recommended and modern tool for interacting with
Google Cloud Storage, offering better performance, unified
authentication, and a more consistent command structure with other
`gcloud` components. 🚀
### Automation Details
This change was **generated automatically** by an agent that targets
users of `gsutil`.
The transformations applied are based on the [gsutil to gcloud storage
migration guide](http://go/gsutil-gcloud-storage-migration-guide).
### ⚠️ Action Required: Please Review and Test Carefully
While we have based the automation on the migration guide, every use
case is unique.
**It is crucial that you thoroughly test these changes in environments
appropriate to your use-case before merging.**
Be aware of potential differences between `gsutil` and `gcloud storage`
that could impact your workflows.
For instance, the structure of command output may have changed,
requiring updates to any scripts that parse it. Similarly, command
behavior can differ subtly; the `gcloud storage rsync` command has a
different file deletion logic than `gsutil rsync`, which could lead to
unintended file deletions.
Our migration guides can help guide you through a list of mappings and
some notable differences between the two tools.
Standard presubmit tests are run as part of this CL's workflow. **If you
need to target an additional test workflow or require assistance with
testing, please let us know.**
Please verify that all your Cloud Storage operations continue to work as
expected to avoid any potential disruptions in production.
### Support and Collaboration
The `GCS CLI` team is here to help! If you encounter any issues, have a
complex use case that this automated change doesn't cover, or face any
other blockers, please don't hesitate to reach out.
We are happy to work with you to test and adjust these changes as
needed.
**Contact:** `gcs-cli-hyd@google.com`
We appreciate your partnership in this important migration effort!
#gsutil-migration
Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com>
Add a warning that generation_number will be deprecated in the next major release.
The PR sync the main and async branch. --------- Co-authored-by: Chandra Shekhar Sirimala <chandrasiri@google.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com> Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Adds async credential wrapper for async client. Context: - The standard google auth credentials are currently synchronous, and it's asynchronous credentials classes are either not available or marked private. - Credential retrieval and refreshing will remain synchronous under the hood. Rationale: As authentication tokens typically possess an expiration lifetime of one hour, the blocking time required for token fetching occurs infrequently. The performance impact of blocking (or utilizing a separate thread for offloading) once per hour is deemed negligible when weighed against the considerable engineering cost of developing and maintaining a asynchronous authentication.
…leapis#1655) This PR introduces an abstract class to support the upcoming async client for the Python SDK. This refactor defines the public interface and enables code sharing between sync and async client, without introducing new logic.
…oogleapis#1696) 1. Add async client implementation. 2. Add AsyncHTTPIterator deriving from google.api_core.page_iterator_async.AsyncIterator as an alternative to google.api_core.page_iterator.HTTPIterator The AsyncHTTPIterator doesn't exists, and hence needs to be implemented.
product-auto-label
Bot
added
size: xl
Contributor
googlyrahman
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request is a significant step forward, primarily focused on promoting the asyncio features from experimental to a stable API surface. This includes moving modules, adding deprecation warnings, and introducing a new BaseClient to share logic between the sync and async clients, which is a great architectural improvement. The PR also includes updates to CI/CD configurations to support newer Python versions and adds a comprehensive suite of new system, conformance, and performance tests.
My review has identified a couple of areas for improvement:
- The
CHANGELOG.mdhas some duplicate and redundant entries that could be cleaned up for clarity. - A system test for data corruption in resumable uploads appears to have been removed, which could be risky if the behavior is not tested elsewhere.
Overall, this is a solid set of changes that improves the library's structure and test coverage.
I am having trouble creating individual review comments. Click here to see my feedback.
tests/resumable_media/system/requests/test_upload.py (375-395)
The test test_resumable_upload_with_bad_checksum has been removed. This test was important as it verified that a DataCorruption exception is raised when the checksum of an uploaded file does not match the server's checksum. Removing a test for data integrity validation is risky. Was this intentional? If this scenario is not covered by another test, it would be safer to restore this test to prevent potential regressions in data corruption handling.
CHANGELOG.md (24-39)
This section of the changelog has some duplicate and redundant entries that could be cleaned up for better readability:
- Lines 24-25: These are duplicates for the same change (
add samples for appendable objects writes and reads). - Lines 26-28: These three items all come from the same commit (
ea0f5bf...). They could be consolidated into a single, more descriptive entry like: "add support forgeneration=0to avoid overwriting existing objects and addis_stream_openproperty for stream status check (#1709)". - Lines 38-39: These are also duplicates for the same change (
implement requests_done method...).
Consolidating these would make the release notes clearer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge main into async branch