iframe-proxy

AlexZMLyu · 2025-12-11T05:43:02Z

Bases batching on size rather than row count to avoid exceeding an internal 10MB limit. Also removes an obsolete assertion in the test.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

Bases batching on size rather than row count to avoid exceeding an internal 10MB limit. Also removes an obsolete assertion in the test.

gemini-code-assist · 2025-12-11T05:43:17Z

Summary of Changes

Hello @AlexZMLyu, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refactors the data batching mechanism within the BigQuery Storage Arrow sample. The primary goal is to prevent AppendRowsRequest payloads from exceeding BigQuery's internal 10MB size limit by implementing a size-aware batching strategy. This change improves the robustness and reliability of data ingestion when using PyArrow with BigQuery Storage.

Highlights

Batching Logic Update: The generate_write_requests function in the BigQuery Storage Arrow sample has been updated to base batching on the serialized size of PyArrow record batches rather than row count. This ensures that individual AppendRowsRequest payloads do not exceed the 10MB internal limit, using a conservative 7MB soft limit.
Improved Error Handling: A new check has been added to raise a ValueError if a single PyArrow batch containing just one row is larger than the maximum allowed request size (7MB), preventing potential issues with oversized individual records.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request updates the batching logic in the BigQuery Storage Arrow sample to be based on byte size instead of row count. This is a great improvement as it prevents requests from exceeding the 10MB size limit. The new implementation is more robust. I've suggested a small refactoring to reduce code duplication, which will improve maintainability.

- Changes query to SELECT DISTINCT int64_col to count unique rows. - Asserts the count is exactly TABLE_LENGTH, removing the allowance for extra rows from potential retries.

…ample - Updates batching logic to use serialized size to avoid exceeding API limits. - Ensures all rows in the PyArrow table are serialized for the request. - Includes enhancements for measuring serialized row sizes.

- Changed `generate_write_requests` to be a generator, yielding requests instead of returning a list. - Made `stream.send()` calls blocking by calling `future.result()` immediately, ensuring requests are sent sequentially.

AlexZMLyu · 2026-01-02T05:11:30Z

The test failures are in other PRs as well. https://github.com/googleapis/google-cloud-python/actions/runs/20482816437/job/58859390604

It is unrelated to my changes.

AlexZMLyu · 2026-01-06T23:25:45Z

PR created by the Librarian CLI to initialize a release. Merging this PR will auto trigger a release. Librarian Version: v0.0.0-20260210205451-43b900a12d48 Language Image: us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:f5426423676c75008c2135037e7b98f78cbb99f78b3c46fe043b6897be92d836 <details><summary>google-cloud-artifact-registry: v1.20.0</summary> ## [v1.20.0](google-cloud-artifact-registry-v1.19.0...google-cloud-artifact-registry-v1.20.0) (2026-02-12) ### Features * add DIRSUM_SHA256 hash type (PiperOrigin-RevId: 865552557) ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-bigquery-storage: v2.36.1</summary> ## [v2.36.1](google-cloud-bigquery-storage-v2.36.0...google-cloud-bigquery-storage-v2.36.1) (2026-02-12) ### Bug Fixes * fix mypy (#15531) ([f2536ae](f2536ae3)) ### Documentation * Update BigQuery Storage Arrow samples batching logic (#14961) ([45d4cee](45d4cee9)) </details> <details><summary>google-cloud-confidentialcomputing: v0.8.0</summary> ## [v0.8.0](google-cloud-confidentialcomputing-v0.7.0...google-cloud-confidentialcomputing-v0.8.0) (2026-02-12) ### Features * add Nvidia Attestation proto message with its relevant fields and ConfidentialGke options (PiperOrigin-RevId: 866189208) ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-databasecenter: v0.5.0</summary> ## [v0.5.0](google-cloud-databasecenter-v0.4.0...google-cloud-databasecenter-v0.5.0) (2026-02-12) ### Features * Adding Method QueryIssues of Database Center API v1beta (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) ### Documentation * A comment for field `order_by` in message `.google.cloud.databasecenter.v1beta.AggregateFleetRequest` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for field `order_by` in message `.google.cloud.databasecenter.v1beta.QueryDatabaseResourceGroupsRequest` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for message `AggregateFleetResponse` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for field `filter` in message `.google.cloud.databasecenter.v1beta.AggregateFleetRequest` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for field `filter` in message `.google.cloud.databasecenter.v1beta.AggregateIssueStatsRequest` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for field `filter` in message `.google.cloud.databasecenter.v1beta.QueryDatabaseResourceGroupsRequest` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) * A comment for field `maintenance_version` in message `.google.cloud.databasecenter.v1beta.MaintenanceInfo` is changed (PiperOrigin-RevId: 868397264) ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-dataform: v0.9.0</summary> ## [v0.9.0](google-cloud-dataform-v0.8.0...google-cloud-dataform-v0.9.0) (2026-02-12) ### Features * Update GCP Client Libraries in v1beta1 to support Folders, TeamFolders, and other relevant APIs The v1beta1 API now includes support for Folders and TeamFolders, allowing users to organize repositories and files hierarchically and manage access controls. New Features: - Added TeamFolder resource and methods: CreateTeamFolder, GetTeamFolder, UpdateTeamFolder, DeleteTeamFolder QueryTeamFolderContents to list folder contents. SearchTeamFolders to search for TeamFolders. - Added Folder resource and methods: CreateFolder, GetFolder, UpdateFolder, DeleteFolder QueryFolderContents to list folder contents. Added MoveFolder to move Folders between TeamFolders, other Folders, or the user root folder. - Added MoveRepository to move Repositories between TeamFolders, Folders, or the user root folder. - Added QueryUserRootContents to list contents of a user&#39;s root folder. Repository resource now includes containing_folder and team_folder_name fields to indicate its location within the folder hierarchy. - IAM methods (GetIamPolicy, SetIamPolicy, TestIamPermissions) now support Folder and TeamFolder resources for access control management ([5371e8e](5371e8e9)) ### Documentation * A comment for field `force` in message `.google.cloud.dataform.v1beta1.DeleteRepositoryRequest` is changed PiperOrigin-RevId: 868182714 ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-dialogflow: v2.46.0</summary> ## [v2.46.0](google-cloud-dialogflow-v2.45.0...google-cloud-dialogflow-v2.46.0) (2026-02-12) ### Features * added a new field StreamingAnalyzeContentRequest.output_multiple_utterances (PiperOrigin-RevId: 862959083) ([a6b40b3](a6b40b31)) </details> <details><summary>google-cloud-discoveryengine: v0.17.0</summary> ## [v0.17.0](google-cloud-discoveryengine-v0.16.0...google-cloud-discoveryengine-v0.17.0) (2026-02-12) ### Features * add CrowdingSpec to SearchRequest to set crowding settings (PiperOrigin-RevId: 868345232) ([b483e92](b483e921)) * add Natural Language Query Understanding fields and messages to DiscoveryEngine request, response, and datastore messages (PiperOrigin-RevId: 868345232) ([b483e92](b483e921)) * add output-only field SemanticState to SearchResponse. (PiperOrigin-RevId: 868345232) ([b483e92](b483e921)) ### Documentation * other misc documentation updates Clients can specify Natural Language Query Understanding-related fields, as well as CrowdingSpec. These features are available in the V1 APIs. (PiperOrigin-RevId: 868345232) ([b483e92](b483e921)) * update documentation for search and data store NLQ features (PiperOrigin-RevId: 868345232) ([b483e92](b483e921)) </details> <details><summary>google-cloud-documentai: v3.10.0</summary> ## [v3.10.0](google-cloud-documentai-v3.9.0...google-cloud-documentai-v3.10.0) (2026-02-12) ### Features * A new field `document_prompt` is added to message `.google.cloud.documentai.v1beta3.DocumentSchema` (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * A new field `document_type` is added to message `.google.cloud.documentai.v1beta3.ImportDocumentsRequest` (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * A new field `revisions` is added to message `.google.cloud.documentai.v1beta3.Evaluation` (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * The method `ReviewDocument` in `.google.cloud.documentai.v1beta3.DocumentProcessorService` is deprecated (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * A new field `enable_table_split` is added to message `.google.cloud.documentai.v1beta3.OcrConfig.LayoutParsingParams` (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * Added new messages `Documents` and `RawDocuments` for inline document input (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) * The field `skip_human_review` in messages `.google.cloud.documentai.v1beta3.ProcessRequest` and `.google.cloud.documentai.v1beta3.BatchProcessRequest` is deprecated (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) ### Bug Fixes * Removed the `SpannerIndexingConfig` message and the `spanner_indexing_config` field from `.google.cloud.documentai.v1beta3.Dataset` BREAKING CHANGE: The `SpannerIndexingConfig` message and the `spanner_indexing_config` field within the `Dataset` message have been removed. Client code referencing these will need to stop referencing these in case of an error (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) ### Documentation * Updated comments for various fields and messages (PiperOrigin-RevId: 866382481) ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-gke-hub: v1.21.0</summary> ## [v1.21.0](google-cloud-gke-hub-v1.20.0...google-cloud-gke-hub-v1.21.0) (2026-02-12) ### Features * Add GKE Hub teams scope and RBAC GAPIC (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) ### Documentation * A comment for field `cancel_requested` in message `.google.cloud.gkehub.v1.OperationMetadata` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `state` in message `.google.cloud.gkehub.v1.CommonFeatureState` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for enum value `ENABLING` in enum `State` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for message `CommonFeatureState` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `state` in message `.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `membership_states` in message `.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for enum value `DISABLING` in enum `State` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `request_id` in message `.google.cloud.gkehub.v1.UpdateFeatureRequest` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `spec` in message `.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for message `Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `request_id` in message `.google.cloud.gkehub.v1.CreateFeatureRequest` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for enum value `ACTIVE` in enum `State` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `membership_specs` in message `.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for message `FeatureResourceState` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) * A comment for field `labels` in message `.google.cloud.gkehub.v1.Feature` is changed (PiperOrigin-RevId: 865111649) ([99a1284](99a1284f)) </details> <details><summary>google-cloud-storage-control: v1.10.0</summary> ## [v1.10.0](google-cloud-storage-control-v1.9.0...google-cloud-storage-control-v1.10.0) (2026-02-12) ### Features * add a DeleteFolderRecursive API definition (PiperOrigin-RevId: 866471251) ([5371e8e](5371e8e9)) </details> <details><summary>google-cloud-visionai: v0.4.0</summary> ## [v0.4.0](google-cloud-visionai-v0.3.0...google-cloud-visionai-v0.4.0) (2026-02-12) ### Bug Fixes * An existing default host `visionai.googleapis.com` is changed to `warehouse-visionai.googleapis.com` in service `Warehouse` (PiperOrigin-RevId: 866154811) ([5371e8e](5371e8e9)) ### Documentation * A comment for field `relevance` in message `.google.cloud.visionai.v1.SearchResultItem` is changed (PiperOrigin-RevId: 866154811) ([5371e8e](5371e8e9)) * A comment for field `page_size` in message `.google.cloud.visionai.v1.SearchIndexEndpointRequest` is changed (PiperOrigin-RevId: 866154811) ([5371e8e](5371e8e9)) </details> <details><summary>google-maps-solar: v0.4.0</summary> ## [v0.4.0](google-maps-solar-v0.3.0...google-maps-solar-v0.4.0) (2026-02-12) ### Documentation * Add satellite imagery to `rgb_url` description (PiperOrigin-RevId: 866492024) ([5371e8e](5371e8e9)) * Add pre-GA details for Experiments enum (PiperOrigin-RevId: 866492024) ([5371e8e](5371e8e9)) </details> <details><summary>Bulk Changes</summary> * chore: librarian update image pull request: 20260205T214646Z (#15532) ([3654fe7](3654fe76)) Libraries: google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-documentai,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar * chore: sort repo metadata by key (#15529) ([7da7768](7da7768c)) Libraries: google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-documentai,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar * chore: librarian update image pull request: 20260123T001116Z (#15481) ([871cdee](871cdeeb)) Libraries: google-cloud-artifact-registry,google-cloud-bigquery-storage,google-cloud-confidentialcomputing,google-cloud-databasecenter,google-cloud-dataform,google-cloud-dialogflow,google-cloud-discoveryengine,google-cloud-gke-hub,google-cloud-storage-control,google-cloud-visionai,google-maps-solar </details>

Fix: Update BigQuery Storage Arrow samples batching logic

a975186

Bases batching on size rather than row count to avoid exceeding an internal 10MB limit. Also removes an obsolete assertion in the test.

AlexZMLyu requested review from a team and agrawal-siddharth December 11, 2025 05:43

gemini-code-assist Bot reviewed Dec 11, 2025

View reviewed changes

Comment thread packages/google-cloud-bigquery-storage/samples/pyarrow/append_rows_with_arrow.py

AlexZMLyu added 7 commits December 11, 2025 05:58

Chore: Apply manual formatting to Arrow samples

186d6a6

Chore: Apply manual Black-style formatting

e9a7007

Chore: Manual formatting adjustments

5a91e4e

Refactor: Extract AppendRowsRequest creation to helper

84d0fc1

Fix AttributeError in append_rows_with_arrow.py

eb97b8f

Update append_rows_with_arrow.py

593ac97

Update append_rows_with_arrow.py

1eb8266

GaoleMeng reviewed Dec 15, 2025

View reviewed changes

Comment thread packages/google-cloud-bigquery-storage/samples/pyarrow/append_rows_with_arrow.py

AlexZMLyu added 2 commits December 16, 2025 00:11

Fix: Update verify_result in pyarrow sample

2239db8

- Changes query to SELECT DISTINCT int64_col to count unique rows. - Asserts the count is exactly TABLE_LENGTH, removing the allowance for extra rows from potential retries.

Fix: Update PyArrow serialization in append_rows_with_arrow.py

b510cec

parthea changed the title ~~Fix: Update BigQuery Storage Arrow samples batching logic~~ docs(samples): Update BigQuery Storage Arrow samples batching logic Dec 17, 2025

parthea requested a review from GaoleMeng December 17, 2025 16:16

parthea assigned GaoleMeng Dec 17, 2025

AlexZMLyu added 4 commits December 17, 2025 19:41

feat: Measure request generation time in pyarrow sample

f581b33

Refactor: Use generator for request creation and block on send

501e993

- Changed `generate_write_requests` to be a generator, yielding requests instead of returning a list. - Made `stream.send()` calls blocking by calling `future.result()` immediately, ensuring requests are sent sequentially.

samples: reformat append_rows_with_arrow.py

ec49ec5

GaoleMeng reviewed Dec 18, 2025

View reviewed changes

Comment thread packages/google-cloud-bigquery-storage/samples/pyarrow/append_rows_with_arrow.py

AlexZMLyu and others added 2 commits January 1, 2026 21:00

enhance unit test for write request generation

2464fd3

Merge branch 'googleapis:main' into main

fc14e4a

GaoleMeng previously approved these changes Jan 6, 2026

View reviewed changes

Comment thread packages/google-cloud-bigquery-storage/samples/pyarrow/append_rows_with_arrow.py Outdated

fix: update comment to reference correct variable name

fa73c4b

AlexZMLyu dismissed GaoleMeng’s stale review via fa73c4b January 6, 2026 23:49

fix: logic update in generate_write_requests and updated tests

8867ecd

GaoleMeng previously approved these changes Jan 7, 2026

View reviewed changes

GaoleMeng removed the request for review from agrawal-siddharth January 7, 2026 03:00

fix: update assertion to match new request generation logic

613e330

AlexZMLyu dismissed GaoleMeng’s stale review via 613e330 January 7, 2026 03:23

Merge remote-tracking branch 'upstream/main'

657c6e6

GaoleMeng approved these changes Jan 8, 2026

View reviewed changes

AlexZMLyu closed this Jan 8, 2026

AlexZMLyu reopened this Jan 8, 2026

Linchin merged commit 45d4cee into googleapis:main Jan 8, 2026
45 checks passed

cloud-sdk-librarian-robot mentioned this pull request Feb 12, 2026

chore: librarian release pull request: 20260212T081040Z #15564

Merged

Sunbelt Computer Software

PL/B Language Development and Support

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs(samples): Update BigQuery Storage Arrow samples batching logic#14961

docs(samples): Update BigQuery Storage Arrow samples batching logic#14961
Linchin merged 20 commits into
googleapis:mainfrom
AlexZMLyu:main

AlexZMLyu commented Dec 11, 2025

Uh oh!

gemini-code-assist Bot commented Dec 11, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexZMLyu commented Jan 2, 2026

Uh oh!

AlexZMLyu commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Sunbelt Computer Software

PL/B Language Development and Support

Uh oh!

Conversation

AlexZMLyu commented Dec 11, 2025

Uh oh!

gemini-code-assist Bot commented Dec 11, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexZMLyu commented Jan 2, 2026

Uh oh!

AlexZMLyu commented Jan 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants