Documentation Index
Fetch the complete documentation index at: /llms.txt
Use this file to discover all available pages before exploring further.
Skip to main content
Output:
Data Integrations
Google BigQuery
Enrich data at scale using Parallel’s SQL-native remote functions for BigQuery
For AI agents: a documentation index is available at https://docs.parallel.ai/llms.txt. The full text of all docs is at https://docs.parallel.ai/llms-full.txt. You may also fetch any page as Markdown by appending
This integration is ideal for data engineers who need to enrich large datasets with web intelligence directly in their BigQuery pipelines—without leaving SQL or building custom API integrations.
Parallel provides SQL-native remote functions for Google BigQuery that enable data enrichment directly in your SQL queries. The integration uses Cloud Functions to securely connect BigQuery to the Parallel API.
.md to its URL or sending Accept: text/markdown.View the complete demo notebook:
Features
- SQL-Native: Use
parallel_enrich()directly in BigQuery SQL queries - Secure: API key stored in Secret Manager, accessed via Cloud Functions
- Configurable Processors: Choose from lite-fast to ultra for speed vs thoroughness tradeoffs
- Structured Output: Returns JSON that can be parsed with BigQuery’s
JSON_EXTRACT_SCALAR()
Installation
pip install parallel-web-tools
The standalone
parallel-cli binary does not include deployment commands. You must install via pip to deploy the BigQuery integration.Deployment
Unlike Spark, the BigQuery integration requires a one-time deployment step to set up Cloud Functions and remote function definitions in your GCP project.Prerequisites
- Google Cloud Project with billing enabled
- Parallel API Key from Parallel
-
Google Cloud SDK installed and authenticated:
gcloud auth login gcloud auth application-default login
Deploy with CLI
parallel-cli enrich deploy --system bigquery \
--project=your-gcp-project \
--region=us-central1 \
--api-key=your-parallel-api-key
- Secret in Secret Manager for your API key
- Cloud Function (Gen2) that handles enrichment requests
- BigQuery Connection for remote function calls
- BigQuery Dataset (
parallel_functions) - Remote functions:
parallel_enrich()andparallel_enrich_company()
For manual deployment options, troubleshooting, and cleanup instructions, see the complete BigQuery setup guide.
Basic Usage
Once deployed, useparallel_enrich() in any BigQuery SQL query:
SELECT
name,
`your-project.parallel_functions.parallel_enrich`(
JSON_OBJECT('company_name', name, 'website', website),
JSON_ARRAY('CEO name', 'Founding year', 'Brief description')
) as enriched_data
FROM your_dataset.companies
LIMIT 10;
+--------+----------------------------------------------------------------------------------------------------------------------+
| name | enriched_data |
+--------+----------------------------------------------------------------------------------------------------------------------+
| Google | {"ceo_name": "Sundar Pichai", "founding_year": "1998", "brief_description": "Google is an American...", "basis": []} |
| Apple | {"ceo_name": "Tim Cook", "founding_year": "1976", "brief_description": "Apple Inc. is an American...", "basis": []} |
+--------+----------------------------------------------------------------------------------------------------------------------+
Function Parameters
| Parameter | Type | Description |
|---|---|---|
input_data | JSON | JSON object with key-value pairs of input data for enrichment |
output_columns | JSON | JSON array of descriptions for columns you want to enrich |
Parsing Results
The function returns JSON strings. Field names are converted to snake_case (e.g., “CEO name” →ceo_name).
Use JSON_EXTRACT_SCALAR() to extract individual fields:
WITH enriched AS (
SELECT
name,
`your-project.parallel_functions.parallel_enrich`(
JSON_OBJECT('company_name', name),
JSON_ARRAY('CEO name', 'Industry', 'Headquarters')
) as info
FROM your_dataset.companies
)
SELECT
name,
JSON_EXTRACT_SCALAR(info, '$.ceo_name') as ceo,
JSON_EXTRACT_SCALAR(info, '$.industry') as industry,
JSON_EXTRACT_SCALAR(info, '$.headquarters') as hq
FROM enriched;
+--------+-------------+------------+---------------+
| name | ceo | industry | hq |
+--------+-------------+------------+---------------+
| Google | Sundar Pichai| Technology | Mountain View |
| Apple | Tim Cook | Technology | Cupertino |
+--------+-------------+------------+---------------+
Company Convenience Function
For common company enrichment use cases:SELECT
`your-project.parallel_functions.parallel_enrich_company`(
'Google',
'google.com',
JSON_ARRAY('CEO name', 'Employee count', 'Stock ticker')
) as company_info;
Processor Selection
Choose a processor based on your speed vs thoroughness requirements. See Choose a Processor for detailed guidance and Pricing for cost information. To use a different processor, create a custom remote function with the desired processor in theuser_defined_context:
CREATE OR REPLACE FUNCTION `your-project.parallel_functions.parallel_enrich_pro`(
input_data STRING,
output_columns STRING
)
RETURNS STRING
REMOTE WITH CONNECTION `your-project.us-central1.parallel-connection`
OPTIONS (
endpoint = 'YOUR_FUNCTION_URL',
user_defined_context = [("processor", "pro-fast")]
);
Best Practices
Batch sizing
Batch sizing
Process data in batches to manage costs and avoid timeouts:
SELECT parallel_enrich(...) FROM companies LIMIT 100;
Error handling
Error handling
Failed enrichments return JSON with an Filter these in your downstream processing.
error field:{"error": "error message here"}
Cost management
Cost management
- Use
lite-fastfor high-volume, basic enrichments - Test with small batches before processing large tables
- Store results to avoid re-enriching the same data
Assistant
Responses are generated using AI and may contain mistakes.