bug: Can't sync AbandonedCheckouts in Shopify Plugin · Issue #19638 · cloudquery/cloudquery · GitHub
Skip to content

bug: Can't sync AbandonedCheckouts in Shopify Plugin #19638

Open
Open
bug: Can't sync AbandonedCheckouts in Shopify Plugin#19638
Labels
area/plugin/source/shopifykind/bugpriority/p1Regular issue. Prioritized per sprint
@konung

Description

@konung
konung
opened on Nov 18, 2024

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

Hi.

Getting an error when trying to run a cloudquerey sync. Other tables are syncing correctly.

Loading spec(s) from cloudquery-config/shopify-to-postgresql/
Starting sync for: shopify (cloudquery/shopify@v8.3.2) -> [postgresql-shopify (cloudquery/postgresql@v8.6.7)]
Sync completed with errors, see logs for details. Resources: 566, Errors: 1, Warnings: 0, Time: 13s

Here is the specific error line from the log below. I already tried deleting the table and reruning sync - same error.

This is a brand new setup. Old on is running on legacy cloudquery 3 , I was just testing before replacing / upgrading old setup.


2024-11-18T18:10:58Z ERR table resolver finished with error error="GetAbandonedCheckouts failed: json: cannot unmarshal string into Go struct field AbandonedCheckout.checkouts.shipping_lines of type uint64" client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_abandoned_checkouts

Expected Behavior

Should be syncing table. right now the table is getting generated, but it's empty. I'm expecting several hundred rows in it.

CloudQuery (redacted) config

kind: destination
spec:
  name: "postgresql-shopify"
  path: "cloudquery/postgresql"
  version: "v8.6.7"
  write_mode: "overwrite-delete-stale" # overwrite-delete-stale, overwrite, append
  spec:
    connection_string: "${POSTGRES_CONNECTION_STRING}&search_path=cq_shopify"


kind: source
spec:
  name: shopify
  path: cloudquery/shopify
  version: "v8.3.2"
  tables: [ "*"]
  destinations: ["postgresql-shopify"]
  backend_options:
    table_name: "shopify_postgres_state"
    connection: "@@plugins.postgresql-shopify.connection"
  spec:
    shop_url: "https://${SHOPIFY_STORE_NAME}.myshopify.com"
    access_token: "${SHOPIFY_ACCESS_TOKEN}"
    concurrency: 1000

Steps To Reproduce

No response

CloudQuery (redacted) logs

2024-11-18T18:10:54Z INF Plugin server listening address=/var/folders/6q/mwhf773164q07t2nh3c7hhdr0000gn/T/cq-HRKdRCXkuRqhpWFo.sock module=shopify-source
2024-11-18T18:10:55Z INF Plugin server listening address=/var/folders/6q/mwhf773164q07t2nh3c7hhdr0000gn/T/cq-nqKJyQwXWgAHgxRI.sock module=postgresql-destination
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.951 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=1.821 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.674 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.749 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF Start sync destinations=["postgresql-shopify (cloudquery/postgresql@v8.6.7)"] invocation_id=ed5b3087-8334-4d70-REDACTED module=cli source="shopify (cloudquery/shopify@v8.3.2)" sync_time=2024-11-18T18:10:56Z transformers=[]
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.021 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF Initializing postgresql destination invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest pgx_log_level=error
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=32.829 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.246 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=194.211 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:10:56Z INF Start fetching resources destinations=["postgresql-shopify (cloudquery/postgresql@v8.6.7)"] invocation_id=ed5b3087-8334-4d70-REDACTED module=cli source="shopify (cloudquery/shopify@v8.3.2)"
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.252 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.046 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.093 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=3.447 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_postgres_state
2024-11-18T18:10:56Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_postgres_state
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=37.012 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Read grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=0.369 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF finished call grpc.code=OK grpc.component=server grpc.method=Read grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=3.531 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_abandoned_checkouts
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_orders
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_price_rules
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_products
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_customers
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_product_variants
2024-11-18T18:10:56Z INF top level table resolver started client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_price_rule_discount_codes
2024-11-18T18:10:56Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=129.068 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:10:57Z INF table sync finished client=shopify duration_ms=236.438458 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=32 table=shopify_product_variants
2024-11-18T18:10:57Z INF table sync finished client=shopify duration_ms=319.810833 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=13 table=shopify_price_rules
2024-11-18T18:10:57Z INF table sync finished client=shopify duration_ms=321.2115 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=13 table=shopify_products
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_price_rule_discount_codes
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_price_rule_discount_codes
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_abandoned_checkouts
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_customers
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_customers
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_orders
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_orders
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_products
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_products
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_product_variants
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_product_variants
2024-11-18T18:10:57Z INF Migrating table invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_price_rules
2024-11-18T18:10:57Z INF Table exists, auto-migrating invocation_id=ed5b3087-8334-4d70-REDACTED module=pg-dest table=shopify_price_rules
2024-11-18T18:10:58Z ERR table resolver finished with error error="GetAbandonedCheckouts failed: json: cannot unmarshal string into Go struct field AbandonedCheckout.checkouts.shipping_lines of type uint64" client=shopify invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source table=shopify_abandoned_checkouts
2024-11-18T18:10:58Z INF table sync finished client=shopify duration_ms=1094.08 errors=1 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=0 table=shopify_abandoned_checkouts
2024-11-18T18:10:58Z INF table sync finished client=shopify duration_ms=1329.473708 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=120 table=shopify_price_rule_discount_codes
2024-11-18T18:10:58Z INF table sync finished client=shopify duration_ms=1375.718 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=92 table=shopify_customers
2024-11-18T18:11:08Z INF table sync finished client=shopify duration_ms=11534.350625 errors=0 invocation_id=ed5b3087-8334-4d70-REDACTED module=shopify-source resources=296 table=shopify_orders
2024-11-18T18:11:08Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:11:08-06:00 grpc.time_ms=0.022 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:11:08Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:11:08-06:00 grpc.time_ms=4.244 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:11:08Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=11845.623 module=shopify-source peer.address= protocol=grpc
2024-11-18T18:11:09Z INF Sync summary invocation_id=ed5b3087-8334-4d70-REDACTED module=cli summary={"cli_version":"6.11.2","destination_errors":0,"destination_name":"postgresql-shopify","destination_path":"cloudquery/postgresql","destination_version":"v8.6.7","destination_warnings":0,"resources":566,"source_errors":1,"source_name":"shopify","source_path":"cloudquery/shopify","source_tables":[],"source_version":"v8.3.2","source_warnings":0,"sync_id":"ed5b3087-8334-4d70-REDACTED","sync_time":"2024-11-18T18:10:56.599056Z"}
2024-11-18T18:11:09Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:10:56-06:00 grpc.time_ms=12879.765 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:11:09Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:11:09-06:00 grpc.time_ms=0.005 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:11:09Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-11-18T12:11:09-06:00 grpc.time_ms=0.053 module=postgresql-destination peer.address= protocol=grpc
2024-11-18T18:11:09Z INF Sync summary duration=13s errors=1 invocation_id=ed5b3087-8334-4d70-REDACTED module=cli resources=566 result="Sync completed with errors, see logs for details" warnings=0
2024-11-18T18:11:09Z INF End sync destinations=["postgresql-shopify (cloudquery/postgresql@v8.6.7)"] invocation_id=ed5b3087-8334-4d70-REDACTED module=cli source="shopify (cloudquery/shopify@v8.3.2)" sync_time=2024-11-18T18:10:56Z transformers=[]
2024-11-18T18:11:09Z INF Sending sync summary to analyticsv1.cloudquery.io:443 invocation_id=ed5b3087-8334-4d70-REDACTED module=cli
2024-11-18T18:11:09Z INF waiting for source plugin to terminate invocation_id=ed5b3087-8334-4d70-REDACTED module=cli
2024-11-18T18:11:09Z INF Got stop signal. Plugin server shutting down address=/var/folders/6q/mwhf773164q07t2nh3c7hhdr0000gn/T/cq-nqKJyQwXWgAHgxRI.sock module=postgresql-destination signal=interrupt
2024-11-18T18:11:09Z INF waiting for source plugin to terminate invocation_id=ed5b3087-8334-4d70-REDACTED module=cli
2024-11-18T18:11:09Z INF Got stop signal. Plugin server shutting down address=/var/folders/6q/mwhf773164q07t2nh3c7hhdr0000gn/T/cq-HRKdRCXkuRqhpWFo.sock module=shopify-source signal=interrupt

CloudQuery version

6.11.2

Additional Context

Dev/Testing on macOSX (15.1) . Have not tried in docker or Linux yet ( produciton env)

Pull request (optional)

  • I can submit a pull request

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/plugin/source/shopifykind/bugpriority/p1Regular issue. Prioritized per sprint

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions