[KYUUBI #6314][SPARK] Preserve lineage for temporary view inserts by wayne-x-wang · Pull Request #7791 · apache/kyuubi · GitHub
Skip to content

[KYUUBI #6314][SPARK] Preserve lineage for temporary view inserts - #7791

Open
wayne-x-wang wants to merge 2 commits into
apache:masterfrom
wayne-x-wang:kyuubi-6314-jdbc-temp-view
Open

[KYUUBI #6314][SPARK] Preserve lineage for temporary view inserts#7791
wayne-x-wang wants to merge 2 commits into
apache:masterfrom
wayne-x-wang:kyuubi-6314-jdbc-temp-view

Conversation

@wayne-x-wang

@wayne-x-wang wayne-x-wang commented Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Inserts into JDBC and file-source temporary views can lack target catalogTable metadata. The guarded column-lineage mappings throw MatchError even though the writes succeed.

Keep the optional target table name in both InsertIntoDataSourceCommand and InsertIntoHadoopFsRelationCommand. When metadata is absent, preserve query column names and source lineage without inventing a destination table. outputTables remains empty. Catalog-backed inserts retain their existing behavior.

Fixes #6314.

How was this patch tested?

The embedded Derby regression creates a JDBC temporary view, executes an insert, and reads the stored row through JDBC. Added JSON and Parquet temporary-view regressions that execute INSERT OVERWRITE, read the stored rows, and verify InsertIntoHadoopFsRelationCommand with absent catalogTable. All three cases assert the full Lineage through both parse and transformToLineage, including source tables, column mappings, and empty output tables. Both inherited parser suites run these tests. Existing catalog-backed insert tests check the metadata-present behavior.

Before the file-source fix, the new JSON and Parquet tests both failed with MatchError while the JDBC test passed. The original JDBC regression was also verified to fail without its fix.

Spark 3.5.8 / Scala 2.12: all 81 lineage tests passed, along with Spotless and Scalastyle.

Spark 4.2.0 / Scala 2.13: all 81 lineage tests passed.

dev/reformat and git diff --check passed; formatting preserved the tested patch unchanged.

build/mvn -B -ntp test spotless:check scalastyle:check \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-3.5,spark-provided,flink-provided,hive-provided \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage

build/mvn -B -ntp test \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-4.2,scala-2.13,spark-provided,flink-provided,hive-provided \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage

The historical Kyuubi 1.8.0 / Spark 3.2.3 combination and MySQL were not tested.

Was this patch assisted by generative AI tooling?

Assisted-by: Codex (GPT-6)

@github-actions

github-actions Bot commented Oct 19, 2026

Copy link
Copy Markdown

@wayne-x-wang

wayne-x-wang commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Could a maintainer please re-run the failed jobs? All 77 lineage tests passed. The two failures are in EngineRefWithZookeeperSuite and the ZooKeeper 3.7 integration test, outside this patch’s changes.

@wayne-x-wang

wayne-x-wang commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Locally, the ZooKeeper 3.7 test passed three consecutive runs, while the EngineRef timeout test reproduced the failure. The EngineRef test passes a null engineManager, causing an NPE during timeout cleanup, and relies on unreliable Spark startup timing assumptions. These failures are unrelated to this PR’s lineage changes; all 77 lineage tests, including the new JDBC regression cases, passed in CI.

pan3793
pan3793 reviewed Oct 20, 2026
View reviewed changes

@pan3793 pan3793 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is correct for the JDBC temp-view path, but the same failure class remains in the sibling file-source branch, so the change is not yet complete.

1. Blocking - same MatchError still live in InsertIntoHadoopFsRelationCommand (SparkSQLLineageParseHelper.scala:281). A temp view over a file source is a LogicalRelation with catalogTable = None, so INSERT OVERWRITE TABLE <json/parquet temp view> throws scala.MatchError: (key#0,{key#0}) at line 274. Confirmed at 9b6c365. Fix both branches the same way: carry Option[String] and map it instead of getOrElse("") plus a guard, and add a file-source temp view regression (the suite's SimpleInsertSource reproduces it without Derby).

2. SparkSQLLineageParserHelperSuite.scala:385-388 asserts Derby shutdown SQLState inside finally; a best-effort Try(...) avoids masking the real failure.

3. Drop the two info() plan dumps at lines 354-355; they run in both suites.

4. assert(outputTables == List()) also held before the change; the parse equality is what pins it.

Verified: the new test fails with MatchError when the fix is reverted, 77/77 lineage tests pass at head, plan node/fields unchanged on Spark 3.5.8 and 4.2.0.

@wayne-x-wang

wayne-x-wang commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@wayne-x-wang wayne-x-wang changed the title [KYUUBI #6314][SPARK] Preserve lineage for JDBC temporary view inserts [KYUUBI #6314][SPARK] Preserve lineage for temporary view inserts Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pan3793 pan3793 pan3793 left review comments

Assignees

No one assigned

Labels

module:extensions module:spark

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] Insert into a jdbc table will cause exception when parse lineage in spark

2 participants

@wayne-x-wang @pan3793