{{ message }}
Repository navigation
feat(tidb): add omni parser shim (Phase 1 Task 1.1) - #20107
Merged
Merged
Conversation
Phase 1 Task 1.1 of the omni/tidb completion plan. Adds the thin bridge layer that downstream tasks (tidb.go extension, completion.go, normalize.go, restore.go) will call into omni/tidb through, so individual callers do not take a direct dependency on the omni package. Mirrors backend/plugin/parser/mysql/omni.go with three deliberate differences, all driven by empirical evidence (see below): - No AsANTLRAST/AsPingCapAST fallback method. TiDB has no TiDB-specific ANTLR grammar, and the 51 tidb advisors in backend/plugin/advisor/tidb/ reach the parsed AST via getTiDBNodes -> tidbparser.GetTiDBAST, which asserts to *tidb.AST (native pingcap), never *OmniAST. No current consumer would exercise a fallback. Task 1.2 keeps the registered ParseStatementsFunc pointed at the native parser, so *OmniAST will not reach advisors in Phase 1. When a post-Phase 1 migration flips the default, add AsPingCapAST + a matching base.PingCapASTProvider interface alongside that migration PR, with real consumers to exercise it. - Mid-rune byte offset handling in ByteOffsetToRunePosition treats the offset as pointing at the start of the enclosing rune (does not count the partial rune). mysql's version silently counted it; tidb's comment documents the semantics so callers know what to expect. - Inline unit tests for ByteOffsetToRunePosition mandatory per plan §167 (mysql's version has no dedicated tests). 14 subtests cover BMP multi-byte, surrogate pairs, mid-rune clamping, newline boundaries, past-end clamping, negative offsets. Verification: go test -count=1 ./backend/plugin/parser/tidb/... — ok go test -count=1 ./backend/plugin/parser/mysql/... — ok go build -ldflags "-w -s" -p=16 ... — ok golangci-lint run --allow-parallel-runners — 0 issues Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rebelice
approved these changes
Apr 27, 2026
6 of 8 tasks
vsai12
added a commit
that referenced
this pull request
Apr 30, 2026
…0179) * feat(tidb): add AsPingCapAST bridge on OmniAST (Phase 1.5 §1.5.2) Mirror of mysql/OmniAST.AsANTLRAST for the Phase 1.5 advisor migration. Required prerequisite for the eventual dispatcher flip (§1.5.N+1): once the registered ParseStatementsFunc is flipped to omni and *OmniAST values reach the ~50 un-migrated tidb advisors at any point in the migration, those advisors keep working because GetTiDBAST falls through to AsPingCapAST() automatically. Implementation mirrors mysql/omni.go:39-55: - Lazy parse: native pingcap parse runs once on first AsPingCapAST call per OmniAST instance. - Cached result: pingcapParsed bool flag distinguishes "not yet parsed" from "parsed and got nil"; cache hits skip the work. - Negative caching: pingcap parse failure also short-circuits future calls so unparseable-by-pingcap SQL doesn't re-parse per advisor. backend/plugin/parser/tidb/ast.go: - Adds PingCapASTProvider interface (analog of base.AntlrASTProvider). - GetTiDBAST extended to check for the provider, mirroring base.GetANTLRAST's pattern. backend/plugin/parser/tidb/omni.go: - OmniAST gains pingcapAST + pingcapParsed unexported fields. - AsPingCapAST method implements the provider contract. - Godoc updated: removes the "neither fallback" rationale (correct for PR #20107's scope but inverted by the migration plan), points at plan §1.5.0 invariant #4 + bridge persistence rule for context. Tests: - TestAsPingCapASTReturnsNativeAST: bridge contract for valid SQL. - TestAsPingCapASTCachesAcrossCalls: a.Same() pointer equality on repeat calls proves cache hit, not fresh re-parse. - TestAsPingCapASTCachesNegativeResult: parse failure cached too. - TestGetTiDBASTFallsBackToProvider: end-to-end bridge through GetTiDBAST — the path un-migrated advisors will use. Bridge persistence: the bridge stays in the codebase indefinitely post-Phase-1.5. The one Class III advisor (dml_dry_run, blocked on omni's NonTransactionalDMLStmt grammar — Phase 2 §Tier 4g) requires this fallback. Bridge cleanup is gated on dml_dry_run unblocking, not on Phase 1.5 completion. Validation: go test -count=1 ./backend/plugin/parser/tidb/... # green (incl. 4 new bridge tests) go test -count=1 ./backend/plugin/advisor/tidb/... # green go test -count=1 ./backend/plugin/advisor/mysql/... # green (no cross-engine regression) go build ... # green golangci-lint run ... # 0 issues Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tidb): bridge line-tracking + symmetric logging + scoped godoc PR review caught three issues. All addressed before merge. #3 (ship-blocker) — Line-tracking regression. The original bridge called bare ParseTiDB and skipped the SetOriginTextPosition + per-column line work that the canonical pre-flip path (ParseTiDBForSyntaxCheck) applies at tidb.go:64-75. 49 of 51 tidb advisors call node.OriginTextPosition() for advice line numbers; without the fix, post-flip un-migrated advisors silently regress to line 1 for every statement. Fix: extract applyTiDBLineTracking(node, baseLine, originalText) helper from ParseTiDBForSyntaxCheck. Both the canonical path and the bridge call it, so consumers reading OriginTextPosition() see consistent values across both paths. Helper returns the 1-based actualStartLine for *AST.StartPosition use. Regression test: TestAsPingCapASTLineTrackingMatchesCanonical parses a two-statement input via both paths and asserts the second statement's node.OriginTextPosition() matches (both must report 2, not 1). #1 — Godoc overpromise. Original wording claimed "review continuity from both directions" but the bridge only handles one direction — omni-success → pingcap-fallback. The opposite (omni rejects at dispatcher level so no *OmniAST exists) is the dispatcher-fallback contract, not the bridge's job. Godoc now narrows the claim and forward-references plan §1.5.0 invariant #5 (the dispatcher-fallback invariant being added in a follow-up plan amendment). #2 — Logging asymmetry. Pilot side soft-fails with slog.Debug + log. Bridge silently returned (nil, false). Now logs symmetrically on each failure mode (parse error, zero nodes, line-tracking error). Negative cache means a single Debug per OmniAST instance, not per call. #4 — Strict ParseTiDB note. Godoc now explains why the bridge uses strict ParseTiDB (no error-recovery) instead of mysql's lenient ANTLR parse: pingcap has no equivalent error-recovery mode, and the bridge operates on omni-validated SQL where strict parsing is correct. #5 — Multi-statement assumption made explicit. nodes[0] used; nodes[1:] logged as a contract violation instead of silently dropped. Validation: go test -count=1 ./backend/plugin/parser/tidb/... # green incl. new regression test go test -count=1 ./backend/plugin/advisor/tidb/... # green go test -count=1 ./backend/plugin/advisor/mysql/... # green (cross-engine regression) go build ... # green golangci-lint run ... # 0 issues Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(tidb): correct godoc reference to plan invariant #8 Numbering drift caught in review: the godoc cited "§1.5.0 invariant #5" but the landed plan invariant is #8 (the dispatcher-fallback contract). The §5 numbering came from an earlier draft of the plan amendment; final landed numbering is #8 because three additional invariants (#5 no-mutation, #6 recipe-default, #7 shape-audit) had already accreted. One-line fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(advisor/tidb): soft-fail getTiDBNodes on bridge miss Codex P1 catch on PR #20179. Without this fix, post-§1.5.N+1 dispatcher flip: ParsedStatement.AST = *OmniAST → un-migrated advisor calls getTiDBNodes → tidbparser.GetTiDBAST(stmt.AST) → AsPingCapAST fails (pingcap rejects what omni accepted) → GetTiDBAST returns (nil, false) → getTiDBNodes returns 'AST type mismatch: expected TiDB parser result' → advisor.Check propagates as 'Rule check failed' → every un-migrated advisor breaks for that review That breaks the migration continuity the bridge is meant to preserve. Phase 1.5 invariant #2 says soft-fail (skip statement, review continues), not hard-fail. Fix: when GetTiDBAST returns false AND stmt.AST is a PingCapASTProvider (i.e. an *OmniAST that just routed through the bridge and missed), skip the statement instead of returning an error. The bridge has already logged the parse failure at debug level. True engine-mismatch (stmt.AST is some unrelated type, neither *tidb.AST nor a PingCapASTProvider) is still surfaced as an error — that's a programmer error in registration, not a soft-fail case. Regression test: TestGetTiDBNodesSoftFailsOnBridgeMiss feeds a context with two ParsedStatements: one *OmniAST whose AsPingCapAST will fail ('SELECT FROM WHERE;') bracketed by one that succeeds ('SELECT 1'). Expects no error and len(got) == 1 (the successful one). Sanity-checked: GetTiDBAST returns false on a pingcap-invalid OmniAST, confirming the pre-fix path would have errored. Validation: go test -count=1 ./backend/plugin/advisor/tidb/... # green incl. new test go test -count=1 ./backend/plugin/advisor/mysql/... # green go test -count=1 ./backend/plugin/parser/tidb/... # green golangci-lint run ... # 0 issues Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.




Summary
backend/plugin/parser/tidb/omni.go(~95 LOC), the thin bridge layer that Phase 1 Tasks 1.2–1.5 will call through to avoid taking directgithub.com/bytebase/omni/tidb/*dependencies in every consumer.OmniAST(implementsbase.AST),ParseTiDBOmni,GetOmniNode,ByteOffsetToRunePosition.base.ParseStatementsFuncforEngine_TIDBstill points at the native pingcap parser (parseTiDBStatementsintidb.go); that's Task 1.2's job, which consumes the symbols added here.Mirror of
mysql/omni.go, with three deliberate divergences1. No
AsANTLRAST/AsPingCapASTfallback methodThe plan (§161–166) mandated an empirical check before finalizing the backward-compat shape. Evidence:
grep -rn AsANTLRAST backend/→ 28 consumers inadvisor/oracle/*+ 3 inadvisor/snowflake/*. All call the genericbase.GetANTLRAST()which routes through theAntlrASTProviderinterface. None are TiDB advisors.backend/plugin/advisor/tidb/*reach the AST viagetTiDBNodes()atbackend/plugin/advisor/tidb/utils.go:84, which callstidbparser.GetTiDBAST(stmt.AST)— asserts to native*tidb.AST(pingcap), never*OmniAST.backend/plugin/parser/tidb/backup.gois the only tidb parser file using the mysql ANTLR grammar; it parses its own text, does NOT consume*OmniAST.parseTiDBStatementspointed at the native parser ("no behavioral change yet"). So*OmniASTwill not reach any tidb advisor in Phase 1.Decision: ship NEITHER fallback method. Adding either one today is unreachable code. When a post-Phase 1 migration flips the default registration to omni, the migrator will add
AsPingCapAST()+ a matchingbase.PingCapASTProviderinterface in the same PR, with real consumers to exercise the round-trip. Rationale is documented inline in theOmniASTgodoc.2. Mid-rune byte offset handling is documented and tested
mysql's
ByteOffsetToRunePositionsilently counts a rune when the requested offset lands inside its multi-byte UTF-8 sequence. tidb's version treats the offset as pointing at the start of the enclosing rune (does not count it) and the godoc says so. Behaviorally equivalent for the normal case where callers pass rune-aligned offsets; the divergence only shows when an upstream bug hands us a mid-rune offset, where "point at the rune start" is easier to reason about than "off by one past end of prior rune."3. Inline unit tests mandatory per plan §167
mysql's
omni.gohas no dedicated tests forByteOffsetToRunePosition. The plan tightened this for tidb:TestByteOffsetToRunePositionsubtests: ASCII, newline, BMP 3-byte (CJK), surrogate pair (emoji), mid-rune clamping (both 3-byte and 4-byte), past-end clamp, negative offset, offset-at-newline, multi-byte across newline.TestGetOmniNode: nil input, wrong-type input, valid*OmniASTround-trip.TestOmniASTStartPosition: trivial accessor.Verification
Pre-PR checklist (
docs/pre-pr-checklist.md): §2 not breaking (pure additive exports, no registration changes, no wired consumer), §3 N/A (nobackend/store/orbackend/migrator/changes), §4 lint/format clean, §5 tests + full-server build pass, §6 sonarcloud globs already cover the new files (backend/**/*_test.gofor the test file; non-test file falls under default source inclusion), §7 no unrelated files staged.Test plan
go test -count=1 ./backend/plugin/parser/tidb/...— greengo test -count=1 ./backend/plugin/parser/mysql/...— no cross-engine regressiongo buildof full server — greengolangci-lint— 0 issuesgofmt -l— cleantidb.goextension) PR imports and usesOmniAST/ParseTiDBOmni/ByteOffsetToRunePosition— exercises the surface end-to-end; lands separately🤖 Generated with Claude Code