feat(tidb): add omni parser shim (Phase 1 Task 1.1) by vsai12 · Pull Request #20107 · bytebase/bytebase · GitHub
Skip to content

feat(tidb): add omni parser shim (Phase 1 Task 1.1) - #20107

Merged
vsai12 merged 1 commit into
mainfrom
feat/tidb-omni-shim
Apr 27, 2026
Merged

vsai12 merged 1 commit into
mainfrom
feat/tidb-omni-shim

Conversation

@vsai12

@vsai12 vsai12 commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds backend/plugin/parser/tidb/omni.go (~95 LOC), the thin bridge layer that Phase 1 Tasks 1.2–1.5 will call through to avoid taking direct github.com/bytebase/omni/tidb/* dependencies in every consumer.
  • Exposes: OmniAST (implements base.AST), ParseTiDBOmni, GetOmniNode, ByteOffsetToRunePosition.
  • No behavioral change. No caller wired in this PR. The registered base.ParseStatementsFunc for Engine_TIDB still points at the native pingcap parser (parseTiDBStatements in tidb.go); that's Task 1.2's job, which consumes the symbols added here.
  • Phase 1 Task 1.1 of plans/2026-04-23-omni-tidb-completion-plan.md. Parent Linear: BYT-9141.

Mirror of mysql/omni.go, with three deliberate divergences

1. No AsANTLRAST / AsPingCapAST fallback method

The plan (§161–166) mandated an empirical check before finalizing the backward-compat shape. Evidence:

  • grep -rn AsANTLRAST backend/ → 28 consumers in advisor/oracle/* + 3 in advisor/snowflake/*. All call the generic base.GetANTLRAST() which routes through the AntlrASTProvider interface. None are TiDB advisors.
  • The 51 tidb advisors under backend/plugin/advisor/tidb/* reach the AST via getTiDBNodes() at backend/plugin/advisor/tidb/utils.go:84, which calls tidbparser.GetTiDBAST(stmt.AST) — asserts to native *tidb.AST (pingcap), never *OmniAST.
  • backend/plugin/parser/tidb/backup.go is the only tidb parser file using the mysql ANTLR grammar; it parses its own text, does NOT consume *OmniAST.
  • Task 1.2 (next PR) explicitly keeps parseTiDBStatements pointed at the native parser ("no behavioral change yet"). So *OmniAST will not reach any tidb advisor in Phase 1.

Decision: ship NEITHER fallback method. Adding either one today is unreachable code. When a post-Phase 1 migration flips the default registration to omni, the migrator will add AsPingCapAST() + a matching base.PingCapASTProvider interface in the same PR, with real consumers to exercise the round-trip. Rationale is documented inline in the OmniAST godoc.

2. Mid-rune byte offset handling is documented and tested

mysql's ByteOffsetToRunePosition silently counts a rune when the requested offset lands inside its multi-byte UTF-8 sequence. tidb's version treats the offset as pointing at the start of the enclosing rune (does not count it) and the godoc says so. Behaviorally equivalent for the normal case where callers pass rune-aligned offsets; the divergence only shows when an upstream bug hands us a mid-rune offset, where "point at the rune start" is easier to reason about than "off by one past end of prior rune."

3. Inline unit tests mandatory per plan §167

mysql's omni.go has no dedicated tests for ByteOffsetToRunePosition. The plan tightened this for tidb:

  • 14 TestByteOffsetToRunePosition subtests: ASCII, newline, BMP 3-byte (CJK), surrogate pair (emoji), mid-rune clamping (both 3-byte and 4-byte), past-end clamp, negative offset, offset-at-newline, multi-byte across newline.
  • TestGetOmniNode: nil input, wrong-type input, valid *OmniAST round-trip.
  • TestOmniASTStartPosition: trivial accessor.

Verification

$ go test -count=1 ./backend/plugin/parser/tidb/...
ok  	github.com/bytebase/bytebase/backend/plugin/parser/tidb	1.423s

$ go test -count=1 ./backend/plugin/parser/mysql/...
ok  	github.com/bytebase/bytebase/backend/plugin/parser/mysql	0.623s

$ go build -ldflags "-w -s" -p=16 -o ./bytebase-build/bytebase ./backend/bin/server/main.go
ok

$ golangci-lint run --allow-parallel-runners ./backend/plugin/parser/tidb/...
0 issues.

$ gofmt -l backend/plugin/parser/tidb/omni.go backend/plugin/parser/tidb/omni_test.go
(clean)

Pre-PR checklist (docs/pre-pr-checklist.md): §2 not breaking (pure additive exports, no registration changes, no wired consumer), §3 N/A (no backend/store/ or backend/migrator/ changes), §4 lint/format clean, §5 tests + full-server build pass, §6 sonarcloud globs already cover the new files (backend/**/*_test.go for the test file; non-test file falls under default source inclusion), §7 no unrelated files staged.

Test plan

  • go test -count=1 ./backend/plugin/parser/tidb/... — green
  • go test -count=1 ./backend/plugin/parser/mysql/... — no cross-engine regression
  • go build of full server — green
  • golangci-lint — 0 issues
  • gofmt -l — clean
  • Task 1.2 (tidb.go extension) PR imports and uses OmniAST / ParseTiDBOmni / ByteOffsetToRunePosition — exercises the surface end-to-end; lands separately

🤖 Generated with Claude Code

Phase 1 Task 1.1 of the omni/tidb completion plan. Adds the thin bridge
layer that downstream tasks (tidb.go extension, completion.go, normalize.go,
restore.go) will call into omni/tidb through, so individual callers do not
take a direct dependency on the omni package.

Mirrors backend/plugin/parser/mysql/omni.go with three deliberate
differences, all driven by empirical evidence (see below):

- No AsANTLRAST/AsPingCapAST fallback method. TiDB has no TiDB-specific
  ANTLR grammar, and the 51 tidb advisors in backend/plugin/advisor/tidb/
  reach the parsed AST via getTiDBNodes -> tidbparser.GetTiDBAST, which
  asserts to *tidb.AST (native pingcap), never *OmniAST. No current
  consumer would exercise a fallback. Task 1.2 keeps the registered
  ParseStatementsFunc pointed at the native parser, so *OmniAST will not
  reach advisors in Phase 1. When a post-Phase 1 migration flips the
  default, add AsPingCapAST + a matching base.PingCapASTProvider interface
  alongside that migration PR, with real consumers to exercise it.
- Mid-rune byte offset handling in ByteOffsetToRunePosition treats the
  offset as pointing at the start of the enclosing rune (does not count
  the partial rune). mysql's version silently counted it; tidb's comment
  documents the semantics so callers know what to expect.
- Inline unit tests for ByteOffsetToRunePosition mandatory per plan §167
  (mysql's version has no dedicated tests). 14 subtests cover BMP
  multi-byte, surrogate pairs, mid-rune clamping, newline boundaries,
  past-end clamping, negative offsets.

Verification:
  go test -count=1 ./backend/plugin/parser/tidb/... — ok
  go test -count=1 ./backend/plugin/parser/mysql/... — ok
  go build -ldflags "-w -s" -p=16 ... — ok
  golangci-lint run --allow-parallel-runners — 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vsai12
vsai12 requested a review from a team as a code owner April 24, 2026 00:58
@cla-bot cla-bot Bot added the cla-signed label Apr 24, 2026
@sonarqubecloud

Copy link
Copy Markdown

@vsai12
vsai12 merged commit 1d0003e into main Apr 27, 2026
15 of 16 checks passed
@vsai12
vsai12 deleted the feat/tidb-omni-shim branch April 27, 2026 23:20
vsai12 added a commit that referenced this pull request Apr 30, 2026
…0179)

* feat(tidb): add AsPingCapAST bridge on OmniAST (Phase 1.5 §1.5.2)

Mirror of mysql/OmniAST.AsANTLRAST for the Phase 1.5 advisor migration.
Required prerequisite for the eventual dispatcher flip (§1.5.N+1):
once the registered ParseStatementsFunc is flipped to omni and
*OmniAST values reach the ~50 un-migrated tidb advisors at any point
in the migration, those advisors keep working because GetTiDBAST
falls through to AsPingCapAST() automatically.

Implementation mirrors mysql/omni.go:39-55:
  - Lazy parse: native pingcap parse runs once on first AsPingCapAST
    call per OmniAST instance.
  - Cached result: pingcapParsed bool flag distinguishes "not yet
    parsed" from "parsed and got nil"; cache hits skip the work.
  - Negative caching: pingcap parse failure also short-circuits future
    calls so unparseable-by-pingcap SQL doesn't re-parse per advisor.

backend/plugin/parser/tidb/ast.go:
  - Adds PingCapASTProvider interface (analog of base.AntlrASTProvider).
  - GetTiDBAST extended to check for the provider, mirroring
    base.GetANTLRAST's pattern.

backend/plugin/parser/tidb/omni.go:
  - OmniAST gains pingcapAST + pingcapParsed unexported fields.
  - AsPingCapAST method implements the provider contract.
  - Godoc updated: removes the "neither fallback" rationale (correct
    for PR #20107's scope but inverted by the migration plan), points
    at plan §1.5.0 invariant #4 + bridge persistence rule for context.

Tests:
  - TestAsPingCapASTReturnsNativeAST: bridge contract for valid SQL.
  - TestAsPingCapASTCachesAcrossCalls: a.Same() pointer equality on
    repeat calls proves cache hit, not fresh re-parse.
  - TestAsPingCapASTCachesNegativeResult: parse failure cached too.
  - TestGetTiDBASTFallsBackToProvider: end-to-end bridge through
    GetTiDBAST — the path un-migrated advisors will use.

Bridge persistence: the bridge stays in the codebase indefinitely
post-Phase-1.5. The one Class III advisor (dml_dry_run, blocked on
omni's NonTransactionalDMLStmt grammar — Phase 2 §Tier 4g) requires
this fallback. Bridge cleanup is gated on dml_dry_run unblocking, not
on Phase 1.5 completion.

Validation:
  go test -count=1 ./backend/plugin/parser/tidb/...    # green (incl. 4 new bridge tests)
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green (no cross-engine regression)
  go build ...                                         # green
  golangci-lint run ...                                # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tidb): bridge line-tracking + symmetric logging + scoped godoc

PR review caught three issues. All addressed before merge.

#3 (ship-blocker) — Line-tracking regression. The original bridge called
bare ParseTiDB and skipped the SetOriginTextPosition + per-column line
work that the canonical pre-flip path (ParseTiDBForSyntaxCheck) applies
at tidb.go:64-75. 49 of 51 tidb advisors call node.OriginTextPosition()
for advice line numbers; without the fix, post-flip un-migrated
advisors silently regress to line 1 for every statement.

Fix: extract applyTiDBLineTracking(node, baseLine, originalText) helper
from ParseTiDBForSyntaxCheck. Both the canonical path and the bridge
call it, so consumers reading OriginTextPosition() see consistent
values across both paths. Helper returns the 1-based actualStartLine
for *AST.StartPosition use.

Regression test: TestAsPingCapASTLineTrackingMatchesCanonical parses a
two-statement input via both paths and asserts the second statement's
node.OriginTextPosition() matches (both must report 2, not 1).

#1 — Godoc overpromise. Original wording claimed "review continuity
from both directions" but the bridge only handles one direction —
omni-success → pingcap-fallback. The opposite (omni rejects at
dispatcher level so no *OmniAST exists) is the dispatcher-fallback
contract, not the bridge's job. Godoc now narrows the claim and
forward-references plan §1.5.0 invariant #5 (the dispatcher-fallback
invariant being added in a follow-up plan amendment).

#2 — Logging asymmetry. Pilot side soft-fails with slog.Debug + log.
Bridge silently returned (nil, false). Now logs symmetrically on each
failure mode (parse error, zero nodes, line-tracking error). Negative
cache means a single Debug per OmniAST instance, not per call.

#4 — Strict ParseTiDB note. Godoc now explains why the bridge uses
strict ParseTiDB (no error-recovery) instead of mysql's lenient ANTLR
parse: pingcap has no equivalent error-recovery mode, and the bridge
operates on omni-validated SQL where strict parsing is correct.

#5 — Multi-statement assumption made explicit. nodes[0] used; nodes[1:]
logged as a contract violation instead of silently dropped.

Validation:
  go test -count=1 ./backend/plugin/parser/tidb/...   # green incl. new regression test
  go test -count=1 ./backend/plugin/advisor/tidb/...  # green
  go test -count=1 ./backend/plugin/advisor/mysql/... # green (cross-engine regression)
  go build ...                                        # green
  golangci-lint run ...                               # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(tidb): correct godoc reference to plan invariant #8

Numbering drift caught in review: the godoc cited "§1.5.0 invariant #5"
but the landed plan invariant is #8 (the dispatcher-fallback contract).
The §5 numbering came from an earlier draft of the plan amendment;
final landed numbering is #8 because three additional invariants (#5
no-mutation, #6 recipe-default, #7 shape-audit) had already accreted.

One-line fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(advisor/tidb): soft-fail getTiDBNodes on bridge miss

Codex P1 catch on PR #20179.

Without this fix, post-§1.5.N+1 dispatcher flip:
  ParsedStatement.AST = *OmniAST → un-migrated advisor calls
  getTiDBNodes → tidbparser.GetTiDBAST(stmt.AST) →
  AsPingCapAST fails (pingcap rejects what omni accepted) →
  GetTiDBAST returns (nil, false) →
  getTiDBNodes returns 'AST type mismatch: expected TiDB parser result' →
  advisor.Check propagates as 'Rule check failed' →
  every un-migrated advisor breaks for that review

That breaks the migration continuity the bridge is meant to preserve.
Phase 1.5 invariant #2 says soft-fail (skip statement, review
continues), not hard-fail.

Fix: when GetTiDBAST returns false AND stmt.AST is a
PingCapASTProvider (i.e. an *OmniAST that just routed through the
bridge and missed), skip the statement instead of returning an error.
The bridge has already logged the parse failure at debug level.

True engine-mismatch (stmt.AST is some unrelated type, neither
*tidb.AST nor a PingCapASTProvider) is still surfaced as an error —
that's a programmer error in registration, not a soft-fail case.

Regression test:
  TestGetTiDBNodesSoftFailsOnBridgeMiss feeds a context with two
  ParsedStatements: one *OmniAST whose AsPingCapAST will fail
  ('SELECT FROM WHERE;') bracketed by one that succeeds ('SELECT 1').
  Expects no error and len(got) == 1 (the successful one).

Sanity-checked: GetTiDBAST returns false on a pingcap-invalid OmniAST,
confirming the pre-fix path would have errored.

Validation:
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green incl. new test
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green
  go test -count=1 ./backend/plugin/parser/tidb/...    # green
  golangci-lint run ...                                # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants