refactor(advisor/tidb): migrate 4 STATEMENT_DISALLOW_* rules to omni AST (Phase 1.5 §1.5.3 batch 3) by vsai12 · Pull Request #20199 · bytebase/bytebase · GitHub
Skip to content

refactor(advisor/tidb): migrate 4 STATEMENT_DISALLOW_* rules to omni AST (Phase 1.5 §1.5.3 batch 3) - #20199

Merged
d-bytebase merged 4 commits into
mainfrom
feat/tidb-omni-advisor-batch-3
May 7, 2026
Merged

d-bytebase merged 4 commits into
mainfrom
feat/tidb-omni-advisor-batch-3

Conversation

@vsai12

@vsai12 vsai12 commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

First Class IIa batch — advisors that consume both AST nodes AND the statement's source text for advice content. Four rules migrated:

Rule File Recipe
STATEMENT_DISALLOW_COMMIT advisor_stmt_disallow_commit.go A — top-level type switch on *ast.CommitStmt
STATEMENT_SELECT_NO_SELECT_ALL advisor_stmt_no_select_all.go B — omniast.Walk (wildcard can appear in nested subqueries)
STATEMENT_DISALLOW_ORDER_BY advisor_stmt_disallow_order_by.go A — top-level switch on Update/Delete
STATEMENT_DISALLOW_LIMIT advisor_stmt_disallow_limit.go A — top-level switch on Update/Delete/Insert

Migration progress: 14 of 51 advisors on omni AST.

Independent of PRs #20195 (batch 1) and #20198 (batch 2) — different files, parallel-mergeable.

First Recipe B in the migration

no_select_all is the first advisor in this migration that genuinely required Recipe B. The existing yaml fixture exercises this with:

SELECT a, b FROM (SELECT * FROM t1 JOIN t2) t

The outer SELECT has no wildcard; the inner subquery does. A top-level type switch silently misses it (caught immediately by TestTiDBRules). Switched to omniast.Walk with a Visit(node) Visitor implementation — returning v to recurse, nil to skip subtrees, and ignoring the Visit(nil) post-order signal since this advisor has no post-order work.

New shared helper: OmniStmt.TrimmedText()

Added to utils.go:

func (s OmniStmt) TrimmedText() string {
    return strings.TrimSpace(s.Text)
}

Mirrors mysql omni's TrimmedStmtText pattern. Needed because multi-statement input passes leading newlines through Statement.Text; pingcap's node.Text() returned the trimmed form. Without this helper, advice content embeds raw newlines:

  • Pre-fix: Commit is not allowed, related statement: "\nCOMMIT;"
  • Post-fix: Commit is not allowed, related statement: "COMMIT;"

All 4 advisors in this batch use it.

Notable API mappings

Pingcap Omni
node.Order != nil (UpdateStmt/DeleteStmt) len(node.OrderBy) > 0
node.Limit != nil node.Limit != nil (same shape)
node.Select (interface, can be *SelectStmt or *SetOprStmt) node.Select (concrete *SelectStmt; UNION/INTERSECT/EXCEPT collapse to a single SelectStmt with a SetOp field)
node.Fields.Fields[].WildCard != nil *ast.StarExpr in TargetList[]
*ast.CommitStmt *ast.CommitStmt (same)
node.Text() (statement source text, trimmed by parser) ostmt.TrimmedText()

Pre-batch checklist

  1. Coupling scan. useLimit was local to disallow_limit.go; inlined as insertUsesLimit during migration. No cross-file coupling found.
  2. Shape audit. Pingcap's *ast.SetOprStmt (a separate type for SELECT ... UNION SELECT ...) collapses to omni's *SelectStmt with a SetOp field — same kind of unification we saw in batch 2's ConstrUnique. The branch logic for useLimit simplifies from a type switch to a single field check.
  3. Yaml fixture coverage. TestTiDBRules green for all 4 rules. no_select_all's nested-subquery case is the regression check that forced Recipe B.
  4. Pingcap-import quarantine. Clean for the 4 migrated files.

Validation

go test -count=1 ./backend/plugin/advisor/tidb/...   # green (TestTiDBRules covers all 4 rules)
go test -count=1 ./backend/plugin/advisor/mysql/...  # green (cross-engine regression)
go test -count=1 ./backend/plugin/parser/tidb/...    # green
go build -ldflags "-w -s" -p=16 -o ./bytebase-build/bytebase ./backend/bin/server/main.go  # green
golangci-lint run --allow-parallel-runners ./backend/plugin/advisor/tidb/...  # 0 issues

References

Test plan

  • TestTiDBRules passes for all 4 migrated rules (incl. nested-subquery wildcard for no_select_all)
  • mysql cross-engine regression passes
  • golangci-lint clean
  • Build succeeds
  • Reviewer verifies Recipe B usage in no_select_all is sound (Visit returns v to recurse, no post-order work needed)
  • Reviewer verifies TrimmedText helper is the right granularity (vs per-call strings.TrimSpace at sites)
  • Reviewer verifies SetOprStmt → SelectStmt-with-SetOp unification preserves pingcap behavior

🤖 Generated with Claude Code

…AST (Phase 1.5 §1.5.3 batch 3)

First Class IIa batch — advisors that consume both AST nodes AND the
statement's source text for advice content. Four rules migrated:

  - advisor_stmt_disallow_commit.go    (STATEMENT_DISALLOW_COMMIT)
  - advisor_stmt_no_select_all.go      (STATEMENT_SELECT_NO_SELECT_ALL)
  - advisor_stmt_disallow_order_by.go  (STATEMENT_DISALLOW_ORDER_BY)
  - advisor_stmt_disallow_limit.go     (STATEMENT_DISALLOW_LIMIT)

Three use Recipe A (top-level type switch). One — no_select_all —
uses Recipe B (omniast.Walk) because the wildcard can appear in
nested subqueries; the existing fixture exercises this with
'SELECT a FROM (SELECT * FROM t1 JOIN t2) t' and a top-level switch
silently skips the inner SELECT.

Migration progress: 14 of 51 advisors on omni AST.

New shared helper:
  - OmniStmt.TrimmedText() in utils.go — returns Text with surrounding
    whitespace removed, for embedding into advice content. Multi-
    statement input passes leading newlines through Statement.Text;
    pingcap's node.Text() returned the trimmed form.

Notable mappings:
  pingcap UpdateStmt/DeleteStmt   omni
    Order != nil          →       len(OrderBy) > 0
    Limit != nil          →       Limit != nil
  pingcap InsertStmt              omni
    Select interface (SelectStmt | SetOprStmt) → SelectStmt with SetOp field;
                                    omni unifies both forms, single branch covers them.
  pingcap SelectStmt              omni
    Fields.Fields[].WildCard →    *StarExpr in TargetList[]
  pingcap CommitStmt              omni
    *ast.CommitStmt        →      *ast.CommitStmt (same)

Pre-batch checklist:
  1. Coupling scan: clean — useLimit was local to disallow_limit.go;
     inlined as insertUsesLimit during migration.
  2. Shape audit: pingcap *ast.SetOprStmt collapses to omni's
     *SelectStmt with SetOp field (similar to migration_compatibility's
     ConstrUnique unification). Fixture exercises the inner-subquery
     wildcard case which forced Recipe B for no_select_all.
  3. Yaml fixture coverage: TestTiDBRules green for all 4 rules.
  4. Pingcap-import quarantine: clean for migrated files.

Validation:
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green
  go test -count=1 ./backend/plugin/parser/tidb/...    # green
  go build ...                                         # green
  golangci-lint run ...                                # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vsai12
vsai12 requested a review from a team as a code owner May 4, 2026 23:56
@cla-bot cla-bot Bot added the cla-signed label May 4, 2026
…overage

Round-1 review feedback on PR #20199.

#1 Line-number parity for no_select_all. The pre-migration pingcap
advisor set checker.line once per top-level statement, so every
wildcard hit (including nested-subquery hits) reported the OUTER
statement's line. The omni Recipe B port emitted per-match — using
the inner SelectStmt's Loc — which was an arguably-better behavioral
change but unacknowledged.

Reverted to outer-statement line for migration discipline: capture
ostmt.AbsoluteLine(0) once per statement and reuse for all advices.
Improving advice line precision is a follow-up decision, not a
side-effect of parser migration.

#2 UNION fixture coverage. omni's Walk recurses into SelectStmt.Left
and .Right (verified in walk_generated.go:756-762), so Recipe B
correctly visits UNION arm subqueries. Added three fixture cases to
pin the contract:

  - SELECT * FROM t1 UNION SELECT a FROM t2  → flagged (left arm)
  - SELECT a FROM t1 UNION SELECT * FROM t2  → flagged (right arm)
  - SELECT a FROM t1 UNION SELECT b FROM t2  → no advice (control)

#2 (cont) insertUsesLimit comment. Added documentation that only
the outermost LIMIT is checked — a LIMIT inside a parenthesized
UNION arm lives on Left.Limit, not on the outer .Limit, and was
not flagged by the pre-migration advisor either. Pingcap parity
preserved, behavior intentionally not extended.

Validation:
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green
  golangci-lint run ...                                # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d4a7ee55c1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread backend/plugin/advisor/tidb/advisor_stmt_no_select_all.go Outdated
Codex P2: 4 findings of line-number regression for inputs with leading
trivia (whitespace + comments). Empirically verified:

  Input: "\n-- cmt\nSTMT;"
    pingcap (main):    line 2
    omni (this PR):    line 1   ← regression

Codex's claimed direction was reversed for 3 of the 4 cases (claimed
line 3, actual line 1) but the regression itself is real for all 4.

Root cause: pingcap's lexer strips leading whitespace but keeps
comments in node.Text(), so its OriginTextPosition points at the line
of the first comment OR keyword. Omni's Loc.Start is 0 (start of
segment) for top-level statements; AbsoluteLine(0) gives BaseLine+1.

Fix: new OmniStmt.FirstTokenLine() helper finds the line of the first
non-whitespace character in s.Text. Matches pingcap behavior for the
common cases:

  | Input                            | pingcap | FirstTokenLine |
  |----------------------------------|---------|----------------|
  | "COMMIT;"                        | 1       | 1              |
  | "\n-- cmt\nCOMMIT;"              | 2       | 2              |
  | "\n/* block */\nCOMMIT;"         | 2       | 2              |

One edge case diverges: "   \n  COMMIT;" (leading whitespace, no
comments) — pingcap reports 1, FirstTokenLine reports 2. pingcap's
lexer preserves whitespace-only leading text in node.Text() so its
leadingNewlinesStripped is 0 here. Real SQL rarely has this shape;
both reported lines reasonably point at the statement.

All 4 batch-3 advisors (commit, no_select_all, disallow_order_by,
disallow_limit) updated to use FirstTokenLine() instead of
AbsoluteLine(0) or AbsoluteLine(n.Loc.Start).

Validation:
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green
  golangci-lint run ...                                # 0 issues

Empirical line-parity test across 6 input shapes: 5/6 match pingcap;
the 1 mismatch is the contrived leading-whitespace-only case
documented above.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 793cc8a1a8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread backend/plugin/advisor/tidb/advisor_stmt_no_select_all.go Outdated
Codex P2: targetListHasStar only matched *ast.StarExpr, missing the
qualified-wildcard form. Empirically verified omni's TargetList shape:

  SELECT *      → *ast.StarExpr
  SELECT t.*    → *ast.ColumnRef with Star: true (NOT a StarExpr)

Pre-migration pingcap path used field.WildCard != nil, which was set
for both bare and qualified wildcards. omni splits them into two node
types; targetListHasStar must check both.

Fix: extend the type switch to also match *ast.ColumnRef with
Star == true.

Sanity-checked by reverting: TestTiDBRules fails on the new
"SELECT t.* FROM t" fixture case; restored fix passes.

Fixture additions:
  - SELECT t.* FROM t          → flagged (qualified)
  - SELECT a, t.* FROM t       → flagged (mixed with column)
  - SELECT a, t.b FROM t       → no advice (control: qualified non-star)

Validation:
  go test -count=1 ./backend/plugin/advisor/tidb/...   # green
  go test -count=1 ./backend/plugin/advisor/mysql/...  # green
  golangci-lint run ...                                # 0 issues

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented May 5, 2026

Copy link
Copy Markdown

@d-bytebase
d-bytebase merged commit bb0e73e into main May 7, 2026
16 checks passed
@d-bytebase
d-bytebase deleted the feat/tidb-omni-advisor-batch-3 branch May 7, 2026 03:51
vsai12 added a commit that referenced this pull request May 7, 2026
… AST (Phase 1.5 §1.5.3 batch 4) (#20204)

* refactor(advisor/tidb): migrate naming index/UK/FK convention to omni AST (Phase 1.5 §1.5.3 batch 4)

Migrate 3 advisors to omni AST via Recipe A (top-level type-switch):
- advisor_naming_index_convention.go
- advisor_naming_unique_key_convention.go
- advisor_naming_foreign_key_convention.go

Pre-batch coupling scan promoted 2 cross-file helpers to utils.go:
- getTemplateRegexp (function)
- indexMetaData (struct)

Both are AST-agnostic Go (no pingcap types in their signatures), so the
move is clean. No Linear cleanup ticket needed (unlike canNull/BYT-9362
or tableNewColumn/BYT-9395, which stayed pingcap-typed for un-migrated
consumers).

Also added OmniStmt.TrimmedText() and OmniStmt.FirstTokenLine() helpers
to utils.go. Same helpers also defined on PR #20199 (batch 3); whichever
batch merges second drops the duplicate definition (trivial conflict).

NEW shape divergence caught and pinned:

omni splits ALTER TABLE ADD INDEX (ATAddIndex) vs ADD CONSTRAINT name
INDEX (ATAddConstraint) where pingcap collapsed both into
AlterTableAddConstraint. Same split applies to UNIQUE: ADD UNIQUE INDEX
goes to ATAddIndex, ADD CONSTRAINT name UNIQUE goes to ATAddConstraint.
Switch arms now cover both, mirroring mysql omni rule.

Fixture pins for the discovered divergence:
- naming_index_idx.yaml: ALTER TABLE ADD CONSTRAINT name INDEX
  (pass + violation pair).
- naming_index_uk.yaml: ALTER TABLE ADD UNIQUE INDEX, ALTER TABLE ADD
  UNIQUE KEY (4 cases: 2 forms x pass+violation each). These also pin
  the existing ConstrUnique unification (cumulative table item #2).

Behavior preserved (NOT aligned with mysql divergence):

CREATE FULLTEXT/SPATIAL INDEX statements continue to be checked by the
index naming convention advisor (matching pingcap behavior). The mysql
omni rule excludes them; aligning tidb with mysql on this is a separate
intentional behavior change deferred from this AST migration.

Migration progress: 17 of 51 advisors on omni AST.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(advisor/tidb): extract runNamingConventionRule helper to dedup naming trio

Sonar flagged 21.9% duplication across the 3 new naming-convention
advisors (33.6% / 32.2% / 11.0% per file). The duplication was real:
~140 LOC of structural overlap across the trio in the Check() preamble
(payload validation, template-token check, maxLength default) and the
advice-emission loop (regex match + length overflow advices).

Extracted runNamingConventionRule(checkCtx, cfg, collect) in utils.go.
Each advisor's Check() drops to ~17 lines: a config struct (mismatch
code, type noun, internal-error title) plus a closure that maps statement
nodes to []*indexMetaData. Collector functions (collect{Index,UK,FK}*)
unchanged — those carry the rule-specific logic that genuinely differs.

Net LOC: +93 helper / -167 dedup = -102 LOC. Sonar duplication on the
trio should drop to ~0%.

Mysql analogs left similar duplication inline; that precedent pre-dates
Sonar duplication gating on new code. Since the trio is new code, the
gate applies; the cumulative cost — naming_table, naming_column,
naming_auto_increment_column are queued in the handoff doc and would
each replicate the same ~50 LOC scaffold without the helper — justifies
the extraction now.

Verification:
- TestTiDBRules green — advice content is byte-identical to the
  pre-extraction inline form (helper builds the same "<noun> in table
  ..." and "<noun> `<name>` in table ..." strings via fmt.Sprintf with
  cfg.typeNoun substitution). Existing fixtures are the safety net.
- go test ./backend/plugin/advisor/tidb/ ./backend/plugin/advisor/mysql/
  green (cross-engine non-regression).
- golangci-lint 0 issues.
- Server build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(advisor/tidb): guard nil constraints in collectFKCreateTable

Codex P2 round-1 catch on PR #20204. The line argument
ostmt.AbsoluteLine(constraint.Loc.Start) was evaluated BEFORE
buildFKMetaData could nil-check, so a nil entry in n.Constraints would
panic with a nil pointer dereference instead of being skipped. The
helper's internal `if constraint == nil` was vestigial because of Go's
argument-evaluation order.

Sibling collectIndexCreateTable / collectUKCreateTable already
nil-checked at loop top; collectFKCreateTable was the outlier.
collectFKAlterTable was safe by accident — it precomputes stmtLine
outside the loop.

Fix: nil-check at loop top, mirroring siblings exactly.

Added TestCollectFKCreateTableNilConstraintGuard with a synthetic AST
containing a nil constraint slot. Omni's parser doesn't produce nils
from valid SQL so no YAML fixture can reach this branch — the unit test
pins the contract for future refactors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(advisor/tidb): pin internal-error advice path + clarifying comments

Code review pushback on PR #20204 — the runNamingConventionRule
extraction's byte-identity claim only stood for the regex-mismatch and
length-overflow paths the YAML fixtures exercise. The internal-error
advice path (fires when getTemplateRegexp's regexp.Compile fails on the
post-substitution template) had no test coverage; an accidental swap of
cfg.internalErrorTitle vs cfg.typeNoun, or a regression in Status/Code
fields, would go unnoticed.

Added TestRunNamingConventionRuleEmitsInternalErrorAdvice — uses an
unbalanced-paren format that passes payload-token validation but fails
regexp.Compile after no-op substitution. Asserts the advice's Title
comes from cfg.internalErrorTitle (not cfg.typeNoun or rule.Type),
Code is code.Internal, Status mirrors the rule level, and Content shape
matches the pre-extraction inline form.

Also added clarifying comments per the same review:
- runNamingConventionRule docstring: indexMetaData coupling tradeoff
  (4-field LCD shared by all callers; future divergence may require
  widening or replacing the closure shape).
- omniIndexColumns docstring: silent skip of expression-based parts
  (functional indexes contribute no name to column-list substitution;
  matches pingcap behavior).
- Index/UK Check methods: explain the closure-capture pattern for
  originalMetadata (keeps helper signature uniform across rules that
  don't need metadata).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vsai12 added a commit that referenced this pull request May 7, 2026
…r-batch-2

Conflict on backend/plugin/advisor/tidb/utils.go:
- Kept both batch-2's addColumnTargets helper AND main's TrimmedText/
  FirstTokenLine helpers (introduced by batch 3 / PR #20199, merged).
- Preserved batch-2's intentional concise canNull doc (commit 4e11de1
  trimmed it on purpose) over main's verbose form. Same function body.

Verified post-merge:
- TestTiDBRules + cross-engine mysql tests green.
- parser/tidb tests green.
- golangci-lint 0 issues.
- Server build clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants