feat: add unindexed-scan query counter by imilinovic · Pull Request #4282 · memgraph/memgraph · GitHub
Skip to content

feat: add unindexed-scan query counter#4282

Draft
imilinovic wants to merge 10 commits into
masterfrom
feat/metrics-no-index-counter
Draft

feat: add unindexed-scan query counter#4282
imilinovic wants to merge 10 commits into
masterfrom
feat/metrics-no-index-counter

Conversation

@imilinovic

Copy link
Copy Markdown
Contributor

Adds a per-database metric memgraph_unindexed_scan_queries_total{database,uuid} that counts query executions whose plan does a ScanAll+Filter an index could have served.

What it does
Surfaces, as an aggregable metric, how often queries run a full scan + filter when an index could have served the scan — previously only emitted as a per-query planner hint notification. Exposed via OpenMetrics and SHOW METRICS INFO (row UnindexedScanQueries).

How
The plan-hint provider already detects this case at plan time; it now also reports a has_unindexed_scan bool through PlanHintsResult. The interpreter increments the counter once per qualifying query at the regular-execution site only (EXPLAIN/PROFILE do not count). Counting is per-query: a query with several qualifying scans bumps the counter by 1, so rate(unindexed)/rate(queries) is a stable ratio. Wired through the existing per-database DatabaseMetricHandles machinery (Colin's per-tenant pattern).

Why
Operators had no fleet-level signal for missing-index workloads — only a per-query notification. This gives a canary ("is this happening / is the rate growing?") across tenants. It is intentionally a per-query occurrence signal, not a cost signal (see follow-ups for slow-query enrichment / scan amplification). The legacy JSON metrics endpoint is being deprecated, so the counter is OpenMetrics + SHOW METRICS INFO only.

@imilinovic

Copy link
Copy Markdown
Contributor Author

@imilinovic imilinovic self-assigned this Jun 18, 2026
@imilinovic imilinovic added Docs needed Docs needed feature feature labels Jun 18, 2026
@imilinovic imilinovic added this to the mg-v3.12.0 milestone Jun 18, 2026
@imilinovic imilinovic marked this pull request as draft June 18, 2026 13:09
@imilinovic imilinovic force-pushed the feat/metrics-no-index-counter branch 2 times, most recently from 18e18cc to ddf5c6a Compare June 23, 2026 13:09
Review-driven refinements to the query no-index-lookup counter:

- Count per-query, not per-operator: a query with N qualifying ScanAll+Filter
  scans now bumps memgraph_query_no_index_lookup_total by 1. Makes
  rate(no_index)/rate(queries) a stable alert ratio. The per-operator count is
  still produced in PlanHintsResult for future structured use.
- Unit-test the counter: query_hint_provider asserts no_index_lookup_count in
  every case (Cartesian -> 2, ScanAllByLabel+property -> 0). Was untested before.
- Rename PlanHintsProvider::hints() -> take_hints() (returns by value) to drop
  the misleading std::move on an lvalue-ref accessor.
- Reword the counter Help string for per-query semantics; document the
  intentional ScanAllByLabel non-increment and the OpenMetrics/SHOW-only scope
  (no deprecated JSON-global aggregation).
- e2e: UNION asserts +1 (per-query); add a PROFILE no-op assertion.
The exposed metric is per-query (counts once when a plan has any qualifying
ScanAll+Filter), so the internal per-operator count was never used as a
magnitude. Replace PlanHintsResult::no_index_lookup_count (size_t) with
has_no_index_lookup (bool): expresses the actual contract, removes a field
that had to stay in sync with hints_, and makes per-operator exposure
impossible to reintroduce by accident. Branch-firing coverage is retained via
the existing hint-message assertions.
"no_index_lookup" was ambiguous (a lookup that found no index, or the absence
of a lookup?) and didn't say a scan happened. Rename to the clearer, convention-
aligned (memgraph_*_queries_total) name before release:

- metric: memgraph_query_no_index_lookup_total -> memgraph_unindexed_scan_queries_total
- SHOW METRICS INFO row: QueryNoIndexLookup -> UnindexedScanQueries
- family/handle: query_no_index_lookup* -> unindexed_scan_queries*
- PlanHintsResult field: has_no_index_lookup -> has_unindexed_scan

Pure rename; behavior and tests unchanged.
…ning

The plan-hint provider classified scans with an exact `GetTypeInfo() ==
ScanAll::kType` check. Parallel-execution plans rewrite `Filter -> Scan*`
into `Filter -> ScanChunk -> ParallelMerge -> ScanParallel*`, so the filter
sits on a `ScanChunk` (its own kType), which fell through silently and never
incremented `memgraph_unindexed_scan_queries_total`.

Add `EffectiveScanType` to resolve a `ScanChunk` down to the underlying
`ScanParallel*` and map it back to the equivalent sequential type
(`ScanParallel` -> `ScanAll`, `ScanParallelByLabel` -> `ScanAllByLabel`), so
parallel plans count identically to sequential ones. Index-backed parallel
scans stay unflagged.

Review hardening:
- reuse the guarded `dynamic_cast<ScanAll*>` instead of casting twice
- make `take_hints()`/`has_unindexed_scan()` private + `friend ProvidePlanHints`
  (+ `[[nodiscard]]`) so result accessors can't be read mid-traversal
- extract `EmitPlanHints` for the three duplicated hint loops; keep the counter
  increment only on the regular-execution path, with explicit comments on the
  EXPLAIN/PROFILE no-op and the before-AccessorCompliance ordering
- `Increment(1.0)` -> `Increment()`
- document the property-only-filter non-count case and the JSON-endpoint omission

Tests:
- unit: parallel unindexed scan counts, parallel label scan does not,
  property-only sequential scan does not
- e2e: `USING PARALLEL EXECUTION` unindexed scan counts (+1) and is silenced by
  a matching index
Apply consolidated review feedback (no behavior change to the metric):
- Fix misleading comment: ScanParallelByEdge is a full edge scan, not
  index-backed; state the out-of-scope reason explicitly.
- Reword counter help text to "Planned queries" to match counting before
  AccessorCompliance.
- Mark ProvidePlanHints [[nodiscard]].
- Move HintIndexUsage and EffectiveScanType out-of-line into the .cpp.
- Align metric section labels with the emitted "Query" category.
- Add regression tests: property-only parallel scan and ScanChunkByEdge
  parallel pass-through; make WrapInParallelChunk a fixture helper.
- Condense over-long comments.
- assert ScanChunk/ParallelMerge chain invariants instead of dead null guards
- add unit test covering the ScanParallelByLabelProperties (indexed) branch
- spell out trigger and exclusion conditions in the counter help text
- trim verbose comments to one line; fix the misleading edge-scan note
Replace the no-op DMG_ASSERTs in EffectiveScanType (compiled out under
NDEBUG) with always-on guards that fall back to the scan's own type when
the parallel-rewrite chain has an unexpected shape, so a malformed chain
can no longer misclassify or null-deref in release builds.

Also tighten review-flagged comments: drop the chain description that was
duplicated from the function header, correct the EmitPlanHints contract
comment (it never touches the counter), and collapse the metric-handle
comment to one line.
Reword the metric Help and the EffectiveScanType / has_unindexed_scan
comments so they cover parallel-execution plans (not just the sequential
ScanAll), correct the parallel-rewrite chain description, and fix a
dropped word. Comment/string only; no behavior change.
@imilinovic imilinovic force-pushed the feat/metrics-no-index-counter branch from ddf5c6a to ce59097 Compare June 28, 2026 15:07
- Make the EXPLAIN/PROFILE no-count rule structural: EmitPlanHints now takes
  a DatabaseMetricHandles* and does the increment itself, so every call site
  must explicitly opt in (Cypher passes handles, EXPLAIN/PROFILE pass nullptr).
- Add PlanHintsProvider::AddUnindexedScanHint so a missing-index hint and the
  has_unindexed_scan_ flag are set in lockstep; suboptimal-index branch stays
  plain push_back. Fix the stale "sets accordingly" doc comment.
- Make PlanHintsProvider single-use (delete move ctor) and move the friend
  declaration next to the private accessors it gates.
- e2e: track a running expected count instead of baseline+N chaining; drop
  internal task reference from the docstring.
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Docs needed Docs needed feature feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant