You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
viz smart: guard inequality (Gini/Lorenz) framing for heterogeneous-unit tables; label zero-run as pipeline stage #4222
viz smart --smarter applies inequality framing (Gini + Lorenz curve) to any additive measure with high Gini — but on a table whose unit is a heterogeneous project, a high Gini is expected and is not an equity finding. The inequality vocabulary reads to the public as unfairness, inviting a serious misread.
Why this matters
The CPDB unit is "capital project," and projects are radically heterogeneous — a subway extension should cost a thousand times a playground resurfacing. Concentration across heterogeneous units is nearly tautological, so "NYC capital spending is wildly unequal (Gini 0.96)" is true and meaningless.
Zero-inflation conflates two populations. With 45–60% zeros, the flat run of the Lorenz curve is the zeros — projects with nothing committed/spent yet (a funding-pipeline stage), not "small/poor" projects. Measured inequality is partly a pipeline artifact.
Proposal
Gate or caveat the Gini/Lorenz framing: prefer it when the row unit is a person/household/geography, and add a caveat when the unit is a heterogeneous entity.
When zero-share is high, label the flat Lorenz segment as the zero/not-yet-funded stage and/or additionally report Gini on the non-zero subset.
Consider surfacing the planned → committed → spent pipeline (currently buried in % zeros title annotations) as a funnel — the actually-actionable governance signal.
Related: the inequality-panel encoding itself is covered separately in the skew-aware-encoding issue.
Reproducing example
viz smart --smarter on the NYC Capital Projects Database (Checkbook NYC-sourced), 12,587 rows:
The money columns are extremely right-skewed and zero-inflated (Gini 0.93–0.96; 45–60% zeros).
Filed from a critical design review of the viz smart dashboard; part of a set (skew-aware encoding, robust stats, redundant-dimension collapse, inequality-framing guard, 3D/density panels).
Status (updated 2026-07-25)
Two of the three asks shipped in #4280 (commit c5589d4). This issue stays open for ask 3 only — do not close it as completed.
Done — asks 1 & 2
Every Lorenz panel now carries a caveat line beneath its title:
zero share
subtitle
< 30%
concentration is expected unless rows are comparable units
≥ 30%
flat run = 60% zeros, not small values · concentration is expected unless rows are comparable units
Two decisions worth recording, since both were judgement calls:
The unit caveat is unconditional — there is no gate. There is no row-unit signal anywhere in the stats cache: whether a row is a person, a household, a geography or a heterogeneous capital project is semantics, not a statistic. A name-heuristic proxy would either delete a legitimate equity finding or silently keep a misleading one, so every Lorenz panel carries the caveat instead. No gate, no false negatives.
No Gini over the non-zero subset. The panel's Gini is the cached coefficient, never recomputed, so it agrees with the box hint / pivotp / scoresql. A subset Gini would break that contract for a second number readers would conflate with the first. The zero-run label conveys the same information without recomputing anything.
The zero share comes from the existing zero_share() helper, and ZERO_SHARE_MIN was hoisted out of box_shape_hint so the caveat and the % zeros box-title hint read one threshold through one helper — they appear on the same dashboard and must not disagree.
Remaining — ask 3: the pipeline funnel
Surfacing planned → committed → spent as a funnel is untouched. It is a larger piece of work than the two above:
a new PanelKind plus its render arms (typed subplot grid and inline), and a new data pass
an unsolved design question: how does viz smart recognise that a set of measures forms a pipeline? Column-name heuristics, dictionary concept/role, or an explicit flag. This has the same shape as the unit-heterogeneity problem above — it is semantics, not a statistic — so it deserves the same explicit treatment rather than a guessed proxy.
a monotonicity question: the funnel framing only holds if each stage is a subset of the prior one, which is worth verifying from the data rather than assuming.
Note on ask 3 (the funnel), from implementing it in #4281.
The motivating dataset is not a valid funnel, and the implementation correctly declines to draw one for it. Worth recording so this isn't later filed as a bug.
Running viz smart on the real CPDB (12,587 rows, fetched from NYC Open Data):
viz smart: pipeline funnel skipped — commit_total is not contained
in totalplannedcommit (80% of rows)
The three columns are not nested:
stage
total
violations
totalplannedcommit
$191.9B
—
commit_total
$28.4B
commit > planned on 19.5% of rows
spent_total
$81.0B
spent > commit on 46.3% of rows
Spent is 2.9× committed, and 851 projects have zero planned but non-zero committed. The dataset's own column descriptions explain it:
totalplannedcommit — "total planned commitments for the project allocated in the Capital Commitment Plan"
commit_total — "total committed funding associated with the project within the City's budget"
spent_total — "total funding spent associated with the project within the City's budget"
Three independent aggregates on different bases and time windows, not three stages of one nesting. A funnel over them would assert a containment the source data contradicts — so the detector requires both a stage vocabulary and row-wise containment, and here the second gate refuses. The threshold was deliberately not relaxed to make the panel appear.
Asks 1 and 2 land on this dataset exactly as intended, and it is now a gallery figure for that: Lorenz panels at Gini 0.93/0.96 reading flat run = 60% zeros, not small values — the zero-inflation clause no other gallery dataset exercised.
The funnel itself ships in two forms: auto-detection in viz smart for column-shaped pipelines, and a new standalone qsv viz funnel for pipelines encoded as rows (a stage column plus a value column), which is the shape the smart path explicitly declines to guess at.
Activity
viz smart: guard inequality framing on Lorenz panels (#4222); name me…
viz gallery: add the real NYC Capital Projects dashboard
jqnatividad commentedon Jul 25, 2026
Note on ask 3 (the funnel), from implementing it in #4281.
The motivating dataset is not a valid funnel, and the implementation correctly declines to draw one for it. Worth recording so this isn't later filed as a bug.
Running
viz smarton the real CPDB (12,587 rows, fetched from NYC Open Data):The three columns are not nested:
totalplannedcommitcommit_totalspent_totalSpent is 2.9× committed, and 851 projects have zero planned but non-zero committed. The dataset's own column descriptions explain it:
totalplannedcommit— "total planned commitments for the project allocated in the Capital Commitment Plan"commit_total— "total committed funding associated with the project within the City's budget"spent_total— "total funding spent associated with the project within the City's budget"Three independent aggregates on different bases and time windows, not three stages of one nesting. A funnel over them would assert a containment the source data contradicts — so the detector requires both a stage vocabulary and row-wise containment, and here the second gate refuses. The threshold was deliberately not relaxed to make the panel appear.
Asks 1 and 2 land on this dataset exactly as intended, and it is now a gallery figure for that: Lorenz panels at Gini 0.93/0.96 reading
flat run = 60% zeros, not small values— the zero-inflation clause no other gallery dataset exercised.The funnel itself ships in two forms: auto-detection in
viz smartfor column-shaped pipelines, and a new standaloneqsv viz funnelfor pipelines encoded as rows (a stage column plus a value column), which is the shape the smart path explicitly declines to guess at.describegpt: infer and emit `kind: "pipeline"` relationships (#4222)
viz smart: drive the pipeline funnel from the dictionary (#4222)
viz gallery: showcase the pipeline funnel on the CPDB dashboard (#4222)
viz smart: bridge declared pipelines whose stages do not nest (#4222) (…
viz smart: dictionary-declared pipeline funnel (dathere#4222) (dather…