fix(staggered): scale-equilibrate CS / StaggeredTripleDiff covariate OR fits by igerber · Pull Request #570 · igerber/diff-diff · GitHub
Skip to content

fix(staggered): scale-equilibrate CS / StaggeredTripleDiff covariate OR fits#570

Merged
igerber merged 1 commit into
mainfrom
fix/or-nuisance-scale-equilibration
Jun 28, 2026
Merged

fix(staggered): scale-equilibrate CS / StaggeredTripleDiff covariate OR fits#570
igerber merged 1 commit into
mainfrom
fix/or-nuisance-scale-equilibration

Conversation

@igerber

@igerber igerber commented Jun 28, 2026

Copy link
Copy Markdown
Owner

Summary

  • Route the covariate outcome-regression (OR) nuisance fits in CallawaySantAnna
    (_compute_all_att_gt_covariate_reg, _doubly_robust) and StaggeredTripleDifference
    (_compute_or) through the shared scale-equilibrated solve_ols (column-equilibrated
    SVD/gelsd, return_vcov=False), replacing the prior estimator-local cho_solve(X'X) cache fast
    path (+ bare scipy.linalg.lstsq(cond=1e-7) fallbacks) that bypassed it. Matches
    TripleDifference._fit_predict_mu and R's lm()/QR. Dropped the now-dead cho-cache plumbing.
    Dropped-column NaN coefficients are zeroed for prediction (np.where(np.isnan...)), preserving
    the numerical-failure → NaN-cell safety path.
  • What this fixes (honest scope): a covariate correlated with another regressor at a very
    large scale
    (e.g. a large constant offset, near-collinear with the intercept) previously
    perturbed the point-estimate ATT — and the influence-function SE that follows it — because
    forming the normal equations squares the condition number. The equilibrated SVD is
    offset-invariant to ~1e-11 where the prior solve drifted (~4e-6 at an offset of 1e6, growing with
    scale). Pure orthogonal ill-scaling was already safe (diagonal X'X), so the practical impact
    is confined to ill-conditioned / correlated covariate designs — this is a robustness + R-fidelity
    improvement, not a "fixes visibly-wrong ATTs" change. Not bit-identical (cho/normal-equations
    → SVD); well-scaled designs move only ~1e-12.
  • Retires the CallawaySantAnna / StaggeredTripleDifference OR-nuisance scale-equilibration row
    in TODO.md.

Methodology references (required if estimator / math changes)

  • Method name(s): CallawaySantAnna (Callaway & Sant'Anna 2021) + StaggeredTripleDifference
    doubly-robust / regression-adjustment outcome-regression nuisance solve.
  • Paper / source link(s): docs/methodology/REGISTRY.md — the CallawaySantAnna and
    StaggeredTripleDifference scale-invariance / scope Notes are updated to reflect the fix. R
    reference: did::att_gt / base R lm() (QR decomposition), the same orthogonalization family as
    the SVD used here.
  • Any intentional deviations from the source (and why): None. This aligns the OR solver with the
    documented scale-robust standard already used by TripleDifference and matches R's QR. No change
    to estimands, identifying assumptions, IF-variance formulas, aggregation, the no-covariate path,
    or the propensity-score (logit) fits.

Validation

  • Tests added/updated: tests/test_methodology_callaway.py (CS reg scale-invariance),
    tests/test_methodology_staggered_triple_diff.py (StaggeredTripleDiff reg scale-invariance),
    tests/test_csdid_ported.py (new with_covariates_dr golden att + SE R-did parity, the
    previously-unasserted covariate scenario), tests/test_staggered.py (two nan_cell mock tests
    re-pointed from scipy.linalg.lstsq to the solve_ols seam).
  • Evidence: scale-invariance proven (covariate + 1e6 offset → ATT(g,t) invariant to ~1e-11 under
    the new code vs ~4e-6 drift under the old, captured via git stash); no-regression on
    well-scaled designs (max|dev| 6.66e-16 across 96 values); negligible perf (≈0.008 s for a
    reg fit on a 1500×10 covariate panel — the OR solve is small, so dropping the cho-cache costs
    nothing); CS covariate R-did parity (dr att 9.5e-4 / SE 1.4e-4; reg/ipw att ~1e-11,
    unchanged). 455 affected-suite tests pass; source black/ruff clean; mypy 0 new errors vs
    baseline.

Security / privacy

  • Confirm no secrets/PII in this PR: Yes

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

@igerber igerber force-pushed the fix/or-nuisance-scale-equilibration branch from 050b016 to 35dca44 Compare June 28, 2026 18:18
@github-actions

Copy link
Copy Markdown

🔁 AI review rerun (requested by @igerber)

Head SHA: 35dca444b5a9e6ff4a6b71bf160c40f92f4f83c9


Overall Assessment: ⚠️ Needs changes

Executive Summary

  • Prior P1 is only partially resolved: CS reg underdetermined control cells no longer crash, but the new branch bypasses the documented R-style dropped-column solve.
  • The main full-rank scale-equilibration change is methodologically sound: replacing normal-equation Cholesky with scale-equilibrated SVD/solve_ols is an implementation choice aligned with the registry.
  • Registry/TODO drift from the previous review is fixed.
  • No new inline inference, parameter propagation, security, or performance issues found.
  • One stale test still patches scipy.linalg.lstsq, which no longer exercises the optimized OR solver path.

Methodology

Finding 1

Severity: P1
Location: diff_diff/staggered.py:L1402-L1424, docs/methodology/REGISTRY.md:L538-L542
Impact: The underdetermined CS covariate reg fallback calls _equilibrated_lstsq(pair_X_ctrl, control_change) on the full n < k design. That avoids the solve_ols() crash, but it uses a minimum-norm solution with finite coefficients for all columns instead of the registry’s R-style contract: detect rank deficiency, drop unidentified columns, and set dropped coefficients to NaN before zeroing for prediction. In underdetermined OR cells, extrapolation to treated covariates is not unique, so this can silently change ATT(g,t) and the SE that follows it relative to the documented lm()/solve_ols column-drop convention. The nearby warning says dropped columns are excluded, but this branch does not exclude them.
Concrete fix: In the n < k branch, run _detect_rank_deficiency(pair_X_ctrl), build kept_cols, solve _equilibrated_lstsq(pair_X_ctrl[:, kept_cols], control_change), then expand dropped coefficients to NaN and let the existing NaN zero-fill handle prediction. Alternatively, move this rank-drop-before-n < k behavior into solve_ols(). Add a regression test where the minimum-norm full-design prediction differs from the rank-reduced prediction, and assert the CS ATT matches the rank-reduced/R-style path.

Code Quality

No findings.

Performance

No findings.

Maintainability

No findings.

Tech Debt

No findings. The removed TODO row is appropriate once the P1 underdetermined-cell contract is fixed.

Security

No findings.

Documentation/Tests

Finding 2

Severity: P3
Location: tests/test_staggered.py:L1009-L1034
Impact: test_nan_cell_preserved_not_dropped still patches scipy.linalg.lstsq, but the optimized OR path now reaches solve_ols / the imported scipy_lstsq alias, so this mock may not poison any OR solve. The test can pass without exercising the NaN-cell guard it describes.
Concrete fix: Patch diff_diff.staggered.solve_ols as the later tests do, or patch the actual diff_diff.linalg solver seam being exercised, and assert call_count > 0 plus the expected NaN cell.

Path to Approval

  1. Replace the underdetermined CS reg fallback with a rank-reduced, dropped-column solve before prediction.
  2. Add a targeted underdetermined-cell test that checks R-style dropped-column semantics, not only “no crash.”
  3. Update the stale test_nan_cell_preserved_not_dropped mock to patch the active solver seam.

…OR fits

CallawaySantAnna (_compute_all_att_gt_covariate_reg, _doubly_robust) and
StaggeredTripleDifference (_compute_or) fit their covariate outcome-
regression nuisance via an estimator-local cho_solve(X'X) cache fast path
(with bare scipy.linalg.lstsq(cond=1e-7) fallbacks) that bypassed the shared
scale-equilibrated solver. Route them through solve_ols (column-equilibrated
SVD/gelsd, return_vcov=False), matching TripleDifference._fit_predict_mu and
R's lm()/QR; zero only dropped-column NaN coefficients for prediction
(np.where(np.isnan...)) so the numerical-failure -> NaN-cell safety path is
preserved. Drop the now-dead cho-cache plumbing.

A covariate correlated with another regressor at a very large scale (e.g. a
large constant offset, near-collinear with the intercept) previously
perturbed the point-estimate ATT (and the IF SE that follows it) because
forming the normal equations squares the condition number; the equilibrated
SVD is offset-invariant to ~1e-11 where the prior solve drifted (~4e-6 at
offset 1e6). Pure orthogonal ill-scaling was already safe (diagonal X'X), so
the practical impact is confined to ill-conditioned / correlated covariate
designs. Not bit-identical (cho/normal-equations -> SVD); well-scaled designs
move only ~1e-12.

Tests: reg-anchored scale-invariance tests for CS and StaggeredTripleDiff
(covariate + 1e6 offset -> ATT(g,t) unchanged); a new with_covariates_dr CS
golden parity test (att+SE vs R `did`, ~1e-3); re-pointed two brittle
nan_cell tests from scipy.linalg.lstsq to the solve_ols seam. REGISTRY
scale-invariance/scope notes flipped to "fixed"; CHANGELOG; removed the TODO
scale-equilibration row.

No change to estimands, identifying assumptions, the no-covariate path, or
the propensity-score fits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@igerber igerber force-pushed the fix/or-nuisance-scale-equilibration branch from 35dca44 to e252f16 Compare June 28, 2026 18:30
@github-actions

Copy link
Copy Markdown

@igerber igerber added the ready-for-ci Triggers CI test workflows label Jun 28, 2026
@igerber igerber merged commit 12b4480 into main Jun 28, 2026
33 of 34 checks passed
@igerber igerber deleted the fix/or-nuisance-scale-equilibration branch June 28, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-ci Triggers CI test workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant