# Limitations (living document)

This file is updated throughout the project. Anything checked off here must be true at
time of the result freeze (see `research/claims_registry.md`).

## Scope limitations (by design, stated up front)

- **Not a cosmological discovery claim.** No new physics, no dark-energy claim, no
  anomaly claim. This is a clustering-measurement and correction-methodology study.
- **Small footprint / single tracer to start.** Per the project's own "start small"
  mandate, initial results come from one sky region, one tracer, one redshift range.
  Statistical power is limited accordingly — uncertainties will be reported and are
  expected to be large relative to full-survey analyses.
- **Mock realism is bounded.** Injection-study mocks are simplified relative to full
  survey mocks (e.g. EZmocks/PATCHY): they encode a known clustering field and known
  injected systematics templates, not a full N-body + halo-occupation + fiber-assignment
  forward model. Conclusions about ML vs. linear correction are conditional on this mock
  design and should not be over-generalized to arbitrary systematics.
- **No causal claims from feature importance.** Any permutation-importance/SHAP output
  on the ML systematics model describes predictive association with the injected/real
  templates, not physical causation.
- **Spatial, not i.i.d., data.** Uncertainties use spatial jackknife / mock-ensemble
  variance, never galaxy-level independent-error assumptions.

## Angular-injection-study-specific limitations (added once that study was designed)

- **Not a 3D mock.** The injection study operates on angular (HEALPix) density
  fields, not 3D comoving space — see `decision_log.md` 2026-09-08 entry for why this
  matches the actual research question and the cited literature's own methodology
  (Rezaie et al., Ross et al.). Its "true clustering signal" is an angular power
  spectrum, unrelated to the 3D ξ(r) baseline measurement elsewhere in this repo. Do
  not compare numbers between the two tracks.
- **Input angular power spectrum is a documented stand-in, not a Boltzmann-code
  prediction.** `power_law_cl()` produces a simple power-law C_ell with no claim to
  match the real LRG angular power spectrum's actual shape/amplitude. This is
  appropriate for testing *relative* correction-method behavior (contamination
  removal vs. signal distortion) but means absolute distortion magnitudes are
  study-specific, not survey predictions.
- **Pseudo-Cl, not full mode-coupling deconvolution.** `masked_pseudo_cl` applies only
  an fsky correction, not a MASTER-style mixing-matrix deconvolution. Adequate for
  comparing correction methods against each other and against the known truth within
  this study; NOT adequate for an absolute angular power spectrum measurement that
  would be compared to an external survey result.
- **Injected contamination model is multiplicative-linear-in-templates by
  construction.** The study can therefore only speak to how well each correction
  method recovers a linear-in-templates contamination — it does not test recovery of
  a genuinely nonlinear contamination process, which is exactly the regime where ML
  methods would be expected to have their real advantage over linear regression. This
  structurally favors the linear correction method (it is exactly correctly specified
  for this contamination process). Notably, the STANDARD-tier canonical result (see
  `decision_log.md` 2026-09-09 entry) shows ML correction (gradient boosting, deep
  random forest) matching or modestly beating linear regression on contamination
  removal even under this favorable-to-linear design, with no corresponding
  signal-preservation penalty — if anything mild evidence *for* ML correction on this
  design, not against it. Whether ML's advantage would be larger on genuinely
  nonlinear real-world contamination remains untested and is the natural extension if
  revisited.
- **SMOKE-tier and STANDARD-tier injection-study results told meaningfully different
  stories** (see `decision_log.md` 2026-09-08 vs. 2026-09-09 entries). At SMOKE scale
  (1354 pixels, 2 realizations), deep RF/GBM showed dramatically higher
  negative-control distortion than linear, suggesting flexibility-driven
  overcorrection. At STANDARD scale (17,978 pixels, 10 realizations), negative-control
  distortion was approximately uniform across all methods — the SMOKE-scale gap did
  not hold up and was substantially a small-sample artifact. Only the STANDARD-tier
  numbers should be cited as the study's finding; SMOKE-tier numbers exist for pipeline
  validation only, per their own design intent, and this is a concrete demonstrated
  example of why that distinction matters.
- **Negative-control noise mismatch (found in the Reviewer-2 pass, `reviewer2.md`).**
  The negative control fits correction models on the noiseless true density contrast
  `field.delta_true`, while the contaminated scenario fits on Poisson-shot-noise-laden
  observed counts. These are not noise-matched. The reported negative-control
  distortion numbers should be read as testing each correction method's *intrinsic*
  bias in isolation from shot-noise-driven overfitting — likely a LOWER BOUND on the
  distortion a correction method would introduce on real, noisy, uncontaminated data.
  A noise-matched negative control (zero-coefficient contamination, still
  Poisson-sampled) is the natural fix, not yet run.
- **Injection-study uncertainties are Monte Carlo std across mock realizations on
  scalar summary metrics** (RMS Cℓ distortion, contamination correlation), not a
  jackknife and not a full ℓ-bin-to-ℓ-bin covariance matrix. This is a valid MC
  uncertainty (each realization is an independent random draw) but should not be
  confused with the baseline clustering measurement's jackknife covariance, which is
  a different uncertainty method applied to a different (3D, real-data) measurement.

## Open items (fill in as work proceeds)

- [x] Final survey/release/tracer confirmed against verified public download: DESI
      DR1 LRG SGC, verified live (see `data_provenance.md`, `decision_log.md`).
- [x] Fiducial cosmology used for comoving coordinates documented and matched to the
      survey's own LSS-catalog fiducial cosmology (not assumed): resolved from the
      actual Ross et al. 2025 paper text, see `cosmology_assumptions.md`.
- [x] Baseline ξ(r) pair-counting engine validated against a brute-force reference
      implementation (`tests/test_pair_counts.py`); estimator validated on synthetic
      Poisson (null) and known-clustering mocks (`tests/test_landy_szalay.py`) before
      being trusted on real data.
- [x] STANDARD-tier injection-study run completed 2026-09-09: 10 realizations
      (reduced from an original target of 20 — shared-machine compute constraints,
      documented in `decision_log.md`) at nside=128 (17,978 footprint pixels), 5-fold
      spatial CV per realization, 24.5 min total runtime — see
      `results/injection_study/manifest.json`.
- [x] Spatial cross-validation wired into the correction-model fitting itself
      (`cosmoscope.systematics.spatial_cv` + `out_of_fold_predict` in
      `scripts/run_injection_study.py`) — an initial version evaluated correction
      models in-sample, which let the flexible ML methods appear to overfit training
      pixels rather than genuinely overcorrect; fixed before the STANDARD-tier run
      was trusted. See `decision_log.md` 2026-09-08 entries for the before/after
      numbers that motivated the fix.
- [x] External validation done (2026-09-08): compared our power-law fit (r0 = 9.14 ±
      0.04 Mpc/h, γ = 1.545 ± 0.009, fit over 1-30 Mpc) against real published LRG
      clustering. Zehavi et al. 2005 (ApJ 621, 22; arXiv:astro-ph/0411557, SDSS LRGs
      z=0.16-0.44) find r0 = 9.8-11.2 h⁻¹Mpc (luminosity-dependent) — our r0 is close
      but slightly low, consistent with DESI LRGs being on average less luminous/
      biased than SDSS-I LRGs. Zhai et al. 2017 (arXiv:1607.05383, eBOSS+BOSS LRGs
      z~0.7, the closest redshift match to our 0.40-1.10 sample) report bias b = 2.30
      ± 0.03, consistent order-of-magnitude with our ξ(r) amplitude via ξ_gg ~
      b²·ξ_matter. Our fitted γ = 1.545 is SHALLOWER than the literature's γ≈1.8-2.0 —
      a real, honest discrepancy, most likely because our fit range (1-30 Mpc) spans
      the known one-halo/two-halo transition where Zehavi et al. document a power-law
      break/dip near ~2 h⁻¹Mpc, which a single power law fit across that transition
      will average out and shallow. A future revision fitting only the two-halo
      regime (r ≳ 3-4 Mpc/h) would likely recover a steeper γ closer to the
      literature. Re-fit restricted to r > 3 Mpc (excluding the one-halo regime) to
      test this: γ steepens from 1.545 ± 0.009 to 1.673 ± 0.011 — moving in the
      predicted direction but not fully closing the gap to γ≈1.8-2.0. This is reported
      as a genuine, still-open residual discrepancy, not resolved by the re-fit; both
      fits are stored in `results/baseline_lrg_sgc/power_law_fit.json`. No order-of-
      magnitude problem in shape or amplitude. Full citations in `decision_log.md`
      2026-09-08 entry.
- [x] Feature ablations (Section 43) and map-resolution sensitivity (Section 44) —
      done at STANDARD scale, nside=128, 10 realizations (`scripts/run_robustness_checks.py`,
      `results/robustness/`, `decision_log.md` 2026-09-09 entry). Found a genuine
      numerical-stability tail-risk finding (minimal-feature linear correction under
      spatial CV) alongside the expected ablation/resolution results — see paper
      Section 10.
- [x] Out-of-distribution sky-region robustness (Section 23) — done, STANDARD scale
      (`scripts/run_ood_robustness.py`, `results/ood_robustness/`, `decision_log.md`
      2026-09-09 entry). Found the clearest genuine ML-vs-linear caveat in the whole
      project: gradient boosting's error inflates ~7.9x moving from in-distribution
      holdout to an out-of-distribution sky region (stellar-density extremes), vs.
      only ~1.9x for linear regression — consistent with tree ensembles' inability to
      extrapolate beyond their training range. This nuances (does not contradict) the
      main injection study's headline finding: ML matches/beats linear regression on
      average-case contamination removal, but degrades far more sharply than linear
      regression specifically in unusual sky regions.
- [x] `claims_registry.md` covers the 3D baseline, footprint, and injection-study
      numbers (`scripts/populate_claims_registry.py`, 18 rows as of 2026-09-09).
      Robustness-check numbers (`results/robustness/`) are not yet rows in the
      registry — added directly to the paper/site text with explicit result-file
      citations in the interim; formal registry rows are a remaining nicety.

*(Reviewer-2 pass in section 57 of the brief will add an adversarial pass on top of
this list before result freeze.)*