# Decision Log

Chronological record of substantive decisions. Each entry: date, decision, rationale, alternatives considered.

## 2026-09-07 — Repository audit

- Repo `~/Projects/cosmoscope` did not exist. Created fresh: `git init`, standard layout
  (`research/`, `docs/`, `data/{raw,interim,processed}`, `src/cosmoscope/`, `tests/`,
  `figures/`, `results/`, `configs/`, `notebooks/`, `web/`, `paper/`, `scripts/`).
- No prior code, no prior claims to reconcile. Starting from zero.

## 2026-09-07 — Python environment

- System default `python3` resolved to Homebrew Python 3.14 (very new — Sept 2026).
  Scientific-stack packages with compiled extensions (`healpy`, `fitsio`, potential
  `Corrfunc`) most reliably ship wheels for 3.11/3.12 first. Homebrew `python@3.11`
  (3.11.15) was already installed on this machine.
- **Decision:** build the project venv on Python 3.11, not the system 3.14, to avoid
  chasing missing wheels / build-from-source failures mid-pipeline.
- Verified core stack imports cleanly on 3.11: numpy, scipy, astropy 8.0.1, pandas,
  pyarrow, scikit-learn, matplotlib, healpy 1.20.0, pyyaml, click, tqdm.

## 2026-09-07 — Primary research question selection (provisional, pending data recon)

- Spec offers 4 candidate questions (A: selection-sensitivity of clustering; B: can ML
  systematics correction remove contamination without erasing true structure; C:
  environment-estimator stability; D: approximate/fast clustering methods).
- The spec's own worked example title — "CosmoScope: Testing Machine-Learned
  Survey-Systematics Corrections Without Erasing Large-Scale Structure" — and the
  density of downstream instructions (sections 14–23, 39–44 all elaborate the
  injection/correction/signal-preservation machinery) point at **Question B** as the
  intended spine of the project.
- **Decision:** adopt B as the primary question: *Can ML-based correction methods for
  observational/imaging systematics remove contamination from galaxy angular density
  without erasing genuine large-scale-structure clustering signal, relative to standard
  linear/weight-based corrections?*
- This is a **methodology** study (signal-preservation vs. contamination-removal
  trade-off on mocks with known injected systematics + a real survey baseline), not a
  cosmological-parameter or discovery claim. See `questions.md` for full framing and
  `limitations.md` for what this design explicitly cannot support.
- Two research agents dispatched in parallel to (1) verify which real public LSS
  catalog + companion randoms + imaging systematics maps are actually downloadable
  right now, and (2) ground the ML-systematics-correction literature honestly (real
  citations only, e.g. Ross et al. linear weighting, Rezaie et al. neural-net eBOSS QSO
  systematics, Landy & Szalay 1993 estimator). Findings will finalize
  `data_provenance.md`, `cosmology_assumptions.md`, and `literature_matrix.csv` — do
  not treat those files as final until this entry is updated.

## 2026-09-07 — Data source finalized: DESI DR1 LRG, SGC region

Research agent verified (via live directory listings + official docs, not guesses):

- **DESI DR1 LSS catalogs (Iron reduction, v1.5)** at
  `https://data.desi.lbl.gov/public/dr1/survey/catalogs/dr1/LSS/iron/LSScats/v1.5/` —
  real, public, CC BY 4.0, no application required.
- Clustering-ready LRG SGC data file: 64,272,960 bytes; 18 matching random
  realizations; companion imaging-systematics HEALPix maps at nside=256 in `hpmaps/`
  (E(B-V), stellar density, PSF size, galaxy depth per band, HI column density).
- Alternative considered: SDSS BOSS DR12 LSS catalogs (smaller files, e.g. LOWZ South
  32.3 MB) — rejected as primary because DR12 has no bundled raw imaging-systematics
  map product (systematics are pre-baked into fixed weight columns), which would
  foreclose the controlled-injection design central to research question B. Kept as a
  documented fallback in `data_provenance.md`.
- **Decision:** primary tracer = LRG, region = SGC (South Galactic Cap, the smaller of
  the two DESI hemispheres — satisfies the "start small" mandate). Full rationale and
  verified URLs/columns/citation requirements recorded in `data_provenance.md`.
- Fiducial cosmology is explicitly left OPEN (see `cosmology_assumptions.md`) — will
  be resolved from the actual FITS header / DESI LSS catalog paper (Ross et al. 2025,
  arXiv:2411.12020) at ingestion time, not assumed from memory.

## 2026-09-07 — Literature grounding compiled

Research agent verified real citations (arXiv IDs cross-checked, two flagged as
needing a full-text confirmation pass) for: standard linear/weight-based systematics
correction (Ross et al. 2020, arXiv:2007.09000; Elvin-Poole et al. 2018,
arXiv:1708.01536; Kalus et al. 2018, arXiv:1806.02789), ML-based correction (Rezaie et
al. 2020, arXiv:1907.11355 — the core "SYSnet" neural-net reference; Rezaie et al.
2021, arXiv:2105.10463; Rosado-Marín et al. 2024, arXiv:2411.12024, most directly
DESI-DR1-relevant), overcorrection/signal-erasure precedent (Rezaie et al.
arXiv:2106.13725 — directly grounds this project's negative-control design), the
Landy & Szalay 1993 estimator (no arXiv ID — pre-dates arXiv, cite via ADS/journal),
and mock/covariance methodology (Kitaura et al. 2016 PATCHY, arXiv:1509.06400; Chuang
et al. 2015 EZmocks, arXiv:1409.1124; Favole et al. 2016 jackknife-vs-mock comparison,
arXiv:1606.00233; Mohammad et al. 2020 jackknife-scale sensitivity, arXiv:2004.13436).
Full table in `literature_matrix.csv`, including two rows flagged `unconfirmed_detail`
that need a full-text check before being cited as ML-specific methods.

## 2026-09-07 — Data downloaded; fiducial cosmology resolved

- Downloaded `LRG_SGC_clustering.dat.fits` (verified 64,272,960 bytes, matches
  research-agent report exactly). Opened with astropy: 662,492 rows, 13 columns
  (`TARGETID, Z, NTILE, RA, DEC, PHOTSYS, FRAC_TLOBS_TILES, WEIGHT_ZFAIL, WEIGHT_SYS,
  WEIGHT, WEIGHT_COMP, NX, WEIGHT_FKP`) — one extra column (`FRAC_TLOBS_TILES`) beyond
  what the research agent's data-model page listed; `PHOTSYS` (photometric system
  flag) also present. No NaNs in RA/Z, no duplicate TARGETID, RA/DEC/Z ranges sane
  (RA 0-360°, DEC -19.4 to 33.5°, Z 0.40-1.10 — consistent with LRG target selection).
- Downloading 3 of 18 official random realizations + the LRG imaging-systematics
  HEALPix map (`hpmaps/LRG_mapprops_healpix_nested_nside256_S.fits`) in background
  (connection is slow, ~40 KB/s from this network — sequential fetches, several
  minutes each).
- FITS header carries no cosmology keywords (as expected — DESI does not embed
  fiducial cosmology in per-object LSS catalog headers). Resolved fiducial cosmology
  by fetching the actual paper text (arXiv:2411.12020 via ar5iv HTML render, not
  memory): ωb=0.02237, ωcdm=0.12, h=0.6736, ns=0.9649, Neff=3.044, Σmν=0.06 eV →
  derived Ωm = 0.3137721. Recorded with full citation and the documented
  massive-neutrino approximation in `research/cosmology_assumptions.md` and made
  machine-readable in `configs/fiducial_cosmology.yaml`.

## 2026-09-07/08 — Compute-tier design for the real clustering baseline

- Benchmarked RR pair counting (the bottleneck term) directly on real downloaded data:
  100k randoms 0.87s, 300k 5.9s, 800k 27.2s, 2M 124.5s (cKDTree count_neighbors,
  laptop CPU). A full single realization (~5M randoms) auto-pair run was started and
  did not finish within 10+ minutes; killed rather than block the session — not
  because it would fail, but because the interactive budget didn't justify waiting.
- **Decision:** documented, asymmetric compute-tier design for the canonical baseline
  run (`scripts/run_clustering_baseline.py`): the *mean* xi(r) uses a larger random
  subsample (2,000,000, ~3x oversampling of the 662,492 data objects) for lower RR
  shot noise; the *jackknife covariance* uses a smaller subsample (300,000) with
  nside_jk=4 (27 usable regions with >=5000 objects each), since jackknife only needs
  relative variation across leave-one-region-out samples, not the lowest possible
  absolute RR shot noise, and recomputing RR at 2M scale once per region would take
  30-40+ minutes. This is exactly the runtime/accuracy tradeoff the brief's Section 26
  asks to be benchmarked, not a silently-chosen shortcut.
- Result of the canonical run (git-tracked in `results/baseline_lrg_sgc/`): mean xi(r)
  computed in 195.2s, jackknife in 918.0s (27 regions). Recovered xi(r) shows a
  physically sensible declining profile (~14 at 1.2 Mpc down to ~0.003 at 127 Mpc),
  qualitatively consistent with known LRG clustering amplitudes/shapes in the cited
  literature (order-of-magnitude cross-check only; a quantitative comparison to a
  published DESI/eBOSS LRG measurement is still an open item, see `limitations.md`).

## 2026-09-08 — Injection-study design: angular (HEALPix), not 3D, and why

- Re-scoped the mock-injection study (brief Sections 16-23, 39-44) from an originally
  assumed 3D comoving-space design to an **angular (HEALPix pixel) design**, matching
  how the actual cited literature does this (Rezaie et al. 2020/2021, Ross et al.
  2020): imaging systematics are fundamentally angular effects (depth, seeing,
  extinction, stellar density vary across the sky, not with radial distance), and the
  real DESI systematics-map product we downloaded (`hpmaps/...nside256...fits`) is
  itself a HEALPix angular map. A 3D N-body-level mock would have added enormous
  engineering cost for no corresponding gain in answering the actual research
  question. This is recorded here because it changes what "true clustering signal"
  means for this study: angular power spectrum C_ell / pseudo-Cl of the galaxy
  density-contrast field, not the 3D xi(r) computed for the real-data baseline
  (Section 9-13 work). Both live in this repo; they answer different (complementary)
  questions and should not be conflated. See `limitations.md`.
- Mock field: lognormal Gaussian random field via `healpy.synfast` + exponentiation
  (guarantees non-negative density without ad hoc clipping), power-law input C_ell
  (NOT derived from a Boltzmann code — explicitly a stand-in, documented in
  `src/cosmoscope/mocks/angular_field.py` docstring), Poisson-sampled per pixel.
  Footprint mask = the REAL LRG SGC data footprint (not a toy shape).
  Contamination templates = the REAL DESI hpmaps columns (EBV, STARDENS, PSFDEPTH_*,
  GALDEPTH_*, PSFSIZE_*), degraded from native nside=256 to the analysis nside via
  `healpy.ud_grade`. Injected contamination uses 3 real templates (EBV, STARDENS,
  PSFDEPTH_G) with known coefficients [0.35, 0.25, 0.05] — including one deliberately
  weak/near-zero coefficient so the study can also show correction methods correctly
  leaving near-zero-effect templates alone, not just removing strong ones.
- Correction methods compared, all fit against the SAME 11-template design matrix
  (documented subset of the 33 real hpmaps columns): no-correction (explicit identity
  baseline), linear regression (the Ross-et-al.-style survey-standard method),
  random-forest at two depths (shallow max_depth=3, deep max_depth=12 — the explicit
  model-complexity/bias-variance comparison from Section 42), and gradient boosting.
- Metrics: masked pseudo-Cl (fsky-corrected only, NOT a full MASTER mode-coupling
  deconvolution — documented approximation, adequate for *relative* method comparison,
  not absolute precision cosmology; see `limitations.md`), RMS fractional Cl
  distortion vs. the KNOWN true field's Cl (signal-preservation metric), and Pearson
  correlation between the corrected density map and the injected template
  (contamination-removal metric).
- **Negative control (Section 40) is built into every realization, not a separate
  pass**: every correction method is also fit and applied to the never-contaminated
  true field itself. `no_correction` on the negative control gives RMS distortion
  ~1e-16 (machine epsilon) by construction, serving as the pipeline's own correctness
  check on the distortion metric.
- SMOKE-tier validation run (nside=32, 2 realizations, `results/injection_study_smoke/`)
  produced an interpretable, literature-consistent story before committing to the full
  STANDARD run: linear correction cut contamination-template correlation from ~0.81 to
  ~0.02 with modest negative-control distortion (~0.12 RMS); deep random forest and
  gradient boosting showed substantially higher negative-control distortion (~0.66,
  ~0.41) than linear or shallow RF (~0.14) — i.e., visible overcorrection/signal-
  erasure scaling with model flexibility, which is the actual research question. This
  is a SMOKE-scale, noisy (1354-pixel) result, not yet the canonical finding — the
  STANDARD run (nside=128, ~18k pixels, 20 realizations,
  `results/injection_study/manifest.json`) is the one to cite.

## 2026-09-08 — External validation of the baseline ξ(r) against real literature

- Fit our measured ξ(r) to a power law (r0/r)^gamma over 1-30 Mpc: r0 = 9.14 ± 0.04
  Mpc/h, gamma = 1.545 ± 0.009 (`results/baseline_lrg_sgc/power_law_fit.json`).
- Dispatched a research agent to find real published LRG clustering fits for
  comparison (brief Section 58). Verified findings: Zehavi et al. 2005 (ApJ 621, 22;
  arXiv:astro-ph/0411557, SDSS LRGs z=0.16-0.44) report r0 = 9.8-11.2 h⁻¹Mpc
  (luminosity-dependent) and document a well-known power-law break/dip near ~2
  h⁻¹Mpc (one-halo/two-halo transition) rather than a single clean power law; Zhai et
  al. 2017 (arXiv:1607.05383, eBOSS+BOSS LRGs z~0.7, the closest redshift match to our
  0.40-1.10 sample) report bias b = 2.30 ± 0.03. Our r0 is close to Zehavi's range;
  our gamma (1.545) was shallower than the literature's typical ~1.8-2.0.
- Tested the one-halo-transition explanation directly: re-fit restricted to r > 3 Mpc
  (excluding the regime Zehavi et al. document as non-power-law) gives gamma = 1.673 ±
  0.011 — steeper, moving in the predicted direction, but not fully closing the gap.
  Recorded as a genuine open discrepancy in `research/limitations.md`, not silently
  resolved. Both fit ranges are stored in the same JSON file for transparency.
- Overall conclusion: no order-of-magnitude problem in shape or amplitude of our
  baseline measurement; this constitutes the Section 58 external validation pass.

## 2026-09-09 — STANDARD-tier injection study: canonical result

- After being lost twice (see `docs/HANDOFF.md`), the canonical run completed:
  nside=128 (17,978 occupied pixels, 9.14% of sky), 10 realizations (seeds
  90210-90219; reduced from the Makefile's default 20 because this run shared a
  12-core machine with unrelated heavy jobs at load average 25-52 — documented
  deviation, not silent), 24.5 minutes total runtime
  (`results/injection_study/manifest.json`). Makefile's `systematics` target updated
  to `--n-realizations 10` to match what is actually canonical/reproducible from a
  clean run of this repo.
- **Headline numbers** (mean RMS fractional Cℓ distortion vs. known truth; mean
  residual correlation with the injected template; see
  `results/injection_study/injection_study_summary.json`):

  | scenario | method | distortion | residual corr. |
  |---|---|---|---|
  | contaminated | no_correction | 1.913 | 0.600 |
  | contaminated | linear | 0.0997 | -0.003 |
  | contaminated | random_forest_shallow | 0.159 | 0.085 |
  | contaminated | random_forest_deep | 0.0902 | 0.006 |
  | contaminated | gradient_boosting | **0.0785** | 0.027 |
  | negative_control | no_correction | ~0 (machine eps.) | 0.010 |
  | negative_control | linear | 0.0507 | 0.001 |
  | negative_control | random_forest_shallow | 0.0492 | 0.001 |
  | negative_control | random_forest_deep | 0.0555 | -0.004 |
  | negative_control | gradient_boosting | 0.0565 | 0.0003 |

- **This changes the story told by the SMOKE-tier run.** At SMOKE scale
  (nside=32, 1354 pixels, 2 realizations) deep RF/GBM showed dramatically higher
  negative-control distortion (~0.66, ~0.41) than linear (~0.12) — read at the time as
  "visible overcorrection scaling with model flexibility." At STANDARD scale, with
  more realizations and ~13x more pixels, **negative-control distortion is
  approximately uniform across every correction method** (0.049-0.057, including
  linear) — the earlier flexibility-driven overcorrection gap does not hold up. The
  correct read: the SMOKE run's dramatic gap was substantially a small-sample/noise
  artifact, not a robust method-driven effect. This is exactly why SMOKE-tier results
  are documented as illustrative-only and the STANDARD run is the one to cite — see
  `research/limitations.md`, updated accordingly.
- **What the STANDARD result actually supports:** contamination REMOVAL differs
  more than signal preservation does. All four real correction methods reduce
  contamination-template correlation from ~0.60 to near zero, but gradient boosting
  achieves the lowest post-correction distortion (0.0785) and linear the lowest
  residual correlation (-0.003, i.e. slightly overcorrected in sign but negligible in
  magnitude); shallow RF underperforms both (0.159 distortion, 0.085 residual
  correlation — too inflexible to fully capture even a linear contamination signal
  through 5-fold spatial CV). On THIS injection design (linear-in-templates
  contamination, by construction favoring linear regression — see `limitations.md`),
  ML correction (particularly gradient boosting and deep RF) matches or modestly
  beats linear regression on contamination removal without a corresponding signal-
  preservation penalty. This is a genuine, if narrow and design-conditional, positive
  finding for ML correction — not the "ML overcorrects" story the SMOKE run initially
  suggested, and not a null result either.
- Caveats that still apply and limit how far this can be pushed (already documented,
  reaffirmed here): negative control is not noise-matched to the contaminated scenario
  (`reviewer2.md`); contamination is linear-in-templates by construction, so this
  design cannot show whether ML's relative advantage would be larger on genuinely
  nonlinear real-world contamination; hyperparameters were not tuned via search.

## 2026-09-09 — STANDARD-tier robustness checks (feature ablation + resolution sensitivity)

- Ran `scripts/run_robustness_checks.py` at nside=128 (matching the main study),
  coarse-template-nside=16, 10 realizations — 18.7s total (linear-only, far cheaper
  than the RF/GBM injection study). Results: `results/robustness/`.
- **Resolution sensitivity: clean, consistent, no outliers.** Coarsening the
  systematics templates from native nside=128 to nside=16 (then upgrading back)
  degrades correction quality by roughly 8x in every realization (median RMS
  distortion 0.029 native vs. 0.290 coarsened; every one of 10 realizations shows the
  same direction and similar magnitude of degradation). Template map resolution
  matters a lot for correction quality — an unambiguous finding.
- **Feature ablation: real signal, but the raw mean is outlier-dominated —
  medians tell the honest story.** Dropping either of the two strongly-injected
  templates (EBV, coefficient 0.35; STARDENS, coefficient 0.25) consistently degrades
  correction in every realization (median RMS distortion 0.84 and 0.42 respectively,
  vs. 0.036 for the full 11-template baseline — not just mean-driven, the whole
  distribution shifts). Dropping any of the 8 non-injected templates leaves the
  MEDIAN essentially unchanged (~0.035, matching baseline) but occasionally produces
  a catastrophic MAX in a small number of realizations (e.g. dropping PSFSIZE_G: 8 of
  10 realizations are fine, but one realization spikes to 9.52) — consistent with
  occasional near-collinearity among the 11 real templates (e.g. PSFDEPTH/GALDEPTH or
  PSFSIZE bands correlate with each other) destabilizing a specific spatial fold's
  linear fit once a redundant column is removed, not contamination-relevant signal
  being lost.
- **`injected_only` (design matrix reduced to exactly the 3 true contaminating
  templates) is the clearest example of this tail risk**: median distortion across
  10 realizations is actually the BEST of any configuration tested (~0.02-0.06 in 8 of
  10 realizations), but 2 of 10 realizations blow up catastrophically (3.7 and
  1165.5) — driving the naive mean to 117, which would be a deeply misleading summary
  statistic reported alone. Diagnosed as: with only 3 predictors, a spatial fold whose
  held-out pixels have template values outside the range seen in the other 4 training
  folds lets the fitted linear model's out-of-fold prediction approach -1, and the
  correction formula `(1+delta_obs)/(1+pred) - 1` is singular there. This is a real,
  reproducible numerical-stability finding about minimal-feature-set linear
  correction under spatial CV, not a bug — worth reporting as its own robustness
  result rather than smoothing over with a mean.
- **Decision:** report median (not mean) as the primary summary statistic for the
  feature-ablation table in `paper/paper.md`, with the outlier-tail behavior itself
  reported as a finding, per the brief's own scientific-integrity standard (don't
  silently choose the statistic that tells the cleaner story).

## 2026-09-09 — Out-of-distribution sky-region robustness test (brief Section 23)

- Implemented `scripts/run_ood_robustness.py`: define the OOD region as footprint
  pixels where standardized STARDENS (Gaia stellar density — a genuine "unusual
  observational condition," and one of this study's own injected templates) exceeds
  |z| > 1.5; train correction models on everything else; compare RMS error between
  predicted and TRUE injected contamination fraction (known exactly, since this is a
  controlled mock — `ContaminatedField.contamination_frac`) on (a) a held-out 20% of
  in-distribution pixels vs. (b) the entire OOD region, never seen in training.
- STANDARD-tier run: nside=128, 17,978 footprint pixels, OOD region = 1,370 pixels
  (7.6% of footprint), 8 realizations, linear + gradient boosting, 1m41s total
  (`results/ood_robustness/`).
- **Result:** gradient boosting's error inflates ~7.9x moving from in-distribution
  holdout (0.076) to the OOD region (0.591); linear regression's error inflates only
  ~1.9x (0.034 → 0.059). A smaller SMOKE-scale check (nside=64, 2 realizations) showed
  the same direction and comparable magnitude (5.6x vs. 1.8x) before the STANDARD run
  was launched — unlike the main injection study, this finding was consistent from
  SMOKE to STANDARD scale, so no SMOKE-vs-STANDARD discrepancy here.
- **Interpretation:** this is the physically expected result — tree-ensemble models
  (random forest, gradient boosting) predict via piecewise-constant leaf averages and
  cannot extrapolate beyond the range of template values seen during training; linear
  regression extrapolates smoothly by construction. This is the clearest evidence in
  the whole project for a genuine, robust limitation of ML correction relative to
  linear regression: **ML correction degrades much more sharply than linear regression
  specifically in sky regions with unusual (out-of-training-range) observational
  conditions** — exactly the regions where robust systematics correction matters most
  in a real survey. This nuances the STANDARD-tier injection study's headline finding
  (ML matches/beats linear on typical in-distribution pixels) — the OOD result is not
  a contradiction, it is a genuine caveat on a different axis (generalization to
  unusual conditions, not average-case removal/preservation trade-off).

*(To be continued as the pipeline is built.)*