# Data Provenance

All facts below were verified by live directory listing or official documentation
fetch (see `decision_log.md` 2026-09-07 entry for the research agent that did this).
Nothing here is a guessed URL or invented column name.

## Chosen dataset

**Primary: DESI DR1 LSS catalogs (Iron reduction, v1.5), LRG tracer, SGC region.**

Rationale: DESI DR1 is the current-generation public release, ships a dedicated
"clustering-ready" catalog product (weights pre-combined), provides official random
catalogs, and — critically for the primary research question (ML systematics
correction) — ships companion **HEALPix imaging-systematics maps** (`hpmaps/`)
covering depth, seeing, extinction, and stellar density. SDSS BOSS DR12 (the
alternative) does not ship an equivalent raw-map product; its imaging systematics are
already baked into fixed weight columns, which would foreclose the injection-study
design (we need real systematics *templates* to inject controlled contamination, not
just pre-applied weights). SGC (South Galactic Cap) is used as the "one small sky
region" per the start-small mandate; NGC is larger (~143 MB vs ~64 MB) and deferred.

LRG chosen over QSO/ELG/BGS as first tracer: higher bias / stronger clustering signal
at fixed number density, single well-studied redshift range, most direct match to the
linear/ML systematics-correction literature (Ross, Rezaie, Rosado-Marin papers above
mostly target LRG/ELG-like samples).

## Verified source locations

| Product | URL | Verified size |
|---|---|---|
| LRG SGC clustering catalog (data) | `https://data.desi.lbl.gov/public/dr1/survey/catalogs/dr1/LSS/iron/LSScats/v1.5/LRG_SGC_clustering.dat.fits` | 64,272,960 bytes (~64 MB) |
| LRG SGC randoms (18 realizations) | `.../v1.5/LRG_SGC_{0..17}_clustering.ran.fits` | several × 10s of MB each |
| Imaging systematics HEALPix maps | `.../v1.5/hpmaps/LRG_mapprops_healpix_nested_nside256_S.fits` | ~104 MB |
| Directory index (LSS catalogs) | `https://data.desi.lbl.gov/public/dr1/survey/catalogs/dr1/LSS/iron/LSScats/v1.5/` | — |
| Column/data-model reference | `https://desidatamodel.readthedocs.io/en/latest/DESI_ROOT/vac/RELEASE/lss/VERSION/LSScats/clustering/clustering_dat.html` | — |
| Release documentation | `https://data.desi.lbl.gov/doc/releases/dr1/` | — |
| Acknowledgment requirements | `https://data.desi.lbl.gov/doc/acknowledgments/` | — |

Fallback/reference (not primary): SDSS BOSS DR12 LSS at `https://data.sdss.org/sas/dr12/boss/lss/`
(`galaxy_DR12v5_CMASS_South.fits.gz`, 51.6 MB; `galaxy_DR12v5_LOWZ_South.fits.gz`, 32.3 MB) —
kept as a documented alternative in case DESI access is interrupted; not downloaded unless needed.

## Object type / coordinates / redshift / selection

- Object type: Luminous Red Galaxy (LRG) spectroscopic targets, DESI DR1 (Iron
  reduction), SGC footprint.
- Columns confirmed present in the clustering-ready file (from the DESI data model):
  `RA`, `DEC`, `Z`, `WEIGHT` (combined systematic+completeness weight),
  `WEIGHT_FKP` (= 1 / (1 + NZ · P0)), `WEIGHT_ZFAIL`, `BITWEIGHTS` (PIP bitweights for
  fiber-assignment incompleteness), `NZ` (expected number density at that redshift),
  `TARGETID`, `NTILE`.
- Randoms carry the matching `NZ`, `WEIGHT_FKP`-equivalent columns and sky positions
  but no redshift-independent clustering signal — used to sample the survey's angular
  and radial selection function.

## Weights / quality flags

Per verified DESI data model page (see table above): `WEIGHT` is the single combined
weight to use for clustering pair counts (it already folds in `WEIGHT_SYS`/completeness
terms present in the full, non-clustering-ready catalog — those upstream columns are
not expected to be separately present in the clustering file). `WEIGHT_FKP` is applied
separately for radial-mode optimal weighting (Feldman-Kaiser-Peacock).
`BITWEIGHTS`/PIP handles fiber-assignment incompleteness statistically.

SDSS DR12 fallback columns (for reference, not currently downloaded):
`WEIGHT_FKP`, `WEIGHT_CP` (fiber collision), `WEIGHT_NOZ` (redshift failure),
`WEIGHT_STAR`, `WEIGHT_SEEING`, `WEIGHT_SYSTOT`, `IMATCH` quality flag
(0=missed, 1=BOSS z, 2=legacy z, 3=fiber-collided, 4=star, 5=z-failure).

## Dedicated LSS clustering data product

Confirmed dedicated product, not a repurposed general catalog:
DESI "Large-scale Structure (LSS) Catalogs", described in DESI 2024 II / Ross et al.
2025 (arXiv:2411.12020) and the DR1 release paper, DESI Collaboration et al. 2026
(arXiv:2503.14745).

## Imaging systematics maps

Confirmed present: `hpmaps/LRG_mapprops_healpix_nested_nside256_{N,S}.fits`, HEALPix
nside=256, nested ordering. Documented contents: E(B-V) (Galactic dust extinction),
Gaia-based stellar density, PSF size (`psfsize_g/r/z`), galaxy depth
(`galdepth_g/r/z`), HI column density. These are the real systematics templates used
both for (a) the negative-control/injection study (Section 17 of brief — inject
*documented-style* contamination using structure similar to these real templates on a
mock) and (b) a real-data application of the correction methods.

## License / citation

- DESI DR1 data: **CC BY 4.0**.
- Required acknowledgment text (verbatim from DESI's acknowledgments page): "This
  research used data obtained with the Dark Energy Spectroscopic Instrument (DESI)."
- Citation: DESI Collaboration et al. 2026, "Data Release 1 of the Dark Energy
  Spectroscopic Instrument," arXiv:2503.14745.
- Full acknowledgment requirements: `https://data.desi.lbl.gov/doc/acknowledgments/`
  (must be included verbatim in README, paper, and site footer — see `docs/HANDOFF.md`).

## Official randoms

Confirmed real, distributed files — not fabricated: `LRG_SGC_{0..17}_clustering.ran.fits`,
18 independent random realizations per tracer/region. Multiple realizations should be
combined (concatenated with appropriate downweighting) for adequate RR-pair precision;
document how many are used in each canonical run's manifest (Section 28 of brief).

## Fiducial cosmology — OPEN ITEM, do not assume

DESI's LSS catalogs are constructed and their `NZ`/comoving-distance quantities
implicitly reference DESI's fiducial flat-ΛCDM cosmology. **The exact parameter values
(Ω_m, h, etc.) must be read from the actual DESI LSS/clustering documentation or FITS
header at ingestion time — not from memory or assumption.** This is tracked as an open
item in `cosmology_assumptions.md` and must be resolved (and cross-checked against
Ross et al. 2025 / arXiv:2411.12020) before any comoving-coordinate conversion is
trusted. Do not proceed to Section 7 (coordinate conversion) with an unverified
cosmology value.