…ated prefetch, 2-phase Gaia search

…, download list

….25-23

- ZoneArray::create(): level >= 8 && type == 2 -> DynamicZoneArray
- defaultStarsConfig.json: stars_8=16.75-18.5, stars_9=18.5-20.25, stars_10=20.25-23
- gaia_to_cat.py: add stars_8 generation, include gaia2 for red star coverage
- add compare_bv_formulas.py, verify_conversion.py, check_overlap.py

- gaia_to_cat.py: output Star2 (32B) with pmra/pmdec/plx from SkyChart data
- DynamicZoneArray: Star3→Star2, draw() uses getJ2000Pos for proper motion
- ZoneArray::create(): type=1 (Star2) with level>=8 → DynamicZoneArray
- defaultStarsConfig.json: type=1 for stars_8/9/10

…dynamic-level-8

…n, drop Star3 type-2 support

…ecksums

…_to_cat.py

…or type 1/2 at all levels

…n, factory uses DZ for type 1/2 at all levels

…dynamic-level-8

…ero extra heap)

…toggle DZ vs SpecialZoneArray

…ntion

…with SpecialZoneArray/HipZoneArray)

…dynamic-level-8

guide/app_star_catalogue.tex gains a 'Generating and Verifying Star
Catalogues' section covering: data download (Gaia DR3 bulk files +
extract_columns, SIMBAD scripts with the Windows astropy int64 patch),
lv0-3 generation with hip2cat (all parameters, henrysky GPLv3
attribution), lv4-10 with skychart2cat, dedupcat chain, and
cmpcat/analyze_only verification.

skychart2cat: default level range is now 4-10; stars_4 covers V in
[10.25, 12.0) - a deliberate 0.25 mag overlap with stars_3 so dedupcat
can drop boundary duplicates without losing stars.

…at hip offset; erfa inf distance fallback

- hip2cat: collect Gaia-only stars only up to the requested level's
  magnitude hi (default lv0-3: V<=10.5), cutting peak memory from ~8 GB
  to <0.5 GB; slim 64-byte GaiaRow, no merge into rows
- hip2cat: trim padded object_types/spectral table entries on load so
  '*' maps to the official index instead of being appended
- hip2cat: drop Gaia-only duplicates of SIMBAD rows patched with a
  source_id after the scan (e.g. HIP 81693A)
- hip2cat: force-collect Ross 248 and move it to rows (fake HIP patch);
  move Gaia 3683687763520080256 to rows so its astrometry patch applies;
  manual patch for HD 183559B (ex HIP 95759B, renamed in SIMBAD, no G)
- hip2cat: warn loudly when lv0-2 are generated without --erfa-python
- cmpcat: hip[3] is at bytes 45-47, not 44-46 (match key for id-less
  stars)
- erfa_zone.py: use np.inf (henrysky's 1/parallax) as the zero-parallax
  distance fallback; a finite cap makes ERFA override the distance and
  kills global-zone crossing detection for plx=0 stars
- field_diff.py: exhaustive star-by-star field comparison vs official
  lv0-3 with per-cause classification

- app_star_catalogue.tex: restructured pipeline description (fork vs util/gaiacat),
  corrected --erfa-python behaviour, added field_diff.py verification section,
  added source_id_provenance column and Windows sitecustomize note.
- app_config_ini.tex: added flag_compact_star_storage and missing_bv_value
  parameters to the [stars] section.
- StarMgr.cpp: changed missing_bv_value default from 0.0 to 1.0.
- hip2cat.cpp: sp table append mode -- new spectral types (M0.0III etc.)
  are appended instead of being dropped to index 0 when reusing the table.

…dynamic-level-8

Python (46 issues):
- field_diff.py: remove alignment extra spaces (E221/E241), split
  one-line if/for bodies, rename ambiguous 'l', type the bare except
- erfa_zone.py: alignment spaces, split imports, drop unused os
- analyze_only.py / extract_columns.py: split imports, drop unused
  imports, split one-line bodies, fix f-strings without placeholders

C++ (5 issues):
- hip2cat.cpp: replace system() with run_command_line() using
  CreateProcess (Windows) / fork+execvp (POSIX) without a shell
  (CWE-78); convert find_cols if-else chain to a lookup table
  (complexity 18 -> 4)
- extract_columns.cpp: split apply_exponent out of fast_atod
  (complexity 26 -> 13)
- cmpcat.cpp: split write_mismatch_line out of record_mismatch
  (complexity 21 -> 15)
- cat_writer.cpp: split write_zone_records out of sort_and_write_bucket
  (complexity 15 -> 12)

- Catalog output names now derive from the Stellarium user
  starsConfig.json (%APPDATA%\\Stellarium\\... or --stars-config <path>):
  stars_<level>_<type>v<major>_<minor>.cat keeps the generator-chosen
  star format (type overrides starsConfig), takes the official major, and
  bumps the official minor by 1 (minor 1 when no official entry).
  The file header (type/major/minor) matches the file name.
- Spectral type table output likewise: stars_hip_sp_0v0_6.cat ->
  stars_hip_sp_0v0_7.cat from the hipSpectralFile entry.
- New shared header catalog_naming.hpp with parsing helpers.
- Rename skychart2cat -> gaia2cat and hip2cat -> gaiahip2cat (source
  file, CMake targets, comments, docs, field_diff/erfa_zone references).

- gaia2cat: --exclude-lower <lv0-3.cat...> reads the gaia_ids already
  present in the lower-level catalogs and drops them from levels <= 6,
  matching the official pipeline (lv0-3 contains all HIP stars, lv4-6
  are gaia-only; levels >= 7 keep them). No intermediate file needed.
- gaia2cat: magnitude selection is now (lo, hi] like the official
  generator and gaiahip2cat (was [lo, hi)).
- gaia2cat: _setmaxstdio(2048) on Windows so 7 levels x 96 LRU bucket
  handles fit under the CRT stream limit.
- dedupcat: support Star1 (type 0, 48-byte records) so lv3 (Star1) can
  be used as the dedup reference against lv4 (Star2).
- field_diff.py: derive catalog file names from starsConfig.json
  (official name for OFFICIAL dir, minor+1 for OURS dir).

- Document the starsConfig.json-driven output naming (type = star format
  chosen by the generator, official major, official minor + 1; spectral
  table hipSpectralFile minor + 1) as a new subsection.
- gaia2cat: document --exclude-lower (lv0-3 catalogues as input, stars
  already catalogued there are dropped from lv4-6, mirroring the official
  hip==0 rule; lv7-10 keep HIP stars like the official lv7+), --star3-from
  11 for all-Star2 output, and the (lo, hi] magnitude ranges.
- gaiahip2cat: document --stars-config, sp table append mode, and the
  minor+1 output names.
- dedupcat: only the lv3/lv4 overlap needs deduplication (ranges no longer
  overlap otherwise).
- field_diff.py: mention the starsConfig-derived file names.

Split long command lines in the commands environment and add \\allowbreak
between unbreakable \\file/\\texttt/\\program tokens so no line exceeds
the text width (all Overfull warnings for app_star_catalogue.tex gone).

…any-locals)

…rfull

- The lv3/lv4 overlap is handled by gaia2cat's --exclude-lower, so the
  standard pipeline has no separate deduplication step; dedupcat stays
  documented as a utility for other cases.
- Rewrite the stars-config and spectral-table paragraphs so the long
  \\texttt/\\file paths start on a fresh line (explicit \\linebreak),
  removing the last overfull hbox warnings in app_star_catalogue.tex.

…al lv0-3 files for exclude-lower

- Command lines no longer carry .exe suffixes (extract_columns,
  --erfa-python <python>).
- Removed the '(levels 7-10 keep them...)' note and its equivalent in
  the --exclude-lower description.
- --exclude-lower now mentions that the official lv0-3 catalogues
  shipped with Stellarium can be used when none were generated.
- Removed the now-redundant overview item about the lv3/4 overlap
  deduplication (handled by --exclude-lower).
- Fix the remaining overfull hbox in the --stars-config paragraph by
  splitting the APPDATA path into short \\texttt segments.

…e overview

The sp table file name was still hard-coded as stars_hip_sp_0v0_6.cat;
with minor+1 naming the generated table is stars_hip_sp_0v0_7.cat, so the
script fell back to the 7110-row official table and every appended spInt
(7110..7197) was reported as an out-of-range '?NNNN'. Now the official
name comes from the hipSpectralFile entry and the generated side uses
minor+1, decoding all 7198 types.

… guide

bvextract extracts (source_id, B-V) from official .cat files into a
splitmix64-bucketed binary store; apply_bv streams a generated catalogue
and replaces the B-V field of stars found in the store, keeping the
polynomial value for the rest (partial runs safe, official data only
covers levels 0-8). Guide marks the step optional.

CodeFactor reported Complex Method on extract_cat, the bucket-finalize
loop and apply_cat. Extract per-record B-V decode, bucket finalization,
header validation and argument parsing into dedicated functions;
max CCN drops from 20+ to 13.

Directory layout: tool sources in src/, shared headers in include/,
pipeline Python scripts in python/, one-off diagnostics in attic/
(several assume the removed 60-byte .bin layout; noted in their
headers). gaiahip2cat now finds erfa_zone.py under python/.

New include/cat_reader.hpp (CatReader) unifies the six private .cat
header/zone-table readers (cmpcat, dedupcat, bvextract, apply_bv,
gaia2cat --exclude-lower) and adds the previously missing FILE_MAGIC
validation to cmpcat, dedupcat and load_cat_gaia_ids. gaiahip2cat now
links convert.cpp instead of carrying an identical g_to_v copy.
Also splits cmpcat matchOrRecordB (CCN 18) into within_tolerances /
track_bv_diff.

It is the only git-tracked diagnostic (exhaustive lv0-3 official-vs-ours
Star1 field comparison) and belongs with the other verification tools.

New include/cat_record.hpp provides DecodedStar + decode_record() for
all three record formats, replacing the three private decoder sets in
cmpcat (decode_star1/2/3), dedup_cat (StarInfo::decode) and bvextract
(extract_record_bv), which had already drifted apart (match_key/rv only
in cmpcat). Also centralizes the zero-adaptation encode helpers:
load/store_u24le, bv_milli_to_star3_raw and overwrite_bv_field (shared
by cat_writer and apply_bv). The Star1/Star2 full encoders stay local
because their input unit conventions genuinely differ.

Also fixes a pre-existing Windows bug exposed by the refactor
validation: cmpcat seeked zone offsets with std::fseek (32-bit long on
Windows), silently mis-seeking past 2 GB. lv8 self-compare (6.8 GB)
showed 12820 phantom mismatches; with fseeko it reports IDENTICAL.
Verified: Star1 (lv1), Star2 (lv4, lv8 212M) and Star3 (official
stars_8_2v0_3, 123M) self-compares all IDENTICAL; apply_bv lv1 100%
hit; dedupcat/bvextract counts unchanged.

New subsection 'Building the Tools' in the star-catalogue appendix: the
standalone CMake build of util/gaiacat (targets, bundled zlib, 64-bit
file offsets) and the per-script Python requirements (stdlib-only:
extract_columns.py, analyze_only.py; pandas: field_diff.py; numpy+
astropy: erfa_zone.py). erfa_zone.py now locates the companion repo's
py/geodesic module via the STELLARIUM_STAR_CATALOGS environment variable
instead of only a hardcoded local path. Also fixes the stale
field_diff.py path and cross-references the new subsection from
--erfa-python.