"""Backtest harness — measures the estimator's accuracy vs real ДКП sold prices.

Forgejo issues #648 (asking-core) + #1966 (full spine). **STRICTLY READ-ONLY**:
this script issues only SELECT queries against prod. It never INSERTs, UPDATEs,
or runs DDL.

TWO ENGINES (``--engine``)
--------------------------
  * ``full`` (DEFAULT, #1966) — runs the **full deterministic pricing spine**
    via ``estimator._price_from_inputs``: the same analog tier ladder
    (Tier 0 cohort → Tier A 1 km → wide 2 km → widearea ±25 %), same-building
    anchor comps, ДКП corridor clamp/floor, house Avito-IMV anchor, quarter
    price index, and the asking→sold ratio that yields ``expected_sold``. The
    PRODUCT-relevant prediction is ``expected_sold_per_m2`` (we also keep the
    asking headline ``median_ppm2`` so both are reported).
  * ``asking-core`` (#648) — the legacy asking-median + Tukey-IQR CORE only,
    plus the in-/out-of-sample per-rooms asking→sold correction block. Kept for
    comparison against the full spine.

WHAT IT MEASURES (full spine)
-----------------------------
Ground truth is `deals` — registered rosreestr ДКП sales (source='rosreestr')
that carry geom + price_per_m2 + rooms + area_m2 + floor/total_floors/year/type.
For a held-out sample of ДКП deals we, per deal:

  1. Replicate ``estimate_quality``'s analog tier ladder with ``_fetch_analogs``
     (same constants: DEFAULT_RADIUS_M, FALLBACK_RADIUS_M, MIN_ANALOGS_TIER_0,
     the <5/<3 widen thresholds) and pre-fetch the spine inputs
     (``_fetch_anchor_comps``, ``_fetch_dkp_corridor``,
     ``_fetch_house_imv_anchor``) + inject the DB callables
     (``_get_asking_sold_ratio``, ``_lookup_quarter_index(es)``).
  2. Call ``_price_from_inputs`` for a byte-identical headline + expected_sold.
     Deals the spine cannot price (median<=0) are skipped — prod parity (#1966):
     there is NO analog-count floor, low-analog deals surface at low confidence.
  3. Score ``expected_sold_per_m2`` vs the realised SOLD ppm².

METRICS (full spine)
--------------------
  * EXPECTED_SOLD signed error: median bias % + MAPE (median |%|), OVERALL +
    per-rooms + per-price-segment (эконом/комфорт/бизнес/элит/премиум).
  * RANGE COVERAGE: fraction of deals whose actual sold TOTAL (sold_ppm2 ×
    area) falls inside the predicted expected_sold RUB range — OVERALL +
    per-confidence bucket.
  * CONFIDENCE CALIBRATION: n / coverage / MAPE per confidence bucket
    (high should be tighter & more accurate — surfaces R2 risk).
  * SHARPNESS: median relative range width (high-low)/point — guards against
    gaming coverage with very wide ranges.
  * City-wide deal_median_ppm2 vs ask_median_ppm2 headline spread.

CAVEATS (read these before trusting the numbers)
-----------------------------------------------
  (a) TIME MISMATCH — this compares **CURRENT** active listings against
      **PAST** sold deals. It is NOT a point-in-time backtest: a deal closed
      in 2025-06 is judged against listings active today, so asking prices
      drift away from the historical sold price as the market moves. Treat the
      output as a REGRESSION BASELINE (relative change between runs / engines),
      NOT absolute truth. A faithful point-in-time backtest needs
      `listing_source_snapshots` (#570).
  (b) NETWORK LAYERS EXCLUDED — the spine runs offline, so the on-demand
      network valuation sources (Avito-IMV on-demand eval, Yandex valuation,
      Cian valuation) are NOT exercised (``imv_eval=None``,
      ``yandex/cian_val_present=False``). The populated `house_imv_evaluations`
      anchor IS injected, but it keys on house_id: by default the harness leaves
      ``target_house_id=None`` → no IMV anchor fires. Pass ``--resolve-house-id``
      (#2002) to resolve each deal's canonical house via ``match_house_readonly``
      so the Tier-S analog selector + the house IMV anchor fire, mirroring prod;
      the resulting reach is reported in the ``house_id_resolution`` block. Real
      estimate accuracy with the on-demand network layers may still differ.
  (c) ДКП ≠ TRUE MARKET — a registered ДКП price is what the parties declared
      to rosreestr; it can diverge from the genuine transaction price (tax
      optimisation, related-party sales, etc.).

PERFORMANCE
-----------
The full spine issues several spatial SELECTs per sampled deal (the tier ladder
+ anchor/corridor/imv pre-fetches), so runtime scales with `--sample`. The
default (300) is fine; large samples (thousands) are slow.

USAGE
-----
    DATABASE_URL=postgresql+psycopg://... \
        python -m scripts.backtest_estimator --sample 300 --since 2025-06-01

    # legacy asking-median core + correction block:
    python -m scripts.backtest_estimator --engine asking-core

    # machine-readable:
    python -m scripts.backtest_estimator --json

    # oblast D: per-city validation (exact deals.city name, not a slug):
    python -m scripts.backtest_estimator --city "Нижний Тагил" --sample 300
"""

from __future__ import annotations

import argparse
import dataclasses
import gzip
import json
import logging
import math
import statistics
from collections.abc import Callable
from dataclasses import dataclass
from datetime import date
from decimal import Decimal
from pathlib import Path
from types import SimpleNamespace
from typing import Any

from sqlalchemy import text
from sqlalchemy.orm import Session


def _import_estimator() -> tuple[Any, Any]:
    """Lazy import of the estimator's pure funcs (_filter_outliers, _percentile).

    Deferred so `--help` / the pure-metric unit tests don't pull
    app.core.config.Settings (which fail-fasts when DATABASE_URL is unset).
    Supports both `python -m scripts.backtest_estimator` and stand-alone runs.
    """
    try:
        from app.services.estimator import (  # type: ignore[import-not-found]
            _filter_outliers,
            _percentile,
        )
    except ImportError:  # pragma: no cover — fallback for adhoc invocation
        import sys

        sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
        from app.services.estimator import _filter_outliers, _percentile
    return _filter_outliers, _percentile


def _import_estimator_full() -> SimpleNamespace:
    """Lazy import of the FULL pricing spine + the helpers it needs (#1966).

    Returns a namespace bundling the estimator module (``m``), its ``settings``
    singleton, and the ``GeocodeResult`` dataclass. Deferred for the same reason
    as _import_estimator — importing app.services.estimator pulls
    app.core.config.Settings, which fail-fasts when DATABASE_URL is unset, so we
    keep it out of `--help` / the pure-metric unit tests.

    Everything the full-spine prediction path calls lives on ``ns.m``:
    _price_from_inputs, _fetch_analogs, _fetch_anchor_comps, _fetch_dkp_corridor,
    _fetch_house_imv_anchor, _get_asking_sold_ratio, _lookup_quarter_index(es),
    _target_cohort_range, and the DEFAULT_RADIUS_M / FALLBACK_RADIUS_M /
    MIN_ANALOGS_TIER_0 constants.
    """
    try:
        from app.services import estimator as est_mod  # type: ignore[import-not-found]
    except ImportError:  # pragma: no cover — fallback for adhoc invocation
        import sys

        sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
        from app.services import estimator as est_mod
    # settings + GeocodeResult are re-exported as module attributes of estimator
    # (imported there at module level), so we read them off the same object.
    return SimpleNamespace(
        m=est_mod,
        settings=est_mod.settings,
        GeocodeResult=est_mod.GeocodeResult,
    )


def _session() -> Session:
    """Lazy SessionLocal factory — see _import_estimator for why it's deferred."""
    try:
        from app.core.db import SessionLocal  # type: ignore[import-not-found]
    except ImportError:  # pragma: no cover — fallback for adhoc invocation
        import sys

        sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
        from app.core.db import SessionLocal
    return SessionLocal()


logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger("backtest_estimator")

# Price-per-m² sanity band — shared by the deal sample and the listings
# subquery. Mirrors the estimator's working range for EKB вторичка and drops
# obvious data-entry garbage / commercial outliers.
# (oblast D) When --city is set, _load_sample sources a PER-CITY band from
# deal_city_price_bands instead (see _resolve_city_ppm2_band) — these globals
# remain the fallback (default/unscoped sample, and any city with no band row,
# e.g. Екатеринбург — intentionally excluded from that table).
# NB: the listings subquery (_CANDIDATES_SQL, asking-core engine only) still
# always uses these globals — left global deliberately (not city-scoped) to
# keep the change minimal; the asking-core engine is the legacy comparison
# path, not the primary oblast-D target (the full engine, default).
PPM2_MIN = 30_000
PPM2_MAX = 600_000

# Minimum surviving candidates required to emit a prediction. Below this the
# median is too noisy to be meaningful — count the deal as "no analogs".
MIN_CANDIDATES = 3

# Room buckets for the per-rooms breakdown. 0 = студия; the top bucket is "4+".
ROOM_BUCKETS: tuple[int, ...] = (0, 1, 2, 3, 4)

# Minimum matched deals a rooms-bucket must hold before we trust its OWN
# asking→sold ratio. Thin buckets fall back to the global (all-buckets) ratio,
# whose larger n keeps the correction stable instead of overfitting a handful
# of deals. See _derive_room_ratios.
MIN_BUCKET = 20

# Confidence buckets the spine emits (estimator._compute_confidence /
# anchor / coarse-geo gates only ever return one of these three). Used by the
# range-coverage + calibration breakdowns. Any unexpected value → "other".
CONFIDENCE_BUCKETS: tuple[str, ...] = ("high", "medium", "low")

# Price-segment bands by ₽/m² (EKB вторичка). SINGLE SOURCE OF TRUTH now lives in
# ``estimator.PRICE_SEGMENTS_PPM2`` (#2255: the estimator's segment multiplier and
# this backtest must band identically) — we no longer duplicate the numbers here.
# Exposed lazily via module ``__getattr__``/``_price_segments()`` so importing the
# estimator (→ app.core.config.Settings, which fail-fasts without DATABASE_URL) is
# deferred out of `--help` / the pure-metric unit tests, mirroring _import_estimator.


def _price_segments() -> tuple[tuple[str, float], ...]:
    """Lazy accessor for the shared PRICE_SEGMENTS_PPM2 band table (from estimator).

    Deferred import (same reason as _import_estimator): pulling the estimator
    module eagerly would import Settings, which fail-fasts when DATABASE_URL is
    unset. Every caller here runs at RUNTIME, never at import time.
    """
    ns = _import_estimator_full()
    return ns.m.PRICE_SEGMENTS_PPM2  # type: ignore[no-any-return]


def __getattr__(name: str) -> Any:
    """PEP 562: expose ``PRICE_SEGMENTS_PPM2`` as a module attribute lazily.

    Keeps ``backtest_estimator.PRICE_SEGMENTS_PPM2`` working (tests read it) while
    avoiding an eager estimator/Settings import at module load.
    """
    if name == "PRICE_SEGMENTS_PPM2":
        return _price_segments()
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")


# --------------------------------------------------------------------------- #
# Data carriers
# --------------------------------------------------------------------------- #


@dataclass
class DealSample:
    """One held-out ДКП deal to backtest against.

    The full spine (#1966) needs the extra unit/house fields (area, address,
    floor, total_floors, year_built, house_type) to replicate ``_fetch_analogs``
    + the anchor/corridor pre-fetches; the asking-core engine ignores them.
    """

    id: int
    lon: float
    lat: float
    rooms: int
    sold_ppm2: float
    deal_date: Any  # datetime.date | None — carried through for reporting only
    area_m2: float = 0.0
    address: str | None = None
    floor: int | None = None
    total_floors: int | None = None
    year_built: int | None = None
    house_type: str | None = None


@dataclass
class Prediction:
    """One full-spine prediction for a deal that DID get priced (#1966).

    Carries both the asking headline (``median_ppm2``) and the product-relevant
    ``expected_sold_*`` outputs. Range fields are RUB TOTALS (the spine emits
    expected_sold_range_low/high as totals, not ₽/m²). ``expected_sold_*`` may be
    None when the spine produced a headline but no asking→sold ratio resolved.
    """

    deal_id: int
    rooms: int
    area_m2: float
    sold_ppm2: float
    median_ppm2: float  # asking headline ₽/m²
    confidence: str
    anchor_tier: str | None
    expected_sold_ppm2: float | None
    expected_sold_price: float | None  # RUB total point
    range_low: float | None  # expected_sold_range_low — RUB total
    range_high: float | None  # expected_sold_range_high — RUB total

    @property
    def sold_total(self) -> float:
        """Realised sold price as a RUB total (sold ₽/m² × area)."""
        return self.sold_ppm2 * self.area_m2


# --------------------------------------------------------------------------- #
# Pure metric helpers — NO DB. These are unit-tested in
# tests/test_backtest_estimator.py without a live database.
# --------------------------------------------------------------------------- #


def _rooms_label(rooms: int) -> str:
    """Human label for a room bucket: 0 → 'студия', 4 → '4+', else '<n>к'."""
    if rooms <= 0:
        return "студия"
    if rooms >= ROOM_BUCKETS[-1]:
        return f"{ROOM_BUCKETS[-1]}+"
    return f"{rooms}к"


def _bucketize_rooms(rooms: int) -> int:
    """Clamp a raw room count into a ROOM_BUCKETS key (4+ collapse to 4)."""
    if rooms <= 0:
        return 0
    return min(rooms, ROOM_BUCKETS[-1])


def _errors_summary(signed_errors: list[float]) -> dict[str, Any]:
    """Bias / MAPE / spread for a list of signed error percentages.

    - median_bias_pct  = median(signed_errors)  — systematic over/under-predict
    - mape_pct         = median(|signed_errors|) — typical magnitude (robust;
                          we use the MEDIAN absolute error, matching the brief's
                          MAPE-as-median-abs-error definition rather than mean)
    - p25 / p75        = quartiles of the SIGNED error (skew of the bias)

    Empty input → all-None (caller decides how to render "no data").
    """
    n = len(signed_errors)
    if n == 0:
        return {
            "n": 0,
            "median_bias_pct": None,
            "mape_pct": None,
            "p25_pct": None,
            "p75_pct": None,
        }
    abs_errors = [abs(e) for e in signed_errors]
    _, _percentile = _import_estimator()  # reuse estimator's interpolation percentile
    return {
        "n": n,
        "median_bias_pct": round(statistics.median(signed_errors), 2),
        "mape_pct": round(statistics.median(abs_errors), 2),
        "p25_pct": round(_percentile(sorted(signed_errors), 0.25), 2),
        "p75_pct": round(_percentile(sorted(signed_errors), 0.75), 2),
    }


def _compute_metrics(
    rows: list[tuple[float, float, int]],
    *,
    n_no_analogs: int = 0,
    per_rooms_no_analogs: dict[int, int] | None = None,
) -> dict[str, Any]:
    """Aggregate predicted-vs-sold rows into overall + per-rooms metrics.

    Each input row is (pred_ppm2, sold_ppm2, rooms) for a deal that DID get a
    prediction. `n_no_analogs` / `per_rooms_no_analogs` carry the skipped-deal
    counts so the report can show match coverage; they don't affect the error
    stats (those are computed only over matched deals).

    Returns a dict::

        {
          "overall": {n, n_no_analogs, median_bias_pct, mape_pct, p25_pct, p75_pct},
          "per_rooms": {
              0: {label, n, n_no_analogs, median_bias_pct, mape_pct, p25_pct, p75_pct},
              ...
          },
        }

    Pure: no DB, no I/O. signed_error_pct = 100*(pred - sold)/sold per row.
    Rows with sold_ppm2 <= 0 are dropped (cannot divide) — defensive; the SQL
    sample already excludes them.
    """
    per_rooms_no_analogs = per_rooms_no_analogs or {}

    overall_signed: list[float] = []
    by_bucket_signed: dict[int, list[float]] = {b: [] for b in ROOM_BUCKETS}

    for pred_ppm2, sold_ppm2, rooms in rows:
        if sold_ppm2 <= 0:
            continue
        signed = 100.0 * (pred_ppm2 - sold_ppm2) / sold_ppm2
        overall_signed.append(signed)
        by_bucket_signed[_bucketize_rooms(rooms)].append(signed)

    overall = _errors_summary(overall_signed)
    overall["n_no_analogs"] = n_no_analogs

    per_rooms: dict[int, dict[str, Any]] = {}
    for bucket in ROOM_BUCKETS:
        summary = _errors_summary(by_bucket_signed[bucket])
        summary["label"] = _rooms_label(bucket)
        summary["n_no_analogs"] = per_rooms_no_analogs.get(bucket, 0)
        per_rooms[bucket] = summary

    return {"overall": overall, "per_rooms": per_rooms}


def _derive_room_ratios(
    rows: list[tuple[float, float, int]],
    *,
    min_bucket: int = MIN_BUCKET,
) -> tuple[dict[int, float], dict[str, Any]]:
    """Derive a per-rooms asking→sold correction ratio from matched deals.

    For each row ``(pred_ask_ppm2, sold_ppm2, rooms)`` we want a multiplier that
    maps the estimator's asking-median prediction onto the realised SOLD price::

        ratio[bucket] = median(sold_ppm2) / median(pred_ask_ppm2)

    computed over the deals in that ROOM_BUCKETS bucket. A bucket holding fewer
    than ``min_bucket`` matched deals (or whose own median pred_ask is ≤ 0)
    falls back to the GLOBAL ratio — median(all sold) / median(all pred_ask) —
    so a handful of noisy deals can't overfit a per-rooms factor.

    Returns ``(ratios, meta)`` where:

      - ``ratios``: ``{bucket: float}`` for every bucket that has data (either
        its own ratio or the global fallback). Buckets with no matched deals at
        all are omitted (nothing to correct).
      - ``meta``: ``{"global_ratio": float | None, "fallback_buckets": [int],
        "bucket_n": {bucket: int}}`` — diagnostics for the report.

    !!! IN-SAMPLE WARNING
        These ratios are DERIVED from the same rows they will later be evaluated
        on (unless --holdout-split feeds disjoint rows). Used that way they make
        the corrected bias near-zero BY CONSTRUCTION — that proves the MECHANISM
        (a per-rooms multiplier removes the systematic asking→sold gap), NOT
        out-of-sample accuracy. The production ratio (Stage 2) is fit over a
        separate window and A/B'd on held-out deals.

    Pure: no DB, no I/O. Guards div-by-zero and empty input (→ ({}, meta)).
    """
    _, _percentile = _import_estimator()  # reuse estimator's interpolation percentile

    def _median(values: list[float]) -> float | None:
        if not values:
            return None
        return _percentile(sorted(values), 0.5)

    # Collect valid (pred>0, sold>0) ppm² pairs per bucket and globally.
    by_bucket_pred: dict[int, list[float]] = {b: [] for b in ROOM_BUCKETS}
    by_bucket_sold: dict[int, list[float]] = {b: [] for b in ROOM_BUCKETS}
    all_pred: list[float] = []
    all_sold: list[float] = []

    for pred_ppm2, sold_ppm2, rooms in rows:
        if pred_ppm2 <= 0 or sold_ppm2 <= 0:
            continue
        bucket = _bucketize_rooms(rooms)
        by_bucket_pred[bucket].append(pred_ppm2)
        by_bucket_sold[bucket].append(sold_ppm2)
        all_pred.append(pred_ppm2)
        all_sold.append(sold_ppm2)

    bucket_n = {b: len(by_bucket_pred[b]) for b in ROOM_BUCKETS}

    # Global fallback ratio — None when there's no data or pred median is 0.
    global_pred_med = _median(all_pred)
    global_sold_med = _median(all_sold)
    global_ratio: float | None = None
    if global_pred_med and global_pred_med > 0 and global_sold_med is not None:
        global_ratio = global_sold_med / global_pred_med

    ratios: dict[int, float] = {}
    fallback_buckets: list[int] = []
    for bucket in ROOM_BUCKETS:
        n = bucket_n[bucket]
        if n == 0:
            continue  # no deals in this bucket → nothing to correct
        pred_med = _median(by_bucket_pred[bucket])
        sold_med = _median(by_bucket_sold[bucket])
        if n >= min_bucket and pred_med and pred_med > 0 and sold_med is not None:
            ratios[bucket] = sold_med / pred_med
        elif global_ratio is not None:
            ratios[bucket] = global_ratio
            fallback_buckets.append(bucket)
        # else: no own ratio AND no global fallback → leave bucket uncorrected.

    meta = {
        "global_ratio": (round(global_ratio, 4) if global_ratio is not None else None),
        "fallback_buckets": fallback_buckets,
        "bucket_n": bucket_n,
        "min_bucket": min_bucket,
    }
    return ratios, meta


def _apply_ratios(
    rows: list[tuple[float, float, int]],
    ratios: dict[int, float],
) -> list[tuple[float, float, int]]:
    """Apply per-rooms ratios → corrected rows for re-scoring via _compute_metrics.

    ``pred_sold = pred_ask * ratio[bucket]``. A bucket with no ratio (e.g. no
    global fallback was available) leaves its rows unchanged so they still count
    in the corrected block rather than vanishing. Pure: no DB.
    """
    out: list[tuple[float, float, int]] = []
    for pred_ask, sold, rooms in rows:
        ratio = ratios.get(_bucketize_rooms(rooms), 1.0)
        out.append((pred_ask * ratio, sold, rooms))
    return out


# --------------------------------------------------------------------------- #
# Full-spine metric helpers (#1966) — PURE, no DB. Unit-tested.
# --------------------------------------------------------------------------- #


def _bucketize_confidence(confidence: str) -> str:
    """Map a confidence string to a calibration bucket (unknown → 'other')."""
    return confidence if confidence in CONFIDENCE_BUCKETS else "other"


def _segment_label(ppm2: float) -> str:
    """Price-segment label for a ₽/m² value — delegates to the estimator's helper.

    Single source of truth: estimator._segment_for_ppm2 uses the shared
    PRICE_SEGMENTS_PPM2 band table (#2255), so backtest bucketing and the
    estimator's segment multiplier band identically.
    """
    ns = _import_estimator_full()
    return ns.m._segment_for_ppm2(ppm2)  # type: ignore[no-any-return]


def _segment_metrics(rows: list[tuple[float, float]]) -> dict[str, dict[str, Any]]:
    """Per-price-segment signed-error summary, bucketed by the SOLD ₽/m².

    Each input row is ``(pred_ppm2, sold_ppm2)``. We bucket by the segment of the
    SOLD price (ground truth), compute signed_error_pct = 100*(pred-sold)/sold,
    and run `_errors_summary` per band. Rows with sold<=0 are dropped (can't
    divide). Every band in PRICE_SEGMENTS_PPM2 is present (n=0 when empty) so the
    report renders a stable table. No DB (band table via one lazy estimator import).
    """
    ns = _import_estimator_full()
    segments = ns.m.PRICE_SEGMENTS_PPM2
    seg_for = ns.m._segment_for_ppm2
    by_seg: dict[str, list[float]] = {label: [] for label, _ in segments}
    for pred, sold in rows:
        if sold <= 0:
            continue
        by_seg[seg_for(sold)].append(100.0 * (pred - sold) / sold)
    out: dict[str, dict[str, Any]] = {}
    for label, _ in segments:
        out[label] = _errors_summary(by_seg[label])
    return out


def _range_coverage(rows: list[tuple[float, float, float]]) -> dict[str, Any]:
    """Coverage = fraction of (actual, low, high) rows with low<=actual<=high.

    Boundary INCLUSIVE (a deal sitting exactly on the range edge counts as
    covered). Empty input → coverage_pct None. Pure: no DB.
    """
    n = len(rows)
    if n == 0:
        return {"n": 0, "n_covered": 0, "coverage_pct": None}
    covered = sum(1 for actual, lo, hi in rows if lo <= actual <= hi)
    return {"n": n, "n_covered": covered, "coverage_pct": round(100.0 * covered / n, 2)}


def _sharpness(rows: list[tuple[float, float, float]]) -> dict[str, Any]:
    """Median relative range width = median((high-low)/point) over (point,lo,hi).

    Guards against gaming coverage with arbitrarily wide ranges: a model can
    always cover 100% by predicting an infinite range, so we report how WIDE the
    ranges are relative to the point estimate. Rows with point<=0 are dropped.
    Empty → median_rel_width None. Pure: no DB.
    """
    widths = [(hi - lo) / point for point, lo, hi in rows if point > 0]
    if not widths:
        return {"n": 0, "median_rel_width": None}
    _, _percentile = _import_estimator()  # reuse estimator's interpolation percentile
    return {"n": len(widths), "median_rel_width": round(_percentile(sorted(widths), 0.5), 4)}


def _calibration_metrics(
    rows: list[tuple[str, float | None, bool | None]],
    *,
    bucket_order: list[str] | None = None,
) -> dict[str, dict[str, Any]]:
    """Per-confidence n / coverage% / MAPE% — surfaces the R2 risk.

    Each input row is ``(confidence, signed_error_pct | None, covered | None)``:
      - ``n``           = predictions in this confidence bucket (always counted).
      - ``coverage_pct``= 100 * mean(covered) over rows where covered is not None.
      - ``mape_pct``    = median(|signed_error_pct|) over rows where signed is not
                          None (the brief's median-abs-error MAPE definition).

    A well-calibrated estimator shows HIGH confidence = higher coverage AND lower
    MAPE than LOW. The canonical high/medium/low buckets are always present (n=0
    when empty); an "other" bucket appears only if an off-enum confidence shows
    up. Pure: no DB.
    """
    order = list(bucket_order) if bucket_order is not None else list(CONFIDENCE_BUCKETS)
    coll: dict[str, dict[str, Any]] = {}
    for confidence, signed, covered in rows:
        b = _bucketize_confidence(confidence)
        slot = coll.setdefault(b, {"n": 0, "abs_errors": [], "covered": []})
        slot["n"] += 1
        if signed is not None:
            slot["abs_errors"].append(abs(signed))
        if covered is not None:
            slot["covered"].append(covered)
        if b not in order:
            order.append(b)

    out: dict[str, dict[str, Any]] = {}
    for b in order:
        slot = coll.get(b, {"n": 0, "abs_errors": [], "covered": []})
        cov = slot["covered"]
        abs_errors = slot["abs_errors"]
        n_covered = sum(1 for c in cov if c)
        out[b] = {
            "n": slot["n"],
            "n_covered": n_covered,
            "coverage_pct": (round(100.0 * n_covered / len(cov), 2) if cov else None),
            "mape_pct": (round(statistics.median(abs_errors), 2) if abs_errors else None),
        }
    return out


def _expected_sold_metrics(
    rows: list[tuple[float, float, int]],
    *,
    n_no_prediction: int = 0,
    per_rooms_no_prediction: dict[int, int] | None = None,
) -> dict[str, Any]:
    """expected_sold error block: overall + per-rooms (via _compute_metrics) + per-segment.

    Each input row is ``(expected_sold_ppm2, sold_ppm2, rooms)``. Reuses the
    existing `_compute_metrics` for the overall/per-rooms split (so the no-data
    counts ride along as ``n_no_analogs`` — here meaning "no prediction") and
    attaches a per-price-segment breakdown keyed by the SOLD ₽/m². Pure: no DB.
    """
    m = _compute_metrics(
        rows,
        n_no_analogs=n_no_prediction,
        per_rooms_no_analogs=per_rooms_no_prediction,
    )
    m["per_segment"] = _segment_metrics([(pred, sold) for pred, sold, _ in rows])
    return m


def _compute_full_metrics(
    predictions: list[Prediction],
    *,
    n_no_prediction: int = 0,
    per_rooms_no_prediction: dict[int, int] | None = None,
) -> dict[str, Any]:
    """Aggregate full-spine Prediction records into the complete metrics dict.

    Blocks (all over deals that DID get an expected_sold, except where noted):
      - ``expected_sold``  : signed bias / MAPE overall + per-rooms + per-segment.
      - ``range_coverage`` : sold-total ∈ expected_sold range, overall +
                             per-confidence (counts ALL priced deals; a deal with
                             no expected_sold range contributes to n but not to
                             coverage).
      - ``calibration``    : per-confidence n / coverage% / MAPE%.
      - ``sharpness``      : median relative range width (high-low)/point.

    Pure: no DB. Safe on an empty list (every block renders with None/0).
    """
    # Confidence bucket order: canonical three, then any extras encountered.
    conf_order = list(CONFIDENCE_BUCKETS)
    for p in predictions:
        b = _bucketize_confidence(p.confidence)
        if b not in conf_order:
            conf_order.append(b)

    es_rows: list[tuple[float, float, int]] = []  # (expected_sold_ppm2, sold_ppm2, rooms)
    cov_rows: list[tuple[float, float, float]] = []  # (sold_total, range_low, range_high)
    cov_by_conf: dict[str, list[tuple[float, float, float]]] = {b: [] for b in conf_order}
    sharp_rows: list[tuple[float, float, float]] = []  # (point, range_low, range_high)
    calib_rows: list[tuple[str, float | None, bool | None]] = []

    for p in predictions:
        signed: float | None = None
        if p.expected_sold_ppm2 is not None and p.sold_ppm2 > 0:
            signed = 100.0 * (p.expected_sold_ppm2 - p.sold_ppm2) / p.sold_ppm2
            es_rows.append((p.expected_sold_ppm2, p.sold_ppm2, p.rooms))

        covered: bool | None = None
        if p.range_low is not None and p.range_high is not None:
            covered = p.range_low <= p.sold_total <= p.range_high
            cov_rows.append((p.sold_total, p.range_low, p.range_high))
            cov_by_conf[_bucketize_confidence(p.confidence)].append(
                (p.sold_total, p.range_low, p.range_high)
            )
            if p.expected_sold_price is not None:
                sharp_rows.append((p.expected_sold_price, p.range_low, p.range_high))

        calib_rows.append((p.confidence, signed, covered))

    return {
        "expected_sold": _expected_sold_metrics(
            es_rows,
            n_no_prediction=n_no_prediction,
            per_rooms_no_prediction=per_rooms_no_prediction,
        ),
        "range_coverage": {
            "overall": _range_coverage(cov_rows),
            "per_confidence": {b: _range_coverage(cov_by_conf[b]) for b in conf_order},
        },
        "calibration": _calibration_metrics(calib_rows, bucket_order=conf_order),
        "sharpness": _sharpness(sharp_rows),
        "confidence_order": conf_order,
    }


def _render_table(metrics: dict[str, Any], headline: dict[str, Any]) -> str:
    """Render the aggregated metrics as a plain-text stdout report."""
    lines: list[str] = []
    lines.append("=" * 78)
    lines.append("BACKTEST: estimator asking-median vs rosreestr ДКП sold prices")
    lines.append("=" * 78)

    # Headline city-wide spread (asking median vs deal median, ppm²).
    dm = headline.get("deal_median_ppm2")
    am = headline.get("ask_median_ppm2")
    spread = headline.get("spread_pct")
    lines.append("")
    lines.append("CITY-WIDE HEADLINE (sample medians, ₽/m²):")
    lines.append(f"  deal_median_ppm2 (SOLD):    {_fmt_ppm2(dm)}")
    lines.append(f"  ask_median_ppm2  (ASKING):  {_fmt_ppm2(am)}")
    lines.append(f"  spread (ask vs deal):       {_fmt_pct(spread)}")

    # ASKING block — the estimator's raw asking-median prediction vs SOLD.
    lines.append("")
    lines.append("PER-DEAL ERROR (signed = 100*(pred-sold)/sold; +ve = over-predict):")
    lines.extend(_render_metrics_block("[ASKING] estimator asking-median (uncorrected)", metrics))

    # CORRECTED block — asking-median × per-rooms asking→sold ratio (#648 S1).
    corrected = metrics.get("corrected")
    ratios_meta = metrics.get("ratios_meta") or {}
    if corrected is not None:
        lines.append("")
        lines.extend(
            _render_metrics_block(
                "[CORRECTED] asking-median × per-rooms asking→sold ratio",
                corrected,
            )
        )
        lines.append("")
        lines.extend(_render_ratios(metrics.get("ratios") or {}, ratios_meta))
        lines.append("")
        if ratios_meta.get("holdout_split"):
            lines.append(
                "Ratios fit on EVEN-id deals, evaluated on ODD-id deals "
                "(--holdout-split) → this is an OUT-OF-SAMPLE corrected number."
            )
        else:
            lines.append(
                "!! IN-SAMPLE: ratios were derived AND evaluated on the same deals, so the"
            )
            lines.append(
                "   corrected bias is near-zero BY CONSTRUCTION. This proves the MECHANISM"
            )
            lines.append(
                "   (a per-rooms ratio removes the systematic asking→sold gap), NOT out-of-"
            )
            lines.append(
                "   sample accuracy. Re-run with --holdout-split for an honest number; the"
            )
            lines.append("   real A/B (Stage 2) fits the ratio on a separate window.")

    lines.append("")
    lines.append("Caveats: CURRENT listings vs PAST deals (not point-in-time);")
    lines.append("measures asking-median+IQR core only; ДКП = registered price.")
    lines.append("=" * 78)
    return "\n".join(lines)


def _render_metrics_block(
    title: str, metrics: dict[str, Any], *, no_col: str = "no_analog"
) -> list[str]:
    """Render one OVERALL + per-rooms metrics table (shared by all blocks).

    ``no_col`` labels the skipped-deal column header — "no_analog" for the
    asking-core block, "no_pred" for the full-spine expected_sold block (where it
    counts deals the spine could not price).
    """
    header = (
        f"  {'bucket':<8} {'n':>5} {no_col:>10} {'bias%':>8} {'MAPE%':>8} {'p25%':>8} {'p75%':>8}"
    )
    out: list[str] = [title, header, "  " + "-" * (len(header) - 2)]
    out.append(_fmt_row("OVERALL", metrics["overall"]))
    for bucket in ROOM_BUCKETS:
        row = metrics["per_rooms"][bucket]
        out.append(_fmt_row(row["label"], row))
    return out


def _render_ratios(ratios: dict[int, float], meta: dict[str, Any]) -> list[str]:
    """Render the derived per-rooms asking→sold ratios + fallback flags."""
    out: list[str] = ["DERIVED per-rooms asking→sold ratios (sold_med / ask_med):"]
    fallback = set(meta.get("fallback_buckets") or [])
    bucket_n = meta.get("bucket_n") or {}
    gr = meta.get("global_ratio")
    min_bucket = meta.get("min_bucket", MIN_BUCKET)
    if not ratios:
        out.append("  (none — empty / no-prediction sample)")
    for bucket in ROOM_BUCKETS:
        if bucket not in ratios:
            continue
        flag = f" [global fallback, n<{min_bucket}]" if bucket in fallback else ""
        out.append(
            f"  {_rooms_label(bucket):<8} "
            f"ratio={ratios[bucket]:.4f}  n={bucket_n.get(bucket, 0)}{flag}"
        )
    out.append(f"  global fallback ratio: {gr if gr is not None else 'n/a'}")
    return out


def _fmt_row(label: str, m: dict[str, Any]) -> str:
    """Format one metrics row for the table."""
    return (
        f"  {label:<8} {m.get('n', 0):>5} {m.get('n_no_analogs', 0):>10} "
        f"{_fmt_pct(m.get('median_bias_pct')):>8} {_fmt_pct(m.get('mape_pct')):>8} "
        f"{_fmt_pct(m.get('p25_pct')):>8} {_fmt_pct(m.get('p75_pct')):>8}"
    )


def _fmt_pct(v: float | None) -> str:
    return "  n/a" if v is None else f"{v:+.1f}"


def _fmt_ppm2(v: float | None) -> str:
    return "n/a" if v is None else f"{round(v):,}".replace(",", " ")


def _fmt_cov(v: float | None) -> str:
    """Coverage percent — non-signed (0..100), n/a for None."""
    return "n/a" if v is None else f"{v:.1f}"


def _fmt_ratio(v: float | None) -> str:
    """Relative-width / ratio formatter (n/a for None)."""
    return "n/a" if v is None else f"{v:.3f}"


# --------------------------------------------------------------------------- #
# Full-spine renderer (#1966)
# --------------------------------------------------------------------------- #


def _render_full_table(metrics: dict[str, Any]) -> str:
    """Render the full-spine metrics dict (#1966) as a plain-text stdout report."""
    headline = metrics.get("headline") or {}
    lines: list[str] = []
    lines.append("=" * 78)
    lines.append("BACKTEST [full spine]: _price_from_inputs expected_sold vs ДКП sold")
    lines.append("=" * 78)

    dm = headline.get("deal_median_ppm2")
    am = headline.get("ask_median_ppm2")
    spread = headline.get("spread_pct")
    lines.append("")
    lines.append("CITY-WIDE HEADLINE (sample medians, ₽/m²):")
    lines.append(f"  deal_median_ppm2 (SOLD):        {_fmt_ppm2(dm)}")
    lines.append(f"  ask_median_ppm2  (ASKING head): {_fmt_ppm2(am)}")
    lines.append(f"  spread (asking vs deal):        {_fmt_pct(spread)}")

    es = metrics["expected_sold"]
    lines.append("")
    lines.append("EXPECTED_SOLD ERROR (signed = 100*(expected_sold-sold)/sold; +ve = over):")
    lines.extend(_render_metrics_block("[EXPECTED_SOLD] per-rooms", es, no_col="no_pred"))
    lines.append("")
    lines.extend(_render_segment_block(es["per_segment"]))

    conf_order = metrics.get("confidence_order") or list(CONFIDENCE_BUCKETS)
    lines.append("")
    lines.extend(_render_coverage_block(metrics["range_coverage"], conf_order))
    lines.append("")
    lines.extend(_render_calibration_block(metrics["calibration"], conf_order))

    sh = metrics["sharpness"]
    lines.append("")
    lines.append(
        f"SHARPNESS: median relative range width (high-low)/point = "
        f"{_fmt_ratio(sh.get('median_rel_width'))} (n={sh.get('n', 0)})"
    )

    lines.append("")
    lines.append("Caveats: CURRENT listings vs PAST deals (time-mismatched) →")
    lines.append("regression baseline, not absolute truth; network valuation")
    lines.append("layers (Avito-IMV on-demand / Yandex / Cian) excluded.")
    lines.append("=" * 78)
    return "\n".join(lines)


def _render_segment_block(per_segment: dict[str, Any]) -> list[str]:
    """Render the per-price-segment expected_sold error table."""
    header = f"  {'segment':<10} {'n':>5} {'bias%':>8} {'MAPE%':>8} {'p25%':>8} {'p75%':>8}"
    out: list[str] = [
        "[EXPECTED_SOLD] per price-segment (band by SOLD ₽/m²):",
        header,
        "  " + "-" * (len(header) - 2),
    ]
    # per_segment is built in band order by _segment_metrics → dict insertion
    # order preserves it (no need to re-import the constant in a pure renderer).
    for label, m in per_segment.items():
        out.append(
            f"  {label:<10} {m.get('n', 0):>5} "
            f"{_fmt_pct(m.get('median_bias_pct')):>8} {_fmt_pct(m.get('mape_pct')):>8} "
            f"{_fmt_pct(m.get('p25_pct')):>8} {_fmt_pct(m.get('p75_pct')):>8}"
        )
    return out


# --------------------------------------------------------------------------- #
# #2255 --calibrate-segments — per-segment multiplier proposal (PRINT-ONLY)
# --------------------------------------------------------------------------- #

# Shrinkage denominator for λ = n/(n+SHRINK_N): pulls thin-sample multipliers back
# toward 1.0 (no correction) so a 16-deal segment isn't over-trusted. Larger → more
# conservative. 40 chosen so элит (n≈16) lands ~0.29 weight, бизнес (n≈76) ~0.66.
_SEGMENT_SHRINK_N = 40


def _render_calibrate_segments_block(per_segment: dict[str, Any]) -> list[str]:
    """Propose per-segment multipliers from expected_sold bias (#2255). PRINT-ONLY.

    For each band: raw = 1/(1+bias) undoes the median signed bias; λ = n/(n+40)
    shrinks it toward 1.0 by sample size; suggested m = 1 + λ·(raw − 1). Segments
    with no bias/n render "—". This proposes numbers for
    ``estimate_segment_multipliers``; it applies nothing.
    """
    header = f"  {'segment':<10} {'n':>5} {'bias%':>8} {'raw m':>8} {'λ':>6} {'suggest m':>10}"
    out: list[str] = [
        "[#2255 CALIBRATE] segment multiplier proposal (PRINT-ONLY, applies nothing):",
        f"  m undoes expected_sold bias, shrunk toward 1.0 by λ=n/(n+{_SEGMENT_SHRINK_N}).",
        header,
        "  " + "-" * (len(header) - 2),
    ]
    for label, m in per_segment.items():
        n = int(m.get("n", 0) or 0)
        bias = m.get("median_bias_pct")
        if bias is None or n == 0:
            out.append(f"  {label:<10} {n:>5} {'—':>8} {'—':>8} {'—':>6} {'—':>10}")
            continue
        bias_frac = float(bias) / 100.0
        # raw multiplier that would drive median bias to 0 (guard div-by-~0).
        raw = 1.0 / (1.0 + bias_frac) if abs(1.0 + bias_frac) > 1e-9 else 1.0
        lam = n / (n + _SEGMENT_SHRINK_N)
        suggested = 1.0 + lam * (raw - 1.0)
        out.append(
            f"  {label:<10} {n:>5} {_fmt_pct(bias):>8} {raw:>8.3f} {lam:>6.2f} {suggested:>10.3f}"
        )
    out.append("")
    out.append(
        "  NB: элит capped in config (thin n) — do NOT paste raw suggestions for n<20 verbatim."
    )
    return out


def _render_coverage_block(range_coverage: dict[str, Any], conf_order: list[str]) -> list[str]:
    """Render range-coverage: overall + per-confidence (sold_total ∈ range)."""
    ov = range_coverage["overall"]
    out: list[str] = ["RANGE COVERAGE (sold total within predicted expected_sold range):"]
    out.append(
        f"  OVERALL    n={ov['n']:>5}  covered={ov['n_covered']:>5}  "
        f"coverage={_fmt_cov(ov['coverage_pct'])}%"
    )
    per_conf = range_coverage.get("per_confidence") or {}
    for b in conf_order:
        c = per_conf.get(b, {"n": 0, "n_covered": 0, "coverage_pct": None})
        out.append(
            f"  {b:<10} n={c['n']:>5}  covered={c['n_covered']:>5}  "
            f"coverage={_fmt_cov(c['coverage_pct'])}%"
        )
    return out


def _render_calibration_block(calibration: dict[str, Any], conf_order: list[str]) -> list[str]:
    """Render confidence-calibration: per bucket n / coverage% / MAPE%."""
    header = f"  {'conf':<8} {'n':>5} {'coverage%':>10} {'MAPE%':>8}"
    out: list[str] = [
        "CONFIDENCE CALIBRATION (high should be tighter & more accurate):",
        header,
        "  " + "-" * (len(header) - 2),
    ]
    for b in conf_order:
        c = calibration.get(b, {"n": 0, "coverage_pct": None, "mape_pct": None})
        out.append(
            f"  {b:<8} {c.get('n', 0):>5} "
            f"{_fmt_cov(c.get('coverage_pct')):>10} {_fmt_pct(c.get('mape_pct')):>8}"
        )
    return out


# --------------------------------------------------------------------------- #
# DB layer — READ-ONLY SELECTs only.
# --------------------------------------------------------------------------- #

# ДКП deal sample. lon/lat extracted via ST_X/ST_Y so the per-deal listings
# query can rebuild the point without re-reading geom. Parameterized;
# psycopg3 CAST(:x AS type), never :x::type.
_SAMPLE_SQL = text(
    """
    SELECT
        id,
        ST_X(geom::geometry) AS lon,
        ST_Y(geom::geometry) AS lat,
        rooms,
        price_per_m2 AS sold_ppm2,
        deal_date,
        area_m2,
        address,
        floor,
        total_floors,
        year_built,
        house_type
    FROM deals
    WHERE source = 'rosreestr'
      AND geom IS NOT NULL
      AND price_per_m2 BETWEEN CAST(:ppm2_min AS numeric) AND CAST(:ppm2_max AS numeric)
      AND rooms IS NOT NULL
      AND area_m2 IS NOT NULL
      AND area_m2 > 0
      AND deal_date >= CAST(:since AS date)
    ORDER BY id DESC
    LIMIT CAST(:sample AS integer)
    """
)

# Per-deal candidate active listings. rooms matched within :rooms_lo..:rooms_hi
# (exact when tolerance=0). Returns raw price_per_m2 values — _filter_outliers
# is applied in Python for byte-for-byte fidelity with the estimator.
_CANDIDATES_SQL = text(
    """
    SELECT price_per_m2
    FROM listings
    WHERE is_active
      AND rooms BETWEEN CAST(:rooms_lo AS integer) AND CAST(:rooms_hi AS integer)
      AND price_per_m2 BETWEEN CAST(:ppm2_min AS numeric) AND CAST(:ppm2_max AS numeric)
      AND ST_DWithin(
              geom::geography,
              ST_SetSRID(ST_MakePoint(
                  CAST(:lon AS double precision),
                  CAST(:lat AS double precision)
              ), 4326)::geography,
              CAST(:radius AS double precision)
          )
    """
)

# (oblast D) Per-city PPM2 sanity band — same table the estimator's
# _fetch_dkp_corridor COALESCEs against (migration 178). Looked up only when
# --city is set (see _resolve_city_ppm2_band); the default (city=None) path
# never issues this query.
_CITY_PPM2_BAND_SQL = text(
    """
    SELECT ppm2_min, ppm2_max
    FROM deal_city_price_bands
    WHERE city = CAST(:city AS text)
    """
)


def _sample_sql(city: str | None) -> Any:
    """ДКП deal-sample SELECT — optionally scoped to one ``deals.city`` (oblast D).

    ``city is None`` (default) returns the SAME ``_SAMPLE_SQL`` object used
    before this change — literal identity, not just equal text — so the
    default CLI invocation (and the frozen EKB regression gate, which never
    calls this path at all) see byte-identical SQL.

    When ``city`` is set, an extra ``AND city = CAST(:city AS text)`` predicate
    scopes the sample to that ``deals.city`` value — populated oblast-wide by
    the #C1 migration (368 cities, 96,974 deals). Pass the EXACT canonical
    Russian name as stored in ``deals.city`` (e.g. ``'Нижний Тагил'``), not a
    slug — see the ``--city`` CLI help for the naming decision.
    """
    if city is None:
        return _SAMPLE_SQL
    return text(
        """
        SELECT
            id,
            ST_X(geom::geometry) AS lon,
            ST_Y(geom::geometry) AS lat,
            rooms,
            price_per_m2 AS sold_ppm2,
            deal_date,
            area_m2,
            address,
            floor,
            total_floors,
            year_built,
            house_type
        FROM deals
        WHERE source = 'rosreestr'
          AND geom IS NOT NULL
          AND price_per_m2 BETWEEN CAST(:ppm2_min AS numeric) AND CAST(:ppm2_max AS numeric)
          AND rooms IS NOT NULL
          AND area_m2 IS NOT NULL
          AND area_m2 > 0
          AND deal_date >= CAST(:since AS date)
          AND city = CAST(:city AS text)
        ORDER BY id DESC
        LIMIT CAST(:sample AS integer)
        """
    )


def _resolve_city_ppm2_band(db: Session, city: str | None) -> tuple[float, float]:
    """Per-city PPM2 sanity band (oblast D) — falls back to the module globals.

    ``city is None`` → ``(PPM2_MIN, PPM2_MAX)``, unchanged default behaviour,
    NO extra query. Otherwise looks up ``deal_city_price_bands`` for that
    city's own ``[ppm2_min, ppm2_max]`` — a region-66 town can have a
    materially different sane ₽/m² range than the EKB-tuned 30k..600k globals
    (e.g. Нижний Тагил: 16 955..108 175). No row for the city (e.g.
    Екатеринбург — intentionally excluded from the table, mirrors
    estimator._fetch_dkp_corridor's own COALESCE fallback) or any DB error →
    the globals; read-only best-effort, never raises.
    """
    if city is None:
        return float(PPM2_MIN), float(PPM2_MAX)
    try:
        row = db.execute(_CITY_PPM2_BAND_SQL, {"city": city}).mappings().first()
    except Exception as exc:  # pragma: no cover — defensive, read-only best-effort
        logger.warning("city PPM2 band lookup failed for %r (fallback to global): %s", city, exc)
        return float(PPM2_MIN), float(PPM2_MAX)
    if row is None or row["ppm2_min"] is None or row["ppm2_max"] is None:
        return float(PPM2_MIN), float(PPM2_MAX)
    return float(row["ppm2_min"]), float(row["ppm2_max"])


def _load_sample(
    db: Session, *, sample: int, since: str, city: str | None = None
) -> list[DealSample]:
    """Run the held-out ДКП deal sampling SELECT → list[DealSample].

    ``city`` (oblast D, default None) scopes the sample to one ``deals.city``
    value via ``_sample_sql`` and sources the PPM2 sanity band from
    ``deal_city_price_bands`` for that city (``_resolve_city_ppm2_band``,
    falls back to the module globals). Default None is byte-identical to the
    pre-oblast-D behaviour: same SQL object, same PPM2_MIN/PPM2_MAX globals,
    no extra query.
    """
    if city is None:
        ppm2_min: float = PPM2_MIN
        ppm2_max: float = PPM2_MAX
    else:
        ppm2_min, ppm2_max = _resolve_city_ppm2_band(db, city)
    params: dict[str, Any] = {
        "ppm2_min": ppm2_min,
        "ppm2_max": ppm2_max,
        "since": since,
        "sample": sample,
    }
    if city is not None:
        params["city"] = city
    rows = db.execute(_sample_sql(city), params).mappings().all()
    out: list[DealSample] = []
    for r in rows:
        if r["lon"] is None or r["lat"] is None or r["sold_ppm2"] is None:
            continue
        if r["area_m2"] is None or float(r["area_m2"]) <= 0:
            continue
        out.append(
            DealSample(
                id=r["id"],
                lon=float(r["lon"]),
                lat=float(r["lat"]),
                rooms=int(r["rooms"]),
                sold_ppm2=float(r["sold_ppm2"]),
                deal_date=r["deal_date"],
                area_m2=float(r["area_m2"]),
                address=r["address"],
                floor=(int(r["floor"]) if r["floor"] is not None else None),
                total_floors=(int(r["total_floors"]) if r["total_floors"] is not None else None),
                year_built=(int(r["year_built"]) if r["year_built"] is not None else None),
                house_type=r["house_type"],
            )
        )
    return out


def _predict_for_deal(
    db: Session,
    deal: DealSample,
    *,
    radius: int,
    rooms_tolerance: int,
) -> float | None:
    """Predict asking ppm² for one deal by reusing the estimator's pure funcs.

    Fetches candidate active-listing ppm² values, wraps them as
    ``{"price_per_m2": p}`` dicts, applies the estimator's `_filter_outliers`
    (Tukey IQR), then `_percentile(sorted, 0.5)`. Returns None when fewer than
    MIN_CANDIDATES survive (caller counts it as a no-analog miss).
    """
    rows = db.execute(
        _CANDIDATES_SQL,
        {
            "rooms_lo": deal.rooms - rooms_tolerance,
            "rooms_hi": deal.rooms + rooms_tolerance,
            "ppm2_min": PPM2_MIN,
            "ppm2_max": PPM2_MAX,
            "lon": deal.lon,
            "lat": deal.lat,
            "radius": radius,
        },
    ).all()

    # Build the same dict shape the estimator feeds _filter_outliers.
    lots = [{"price_per_m2": float(r[0])} for r in rows if r[0] is not None]
    if len(lots) < MIN_CANDIDATES:
        return None

    _filter_outliers, _percentile = _import_estimator()
    clean = _filter_outliers(lots)
    prices = sorted(lot["price_per_m2"] for lot in clean if lot["price_per_m2"])
    if len(prices) < MIN_CANDIDATES:
        return None
    return _percentile(prices, 0.5)


@dataclass
class _HouseIdResolution:
    """Coverage counters for ``--resolve-house-id`` (Tier-S + IMV anchor reach).

    Populated only on the full-engine path when the flag is on. ``imv_reachable``
    counts deals whose resolved house hit a band-compatible ``house_imv_evaluations``
    row — i.e. ``_fetch_house_imv_anchor`` returned a non-None anchor that would
    actually fire — so it measures the IMV anchor's true reach, not bare row
    existence. Surfaced as the ``house_id_resolution`` block in ``--json``.
    """

    total: int = 0  # deals processed in the full-engine path under the flag
    resolved: int = 0  # deals where match_house_readonly returned an id
    imv_reachable: int = 0  # of resolved, those whose house_imv anchor fired

    def as_json(self) -> dict[str, int]:
        return {
            "resolved": self.resolved,
            "total": self.total,
            "imv_reachable": self.imv_reachable,
        }


def _resolve_house_id_for_deal(db: Session, deal: DealSample) -> int | None:
    """Resolve a deal's canonical house_id (read-only) — mirrors prod (#2002).

    Calls ``match_house_readonly(db, address=…, lat=…, lon=…)`` exactly as
    ``estimate_quality`` does (estimator.py:2554-2569): read-only, never INSERTs a
    house, takes no advisory lock. Returns ``match[0]`` (the house_id), or ``None``
    on no-match / empty-or-malformed tuple / any exception — best-effort, exactly
    like prod which degrades to the geo tiers when no house resolves.

    A defensive ``db.rollback()`` runs if the lookup raises so a poisoned tx does
    not bleed into the per-deal spine queries (consistent with the loop's own
    rollback-on-failure handling).
    """
    try:
        from app.services.matching.houses import (  # type: ignore[import-not-found]
            match_house_readonly,
        )
    except ImportError:  # pragma: no cover — fallback for adhoc invocation
        import sys

        sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
        from app.services.matching.houses import match_house_readonly

    try:
        match = match_house_readonly(db, address=deal.address, lat=deal.lat, lon=deal.lon)
    except Exception as exc:  # pragma: no cover — defensive, mirrors prod best-effort
        logger.warning("deal %s house_id resolution failed (graceful): %s", deal.id, exc)
        db.rollback()
        return None
    if not match:
        return None
    try:
        return int(match[0])
    except (TypeError, ValueError, IndexError):  # pragma: no cover — malformed return
        logger.warning("deal %s house match returned malformed tuple: %r", deal.id, match)
        return None


def _select_analogs_full(
    db: Session,
    deal: DealSample,
    est: SimpleNamespace,
    *,
    target_house_id: int | None = None,
) -> tuple[list[dict[str, Any]], str, bool, bool]:
    """Replicate estimate_quality's analog tier ladder for one deal (#1966).

    Mirrors the Tier0(cohort)→A(1 km)→wide(2 km)→widearea(±25 %) sequence
    EXACTLY, including the MIN_ANALOGS_TIER_0 gate and the <5 / <3 widen
    thresholds and the "only adopt the wider tier when it returns MORE lots"
    guards. ``target_house_id`` defaults to None — same as a fresh estimate
    without a house match, so Tier S(canonical) never fires. When the harness is
    run with ``--resolve-house-id`` (#2002) a resolved id is threaded in and the
    Tier-S ``WHERE house_id_fk = :target_house_id`` selector fires, mirroring prod.

    Returns ``(listings, analog_tier, fallback_used, area_widened)``.
    """
    m = est.m
    cohort_range = m._target_cohort_range(deal.year_built)
    if cohort_range is not None:
        cy_min, cy_max = cohort_range
        listings_tier0, _, analog_tier = m._fetch_analogs(
            db,
            lat=deal.lat,
            lon=deal.lon,
            rooms=deal.rooms,
            area=deal.area_m2,
            radius_m=m.DEFAULT_RADIUS_M,
            full_address=deal.address,
            target_house_id=target_house_id,
            year_built=deal.year_built,
            house_type=deal.house_type,
            total_floors=deal.total_floors,
            cohort_year_min=cy_min,
            cohort_year_max=cy_max,
        )
    else:
        listings_tier0 = []
        analog_tier = "W"

    if len(listings_tier0) >= m.MIN_ANALOGS_TIER_0:
        listings = listings_tier0
        fallback_used = False
    else:
        listings, fallback_used, analog_tier = m._fetch_analogs(
            db,
            lat=deal.lat,
            lon=deal.lon,
            rooms=deal.rooms,
            area=deal.area_m2,
            radius_m=m.DEFAULT_RADIUS_M,
            full_address=deal.address,
            target_house_id=target_house_id,
            year_built=deal.year_built,
            house_type=deal.house_type,
            total_floors=deal.total_floors,
        )
    area_widened = False

    if len(listings) < 5:
        listings_wide, _, analog_tier_wide = m._fetch_analogs(
            db,
            lat=deal.lat,
            lon=deal.lon,
            rooms=deal.rooms,
            area=deal.area_m2,
            radius_m=m.FALLBACK_RADIUS_M,
            full_address=deal.address,
            target_house_id=target_house_id,
            year_built=deal.year_built,
            house_type=deal.house_type,
            total_floors=deal.total_floors,
        )
        if len(listings_wide) > len(listings):
            listings = listings_wide
            fallback_used = True
            analog_tier = analog_tier_wide

    if len(listings) < 3:
        listings_widearea, _, analog_tier_wa = m._fetch_analogs(
            db,
            lat=deal.lat,
            lon=deal.lon,
            rooms=deal.rooms,
            area=deal.area_m2,
            radius_m=m.FALLBACK_RADIUS_M,
            area_tolerance=0.25,
            full_address=deal.address,
            target_house_id=target_house_id,
            year_built=deal.year_built,
            house_type=deal.house_type,
            total_floors=deal.total_floors,
        )
        if len(listings_widearea) > len(listings):
            listings = listings_widearea
            fallback_used = True
            area_widened = True
            analog_tier = analog_tier_wa

    return listings, analog_tier, fallback_used, area_widened


# --------------------------------------------------------------------------- #
# Fixture capture + hermetic replay (#1966 PR 3/3)
#
# ``--dump-fixture`` freezes per-deal RESOLVED inputs to ``_price_from_inputs``
# into a committed JSON file; ``replay_fixture`` replays the spine offline (ZERO
# DB) so a CI test can assert metrics match a frozen baseline. The 3 DB-backed
# callables (asking→sold ratio + quarter-index lookups) hide all I/O, so we
# capture each as an ordered ``[arg -> return]`` call-list and replay it via an
# exact-match stub. Everything else fed to ``_price_from_inputs`` is pure data.
# --------------------------------------------------------------------------- #

FIXTURE_SCHEMA_VERSION = 1


def _sanitize_json(obj: Any) -> Any:
    """Recursively coerce a captured value into a finite, JSON-plain structure.

    - non-finite float (inf / nan) → None (``json.dump(allow_nan=False)`` would
      otherwise raise);
    - Decimal → float; date / datetime → ISO str; tuple → list; dict / list
      recursed; bool / int / str / None pass through; any other type → ``str()``.

    Idempotent on already-plain data, so applying it both at capture time (to the
    recorded call args) and at replay time (to the live args before stub lookup)
    keeps the lookup keys byte-stable across capture and replay.
    """
    if obj is None or isinstance(obj, bool | int | str):
        return obj
    if isinstance(obj, float):
        return obj if math.isfinite(obj) else None
    if isinstance(obj, Decimal):
        f = float(obj)
        return f if math.isfinite(f) else None
    if isinstance(obj, dict):
        return {str(k): _sanitize_json(v) for k, v in obj.items()}
    if isinstance(obj, list | tuple):
        return [_sanitize_json(x) for x in obj]
    if isinstance(obj, date):  # date | datetime (datetime is a date subclass)
        return obj.isoformat()
    return str(obj)


def _coerce_ratio_return(ret: Any) -> tuple[Any, Any]:
    """Recorded ``[ratio, basis]`` → ``(ratio, basis)`` tuple (unpacked at the call site)."""
    return (ret[0], ret[1])


def _coerce_qi_return(ret: Any) -> tuple[Any, Any] | None:
    """Recorded ``[qi, n]`` or ``null`` → ``(qi, n)`` tuple or None."""
    return None if ret is None else (ret[0], ret[1])


def _coerce_qis_return(ret: Any) -> dict[str, float]:
    """Recorded ``{quarter: qi}`` dict → ``dict[str, float]`` (keys forced to str)."""
    return {str(k): v for k, v in dict(ret).items()}


def load_fixture(path: str) -> dict[str, Any]:
    """Load a backtest fixture JSON — transparently handles gzip (``.gz``) files.

    The committed fixture is gzipped (~3 MB raw); a plain ``.json`` path is also
    accepted for ad-hoc runs. Exported for the CI regression-gate test.
    """
    if str(path).endswith(".gz"):
        with gzip.open(path, "rt", encoding="utf-8") as fh:
            return json.loads(fh.read())
    return json.loads(Path(path).read_text(encoding="utf-8"))


def _make_call_stub(
    calls: list[Any], *, label: str, coerce: Callable[[Any], Any]
) -> Callable[[Any], Any]:
    """Build an ORDER-based (FIFO) replay stub from recorded ``[arg, return]`` pairs.

    The stub returns the recorded RETURNS in invocation order and IGNORES the arg
    value. Rationale: the DB-derived ratio / quarter-index is a FROZEN input — when
    estimator logic later shifts e.g. ``median_ppm2``, the regression gate must
    surface that as a clean metric diff vs the baseline, NOT mask it as a lookup
    miss. Matching on the exact computed float arg is also not bit-stable across
    platforms (libm last-ulp jitter), so a Linux-captured fixture would crash when
    replayed off-Linux. The recorded arg is kept (for debuggability) but never
    matched. Each callable fires ≤1×/deal today (so index 0 in practice); the FIFO
    stays correct if a call site ever loops. Calling the stub MORE times than
    recorded raises RuntimeError — control flow diverged from capture.

    ``coerce`` maps each JSON-plain recorded return back to the live callable's
    return type (tuple / dict) so unpacking at the call site behaves identically.
    """
    returns = [coerce(ret) for _arg, ret in calls]
    idx = 0

    def _stub(_arg: Any) -> Any:
        nonlocal idx
        if idx >= len(returns):
            raise RuntimeError(
                f"{label}: replay made call #{idx + 1} but fixture recorded only "
                f"{len(returns)} — control flow diverged from capture"
            )
        ret = returns[idx]
        idx += 1
        return ret

    return _stub


def replay_fixture(fixture: dict[str, Any]) -> dict[str, Any]:
    """Replay a frozen backtest fixture through the full spine — hermetic, ZERO DB.

    For every captured deal record: rebuild the ``GeocodeResult``, build order-based
    (FIFO) stubs for the 3 DB callables from the recorded call-lists, call
    ``_price_from_inputs`` with the frozen kwargs, and rebuild the ``Prediction``
    EXACTLY as ``_predict_full_spine`` does. Runs the harness's own
    ``_compute_full_metrics`` over the replayed predictions plus a city-wide
    headline computed from the stored (priced) deals.

    The returned dict keeps ``expected_sold`` / ``range_coverage`` / ``calibration``
    / ``sharpness`` / ``confidence_order`` / ``headline`` and DROPS the volatile
    ``params`` block. Touches NO DB / network and does NOT consult
    ``settings_at_capture`` — it prices against the live committed
    ``estimator.settings`` defaults (so a settings change is caught as a metric
    drift, not silently honoured). Deterministic: same fixture → identical dict.
    """
    est = _import_estimator_full()
    m = est.m

    deals = fixture.get("deals") or []
    predictions: list[Prediction] = []
    sold_ppm2_all: list[float] = []
    pred_ppm2_all: list[float] = []

    for rec in deals:
        kw = dict(rec["kwargs"])
        sold_ppm2_all.append(float(rec["sold_ppm2"]))
        kw["geo"] = est.GeocodeResult(**kw["geo"])
        kw["ratio_resolver"] = _make_call_stub(
            rec.get("ratio_calls") or [], label="ratio_resolver", coerce=_coerce_ratio_return
        )
        kw["quarter_index_lookup"] = _make_call_stub(
            rec.get("qi_calls") or [], label="quarter_index_lookup", coerce=_coerce_qi_return
        )
        kw["quarter_indexes_lookup"] = _make_call_stub(
            rec.get("qis_calls") or [], label="quarter_indexes_lookup", coerce=_coerce_qis_return
        )

        pr = m._price_from_inputs(**kw)

        es_ppm2 = float(pr.expected_sold_per_m2) if pr.expected_sold_per_m2 is not None else None
        es_price = float(pr.expected_sold_price) if pr.expected_sold_price is not None else None
        r_low = (
            float(pr.expected_sold_range_low) if pr.expected_sold_range_low is not None else None
        )
        r_high = (
            float(pr.expected_sold_range_high) if pr.expected_sold_range_high is not None else None
        )
        prediction = Prediction(
            deal_id=int(rec["deal_id"]),
            rooms=rec["rooms"],
            area_m2=float(rec["area_m2"]),
            sold_ppm2=float(rec["sold_ppm2"]),
            median_ppm2=float(pr.median_ppm2),
            confidence=pr.confidence,
            anchor_tier=pr.anchor_tier,
            expected_sold_ppm2=es_ppm2,
            expected_sold_price=es_price,
            range_low=r_low,
            range_high=r_high,
        )
        predictions.append(prediction)
        pred_ppm2_all.append(prediction.median_ppm2)

    # The fixture stores ONLY priced deals, so n_no_prediction is 0 here.
    metrics = _compute_full_metrics(predictions, n_no_prediction=0)

    deal_median = statistics.median(sold_ppm2_all) if sold_ppm2_all else None
    ask_median = statistics.median(pred_ppm2_all) if pred_ppm2_all else None
    spread_pct: float | None = None
    if deal_median and ask_median and deal_median > 0:
        spread_pct = round(100.0 * (ask_median - deal_median) / deal_median, 2)
    metrics["headline"] = {
        "deal_median_ppm2": deal_median,
        "ask_median_ppm2": ask_median,
        "spread_pct": spread_pct,
    }
    return metrics


def _predict_full_spine(
    db: Session,
    deal: DealSample,
    est: SimpleNamespace,
    *,
    capture: list[dict[str, Any]] | None = None,
    target_house_id: int | None = None,
    resolution: _HouseIdResolution | None = None,
) -> Prediction | None:
    """Predict one deal through the FULL deterministic spine (#1966).

    Selects analogs via the replicated tier ladder, pre-fetches the spine inputs
    (anchor comps, ДКП corridor, house IMV anchor) and injects the DB callables,
    then calls ``estimator._price_from_inputs`` with parameters that mirror
    ``estimate_quality`` for a sold unit whose repair state is unknown and with
    the network valuation layers excluded (imv_eval=None, yandex/cian absent).

    Returns a Prediction, or None when the spine cannot price the deal
    (median<=0). Prod parity (#1966): there is NO analog-count skip — a positive
    median surfaces an estimate (at low confidence for thin samples), matching
    estimate_quality which never hard-drops a priced result.

    When ``capture`` is a list, every deal that yields a prediction also appends a
    fully JSON-plain replay record (frozen kwargs + the 3 callables' recorded
    ``[arg -> return]`` call-lists) — see ``--dump-fixture`` / ``replay_fixture``.

    ``target_house_id`` (#2002) is None by default — the historical harness
    behaviour, where Tier-S(canonical) and the house Avito-IMV anchor never fire.
    When ``--resolve-house-id`` is on the caller resolves it and threads it here so
    those paths become measurable; ``resolution`` (when provided) is the coverage
    accumulator whose ``imv_reachable`` counter this function bumps once it knows
    whether the house IMV anchor actually resolved a row for this deal.
    """
    m = est.m
    settings = est.settings

    listings, analog_tier, fallback_used, area_widened = _select_analogs_full(
        db, deal, est, target_house_id=target_house_id
    )

    # ── Pre-fetch the spine inputs (same calls estimate_quality hoists) ───────
    # (oblast C2 parity): estimate_quality now resolves a target city and passes
    # it to _fetch_dkp_corridor (estimator.py:3210-3220) — deals.city is
    # populated oblast-wide (368 cities), and an unscoped corridor lets
    # same-named streets in OTHER region-66 towns (e.g. "Ленина" exists in many)
    # contaminate the corridor. Mirror that resolve+pass here UNCONDITIONALLY
    # (independent of --city, which only scopes WHICH deals get sampled) so this
    # harness measures the SAME corridor prod actually computes today, not the
    # pre-C2 unscoped behaviour — else the backtest validates stale semantics.
    target_city = m._resolve_target_city(deal.address)
    dkp_raw = m._fetch_dkp_corridor(
        db, address=deal.address, rooms=deal.rooms, area=deal.area_m2, city=target_city
    )
    # #1966 prod parity: same-building anchor pre-fetch is GATED exactly like
    # estimate_quality — no-area / no-address → ([], None) instead of an
    # unconditional fetch.
    if deal.area_m2 and deal.address:
        # NB(#2002): kept at None deliberately — the deliverable scopes the
        # resolved-id threading to the Tier-S analog ladder + the house IMV anchor.
        # Prod ALSO resolves this same-building anchor (estimator.py:2782); flip to
        # `target_house_id=target_house_id` here if/when full prod parity is wanted.
        anchor_comps, anchor_tier = m._fetch_anchor_comps(
            db,
            address=deal.address,
            target_house_id=None,
            lat=deal.lat,
            lon=deal.lon,
            rooms=deal.rooms,
            area=deal.area_m2,
        )
    else:
        anchor_comps, anchor_tier = [], None
    imv_anchor = m._fetch_house_imv_anchor(
        db, target_house_id=target_house_id, rooms=deal.rooms, area=deal.area_m2
    )
    # #2002: count IMV reach — a resolved house whose house_imv_evaluations row
    # actually fires the anchor (band-compatible match → non-None). Counted before
    # the median<=0 skip below so reach reflects row availability, not pricing.
    if resolution is not None and target_house_id is not None and imv_anchor is not None:
        resolution.imv_reachable += 1

    # Synthetic geo = the deal's own coords + address, confidence "exact" — the
    # faithful analog of estimate_quality's client-coords fast path. Real deal
    # addresses carry a house number, so _geocode_is_coarse → False (no coarse
    # downgrade); if an address lacks a number that deal degrades to "low",
    # exactly as the production gate would.
    geo = est.GeocodeResult(
        lat=deal.lat,
        lon=deal.lon,
        full_address=deal.address or "",
        provider="cache",
        confidence="exact",
    )

    # In capture mode each closure RECORDS its (single-arg -> return) call so the
    # offline replay can rebuild a DB-free stub. The real value is still returned.
    ratio_calls: list[Any] = []
    qi_calls: list[Any] = []
    qis_calls: list[Any] = []

    def _ratio_resolver(appm2: float | None) -> tuple[float | None, str | None]:
        res = m._get_asking_sold_ratio(db, deal.rooms, anchor_ppm2=appm2)
        if capture is not None:
            ratio_calls.append([_sanitize_json(appm2), _sanitize_json(list(res))])
        return res

    def _qi_lookup(q: str) -> tuple[float, int] | None:
        res = m._lookup_quarter_index(
            db,
            quarter_cad_number=q,
            min_n_deals=settings.estimate_quarter_index_min_n_deals,
        )
        if capture is not None:
            ret = _sanitize_json(list(res) if res is not None else None)
            qi_calls.append([_sanitize_json(q), ret])
        return res

    def _qis_lookup(qs: list[str]) -> dict[str, float]:
        res = m._lookup_quarter_indexes(
            db,
            quarter_cad_numbers=qs,
            min_n_deals=settings.estimate_quarter_index_min_n_deals,
        )
        if capture is not None:
            qis_calls.append([_sanitize_json(list(qs)), _sanitize_json(dict(res))])
        return res

    pr = m._price_from_inputs(
        listings=listings,
        area_m2=deal.area_m2,
        rooms=deal.rooms,
        repair_state=None,
        floor=deal.floor,
        total_floors=deal.total_floors,
        target_year=deal.year_built,
        analog_tier=analog_tier,
        fallback_used=fallback_used,
        area_widened=area_widened,
        anchor_comps=anchor_comps,
        anchor_tier_fetched=anchor_tier,
        dkp_raw=dkp_raw,
        imv_anchor=imv_anchor,
        imv_eval=None,
        yandex_val_present=False,
        cian_val_present=False,
        ratio_resolver=_ratio_resolver,
        quarter_index_lookup=_qi_lookup,
        quarter_indexes_lookup=_qis_lookup,
        target_house_cadnum=None,
        dadata_coarse=False,
        geo=geo,
        dadata_qc_geo=None,
    )

    # #1966 prod parity: skip ONLY when the spine could not price the deal
    # (median<=0). NO analog-count floor — estimate_quality surfaces any positive
    # estimate at low confidence. MIN_CANDIDATES still gates the asking-core path.
    if pr.median_price <= 0:
        return None

    es_ppm2 = float(pr.expected_sold_per_m2) if pr.expected_sold_per_m2 is not None else None
    es_price = float(pr.expected_sold_price) if pr.expected_sold_price is not None else None
    r_low = float(pr.expected_sold_range_low) if pr.expected_sold_range_low is not None else None
    r_high = float(pr.expected_sold_range_high) if pr.expected_sold_range_high is not None else None

    if capture is not None:
        capture.append(
            {
                "deal_id": deal.id,
                "sold_ppm2": float(deal.sold_ppm2),
                "area_m2": float(deal.area_m2),
                "rooms": deal.rooms,
                "deal_date": str(deal.deal_date),
                "kwargs": {
                    "listings": _sanitize_json(listings),
                    "area_m2": float(deal.area_m2),
                    "rooms": deal.rooms,
                    "repair_state": None,
                    "floor": deal.floor,
                    "total_floors": deal.total_floors,
                    "target_year": deal.year_built,
                    "analog_tier": analog_tier,
                    "fallback_used": fallback_used,
                    "area_widened": area_widened,
                    "anchor_comps": _sanitize_json(anchor_comps),
                    "anchor_tier_fetched": anchor_tier,
                    "dkp_raw": _sanitize_json(dkp_raw),
                    "imv_anchor": _sanitize_json(imv_anchor),
                    "imv_eval": None,
                    "yandex_val_present": False,
                    "cian_val_present": False,
                    "target_house_cadnum": None,
                    "dadata_coarse": False,
                    "geo": _sanitize_json(dataclasses.asdict(geo)),
                    "dadata_qc_geo": None,
                },
                "ratio_calls": ratio_calls,
                "qi_calls": qi_calls,
                "qis_calls": qis_calls,
            }
        )

    return Prediction(
        deal_id=deal.id,
        rooms=deal.rooms,
        area_m2=deal.area_m2,
        sold_ppm2=deal.sold_ppm2,
        median_ppm2=float(pr.median_ppm2),
        confidence=pr.confidence,
        anchor_tier=pr.anchor_tier,
        expected_sold_ppm2=es_ppm2,
        expected_sold_price=es_price,
        range_low=r_low,
        range_high=r_high,
    )


def _attach_correction(
    metrics: dict[str, Any],
    matched_rows: list[tuple[float, float, int]],
    matched_ids: list[int],
    *,
    holdout_split: bool,
) -> None:
    """Derive the per-rooms ratio, apply it, and attach the CORRECTED block.

    Mutates ``metrics`` in place, adding ``ratios`` (bucket→float),
    ``ratios_meta`` (diagnostics), and ``corrected`` (a full _compute_metrics
    dict for ``pred_sold = pred_ask * ratio``).

    Two modes:
      - default (in-sample): ratios derived on ALL matched rows, evaluated on
        ALL matched rows → near-zero bias by construction (mechanism proof).
      - ``holdout_split``: ratios derived on EVEN-id deals, evaluated on the
        ODD-id half → an honest out-of-sample number. Split is by deal-id
        parity (deterministic, reproducible — no RNG).

    Pure aside from reusing the no-DB helpers; safe on an empty sample.
    """
    if holdout_split:
        paired = list(zip(matched_rows, matched_ids, strict=True))
        fit_rows = [r for r, did in paired if did % 2 == 0]
        eval_rows = [r for r, did in paired if did % 2 == 1]
    else:
        fit_rows = matched_rows
        eval_rows = matched_rows

    ratios, ratios_meta = _derive_room_ratios(fit_rows)
    ratios_meta["holdout_split"] = holdout_split
    ratios_meta["n_fit"] = len(fit_rows)
    ratios_meta["n_eval"] = len(eval_rows)

    corrected_rows = _apply_ratios(eval_rows, ratios)
    corrected = _compute_metrics(corrected_rows)

    # JSON-friendly: stringify int bucket keys, round ratios for readability.
    metrics["ratios"] = ratios
    metrics["ratios_meta"] = ratios_meta
    metrics["corrected"] = corrected


def run_backtest(
    db: Session,
    *,
    sample: int,
    since: str,
    radius: int,
    rooms_tolerance: int,
    holdout_split: bool = False,
    city: str | None = None,
) -> dict[str, Any]:
    """Drive the full read-only backtest and return a metrics dict.

    Steps: load sample → predict per deal (reusing estimator funcs) → collect
    (pred, sold, rooms) rows + no-analog counts → `_compute_metrics` (ASKING
    block) → `_derive_room_ratios` + `_apply_ratios` → `_compute_metrics`
    again (CORRECTED block) + a city-wide headline spread. No writes.

    The CORRECTED block multiplies each asking-median prediction by a per-rooms
    asking→sold ratio derived from the SAME sample (issue #648 Stage 1). With
    ``holdout_split=False`` (default) that ratio is fit and evaluated in-sample,
    so its bias is near-zero by construction — it proves the MECHANISM, not
    out-of-sample accuracy (see _derive_room_ratios). Pass ``holdout_split=True``
    to fit on even-id deals and evaluate on the odd-id half for an honest number.

    ``city`` (oblast D, default None) scopes the deal sample to one
    ``deals.city`` value — see ``_load_sample``. Default None is unscoped
    (byte-identical to the pre-oblast-D behaviour).
    """
    deals = _load_sample(db, sample=sample, since=since, city=city)
    logger.info("loaded sample: %d ДКП deals (since=%s, city=%s)", len(deals), since, city)

    matched_rows: list[tuple[float, float, int]] = []
    matched_ids: list[int] = []
    n_no_analogs = 0
    per_rooms_no_analogs: dict[int, int] = {b: 0 for b in ROOM_BUCKETS}

    # For the city-wide headline: median of all sampled SOLD ppm², and median
    # of all per-deal predicted ASKING ppm² (matched deals only).
    sold_ppm2_all: list[float] = [d.sold_ppm2 for d in deals]
    pred_ppm2_all: list[float] = []

    for i, deal in enumerate(deals, start=1):
        pred = _predict_for_deal(db, deal, radius=radius, rooms_tolerance=rooms_tolerance)
        if pred is None:
            n_no_analogs += 1
            per_rooms_no_analogs[_bucketize_rooms(deal.rooms)] += 1
        else:
            matched_rows.append((pred, deal.sold_ppm2, deal.rooms))
            matched_ids.append(deal.id)
            pred_ppm2_all.append(pred)

        if i % 50 == 0:
            logger.info(
                "progress %d/%d (matched=%d, no_analogs=%d)",
                i,
                len(deals),
                len(matched_rows),
                n_no_analogs,
            )

    metrics = _compute_metrics(
        matched_rows,
        n_no_analogs=n_no_analogs,
        per_rooms_no_analogs=per_rooms_no_analogs,
    )

    _attach_correction(metrics, matched_rows, matched_ids, holdout_split=holdout_split)

    deal_median = statistics.median(sold_ppm2_all) if sold_ppm2_all else None
    ask_median = statistics.median(pred_ppm2_all) if pred_ppm2_all else None
    spread_pct: float | None = None
    if deal_median and ask_median and deal_median > 0:
        spread_pct = round(100.0 * (ask_median - deal_median) / deal_median, 2)

    metrics["headline"] = {
        "deal_median_ppm2": deal_median,
        "ask_median_ppm2": ask_median,
        "spread_pct": spread_pct,
    }
    metrics["params"] = {
        "engine": "asking-core",
        "sample_requested": sample,
        "sample_loaded": len(deals),
        "since": since,
        "radius_m": radius,
        "rooms_tolerance": rooms_tolerance,
        "n_matched": len(matched_rows),
        "n_no_analogs": n_no_analogs,
        "holdout_split": holdout_split,
        "city": city,
    }
    return metrics


def run_backtest_full(
    db: Session,
    *,
    sample: int,
    since: str,
    dump_fixture: str | None = None,
    resolve_house_id: bool = False,
    city: str | None = None,
) -> dict[str, Any]:
    """Drive the FULL-spine read-only backtest and return a metrics dict (#1966).

    Per deal: load sample → ``_predict_full_spine`` (replicate the analog tier
    ladder + pre-fetch spine inputs → ``_price_from_inputs``) → collect Prediction
    records + no-prediction counts → ``_compute_full_metrics`` (expected_sold
    error overall/per-rooms/per-segment + range-coverage + calibration +
    sharpness) + a city-wide asking-vs-deal headline spread. No writes to the DB.

    ``--radius`` / ``--rooms-tolerance`` / ``--holdout-split`` do NOT apply here —
    the tier ladder uses the estimator's OWN constants (DEFAULT_RADIUS_M /
    FALLBACK_RADIUS_M) and there is no per-rooms correction block (the spine
    already emits expected_sold via the asking→sold ratio).

    When ``dump_fixture`` is a path, every priced deal's RESOLVED spine inputs +
    the 3 DB callables' recorded call-lists are frozen into a committed JSON file
    (schema_version=1) so ``replay_fixture`` can re-score offline against a frozen
    baseline (the CI regression gate). This is the only path that writes a file.

    When ``resolve_house_id`` is True (#2002, ``--resolve-house-id``) each deal's
    canonical house_id is resolved per ``match_house_readonly`` and threaded into
    the Tier-S analog ladder + the house Avito-IMV anchor, mirroring prod, and a
    ``house_id_resolution`` coverage block (resolved / total / imv_reachable) is
    attached to the returned metrics. Default False → byte-identical prior output.

    ``city`` (oblast D, default None) scopes the deal sample to one
    ``deals.city`` value — see ``_load_sample``. Default None is unscoped
    (byte-identical to the pre-oblast-D behaviour). Independently of this flag,
    ``_predict_full_spine`` ALWAYS resolves + passes a per-deal target city to
    ``_fetch_dkp_corridor`` (oblast C2 parity fix) — see its docstring.
    """
    est = _import_estimator_full()
    deals = _load_sample(db, sample=sample, since=since, city=city)
    logger.info(
        "loaded sample: %d ДКП deals (since=%s, city=%s) [full spine]", len(deals), since, city
    )

    predictions: list[Prediction] = []
    n_no_prediction = 0
    per_rooms_no_prediction: dict[int, int] = {b: 0 for b in ROOM_BUCKETS}

    sold_ppm2_all: list[float] = [d.sold_ppm2 for d in deals]
    pred_ppm2_all: list[float] = []  # asking headline median_ppm2 (priced deals)
    capture: list[dict[str, Any]] | None = [] if dump_fixture else None
    resolution: _HouseIdResolution | None = _HouseIdResolution() if resolve_house_id else None

    for i, deal in enumerate(deals, start=1):
        target_house_id: int | None = None
        if resolution is not None:
            resolution.total += 1
            target_house_id = _resolve_house_id_for_deal(db, deal)
            if target_house_id is not None:
                resolution.resolved += 1
        try:
            pr = _predict_full_spine(
                db,
                deal,
                est,
                capture=capture,
                target_house_id=target_house_id,
                resolution=resolution,
            )
        except Exception as exc:
            # Read-only: a failed SELECT can poison the tx → rollback so the next
            # deal's queries run clean. Skip this deal (counts as no-prediction).
            logger.warning("deal %s spine failed (graceful, skipped): %s", deal.id, exc)
            db.rollback()
            pr = None

        if pr is None:
            n_no_prediction += 1
            per_rooms_no_prediction[_bucketize_rooms(deal.rooms)] += 1
        else:
            predictions.append(pr)
            pred_ppm2_all.append(pr.median_ppm2)

        if i % 50 == 0:
            logger.info(
                "progress %d/%d (priced=%d, no_pred=%d)",
                i,
                len(deals),
                len(predictions),
                n_no_prediction,
            )

    metrics = _compute_full_metrics(
        predictions,
        n_no_prediction=n_no_prediction,
        per_rooms_no_prediction=per_rooms_no_prediction,
    )

    deal_median = statistics.median(sold_ppm2_all) if sold_ppm2_all else None
    ask_median = statistics.median(pred_ppm2_all) if pred_ppm2_all else None
    spread_pct: float | None = None
    if deal_median and ask_median and deal_median > 0:
        spread_pct = round(100.0 * (ask_median - deal_median) / deal_median, 2)

    metrics["headline"] = {
        "deal_median_ppm2": deal_median,
        "ask_median_ppm2": ask_median,
        "spread_pct": spread_pct,
    }
    metrics["params"] = {
        "engine": "full",
        "sample_requested": sample,
        "sample_loaded": len(deals),
        "since": since,
        "n_matched": len(predictions),
        "n_no_prediction": n_no_prediction,
        "price_segments_ppm2": [list(seg) for seg in _price_segments()],
        "city": city,
    }

    # #2002: house_id resolution coverage — the key Tier-S + IMV reach number.
    # Only emitted under --resolve-house-id so the default output stays unchanged.
    if resolution is not None:
        metrics["house_id_resolution"] = resolution.as_json()
        logger.info(
            "house_id resolution: resolved=%d/%d imv_reachable=%d",
            resolution.resolved,
            resolution.total,
            resolution.imv_reachable,
        )

    if dump_fixture is not None and capture is not None:
        _write_fixture(dump_fixture, capture=capture, since=since, est=est)

    return metrics


def _write_fixture(
    path: str, *, capture: list[dict[str, Any]], since: str, est: SimpleNamespace
) -> None:
    """Freeze captured per-deal replay records into a committed JSON fixture.

    ``settings_at_capture`` records every ``estimate_*`` Settings field as an
    informational snapshot (NOT consulted by ``replay_fixture``). A final
    recursive ``_sanitize_json`` pass guarantees the whole document is finite +
    JSON-plain before ``json.dump(allow_nan=False)``.
    """
    settings_at_capture = {
        name: _sanitize_json(getattr(est.settings, name))
        for name in sorted(type(est.settings).model_fields)
        if name.startswith("estimate_")
    }
    fixture = {
        "schema_version": FIXTURE_SCHEMA_VERSION,
        "engine": "full",
        "since": since,
        "n_deals": len(capture),
        "settings_at_capture": settings_at_capture,
        "deals": capture,
    }
    fixture = _sanitize_json(fixture)
    out_path = Path(path)
    if out_path.parent != Path():
        out_path.parent.mkdir(parents=True, exist_ok=True)
    with out_path.open("w", encoding="utf-8") as fh:
        json.dump(fixture, fh, ensure_ascii=False, indent=2, allow_nan=False)
        fh.write("\n")
    logger.info("dumped fixture: %s (%d deals)", out_path, len(capture))
    print(f"dumped fixture: {out_path} ({len(capture)} deals)")


# --------------------------------------------------------------------------- #
# Entry point
# --------------------------------------------------------------------------- #


def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
    """argparse setup, factored out for testability."""
    p = argparse.ArgumentParser(
        description=(
            "READ-ONLY backtest of the estimator against rosreestr ДКП sold "
            "prices. Default engine 'full' runs the deterministic pricing spine "
            "(_price_from_inputs); 'asking-core' is the legacy median+IQR core "
            "(issues #648 / #1966)."
        ),
    )
    p.add_argument(
        "--engine",
        choices=("full", "asking-core"),
        default="full",
        help="Prediction engine: 'full' (default) = the deterministic pricing "
        "spine via _price_from_inputs (expected_sold + coverage/calibration/"
        "segment metrics); 'asking-core' = legacy asking-median+IQR core with "
        "the per-rooms asking→sold correction block.",
    )
    p.add_argument(
        "--sample",
        type=int,
        default=300,
        help="Held-out ДКП deals to test (default 300). Large samples are slow "
        "— one PostGIS subquery runs per deal.",
    )
    p.add_argument(
        "--since",
        default="2025-06-01",
        help="Only deals with deal_date >= this ISO date (default 2025-06-01).",
    )
    p.add_argument(
        "--city",
        default=None,
        metavar="NAME",
        help="Oblast D: scope the held-out ДКП deal sample to ONE deals.city value "
        "(region 66 is oblast-wide — deals.city covers 368 cities, 96,974 deals). "
        "Pass the EXACT canonical Russian name as stored in deals.city, e.g. "
        "'Нижний Тагил' or 'Каменск-Уральский' — NOT a slug/transliteration (run "
        "`SELECT DISTINCT city FROM deals` to check the exact spelling). Also "
        "sources the deal-sample PPM2 sanity band from deal_city_price_bands for "
        "that city (falls back to the global PPM2_MIN/PPM2_MAX when the city has "
        "no band row, e.g. Екатеринбург — intentionally excluded from that table). "
        "Default None → unscoped sample, byte-identical to the pre-oblast-D SQL.",
    )
    p.add_argument(
        "--radius",
        type=int,
        default=1000,
        help="Analog search radius in metres (default 1000).",
    )
    p.add_argument(
        "--rooms-tolerance",
        type=int,
        default=0,
        help="± room count tolerance for analogs (default 0 = exact match).",
    )
    p.add_argument(
        "--holdout-split",
        action="store_true",
        help="Fit the per-rooms asking→sold ratio on EVEN-id deals and evaluate "
        "the CORRECTED block on the ODD-id half (deterministic out-of-sample "
        "split, no RNG). Default off → ratio is fit AND evaluated in-sample, so "
        "the corrected bias is near-zero by construction (mechanism proof only).",
    )
    p.add_argument(
        "--json",
        action="store_true",
        help="Emit machine-readable JSON instead of the text table.",
    )
    p.add_argument(
        "--resolve-house-id",
        action="store_true",
        help="FULL engine only: resolve each deal's canonical house_id via "
        "match_house_readonly(address, lat, lon) and thread it into the analog "
        "tier ladder + house Avito-IMV anchor, mirroring prod estimate_quality "
        "(estimator.py:2554-2569). Lets the Tier-S 'same-building' selector and "
        "_fetch_house_imv_anchor actually fire (they key on house_id, otherwise "
        "None → never resolved). Adds a 'house_id_resolution' coverage block to "
        "the JSON output. Default OFF → byte-identical to the prior behaviour, so "
        "the frozen regression gate is untouched.",
    )
    p.add_argument(
        "--calibrate-segments",
        action="store_true",
        help="FULL engine only (#2255): after the run, print a per-price-segment "
        "calibration table — segment → n / bias%% / raw multiplier 1/(1+bias) / "
        "shrinkage λ=n/(n+40) / SUGGESTED m (shrunk toward 1.0). PRINT-ONLY: it "
        "proposes estimate_segment_multipliers, it does NOT apply them or touch "
        "the baseline. Run with --resolve-house-id for prod-parity biases.",
    )
    # #1966 PR 3/3 — fixture capture + hermetic replay. --dump-fixture (DB run,
    # full engine) and --from-fixture (NO DB) are mutually exclusive modes.
    fixture_mode = p.add_mutually_exclusive_group()
    fixture_mode.add_argument(
        "--dump-fixture",
        metavar="PATH",
        default=None,
        help="FULL engine only: freeze each priced deal's resolved _price_from_"
        "inputs inputs + the 3 DB callables' recorded calls into a committed JSON "
        "fixture at PATH, so replay_fixture can re-score offline (CI gate).",
    )
    fixture_mode.add_argument(
        "--from-fixture",
        metavar="PATH",
        default=None,
        help="Hermetic replay: load the frozen fixture at PATH (gzip-transparent — "
        "a .gz path is decompressed on the fly), re-run the spine offline (ZERO DB) "
        "via replay_fixture, and print the metrics JSON. No DB connection is opened.",
    )
    p.add_argument(
        "--update-baseline",
        metavar="OUT",
        default=None,
        help="With --from-fixture: also write the replayed metrics to OUT "
        "(sorted-keys JSON + trailing newline) — regenerates the frozen baseline.",
    )
    return p.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
    """CLI entry point. Returns the count of matched (predicted) / replayed deals."""
    args = _parse_args(argv)

    # ── Hermetic replay path (#1966 PR 3/3) — ZERO DB, no SessionLocal opened. ──
    if args.from_fixture:
        fixture = load_fixture(args.from_fixture)
        metrics = replay_fixture(fixture)
        print(json.dumps(metrics, ensure_ascii=False, indent=2, sort_keys=True))
        if args.update_baseline:
            out = Path(args.update_baseline)
            out.write_text(
                json.dumps(metrics, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
                encoding="utf-8",
            )
            logger.info("wrote baseline: %s", out)
        return len(fixture.get("deals") or [])

    if args.update_baseline:
        raise SystemExit("--update-baseline requires --from-fixture")
    if args.dump_fixture and args.engine != "full":
        raise SystemExit("--dump-fixture is only supported with --engine full")
    if args.resolve_house_id and args.engine != "full":
        raise SystemExit("--resolve-house-id is only supported with --engine full")
    if args.calibrate_segments and args.engine != "full":
        raise SystemExit("--calibrate-segments is only supported with --engine full")

    logger.info(
        "backtest start: engine=%s sample=%d since=%s radius=%dm "
        "rooms_tolerance=%d holdout_split=%s dump_fixture=%s resolve_house_id=%s city=%s",
        args.engine,
        args.sample,
        args.since,
        args.radius,
        args.rooms_tolerance,
        args.holdout_split,
        args.dump_fixture,
        args.resolve_house_id,
        args.city,
    )

    db = _session()
    try:
        if args.engine == "full":
            metrics = run_backtest_full(
                db,
                sample=args.sample,
                since=args.since,
                dump_fixture=args.dump_fixture,
                resolve_house_id=args.resolve_house_id,
                city=args.city,
            )
        else:
            metrics = run_backtest(
                db,
                sample=args.sample,
                since=args.since,
                radius=args.radius,
                rooms_tolerance=args.rooms_tolerance,
                holdout_split=args.holdout_split,
                city=args.city,
            )
    finally:
        db.close()

    if args.json:
        print(json.dumps(metrics, ensure_ascii=False, indent=2, default=str))
    elif args.engine == "full":
        print(_render_full_table(metrics))
    else:
        print(_render_table(metrics, metrics["headline"]))

    # #2255 print-only: after the run, propose per-segment multipliers from the
    # expected_sold per-segment bias. Renders even in --json mode (to stderr-free
    # stdout tail) so the operator sees the proposal alongside machine output.
    if args.calibrate_segments:
        per_segment = metrics["expected_sold"]["per_segment"]
        print("\n" + "\n".join(_render_calibrate_segments_block(per_segment)))

    return int(metrics["params"]["n_matched"])


if __name__ == "__main__":  # pragma: no cover
    raise SystemExit(0 if main() >= 0 else 1)
