#992: attach response_model=AnalyzeResponse to POST /{cad_num}/analyze. Model
uses extra="allow" so any result_payload key not explicitly modeled is preserved
in the 200 response (no silent drops that would break Site Finder), and ALL
fields are Optional so the #93 202 fetch-stub serializes without a 500.
#994: add GET /{cad_num}/runs (light summary list, empty 200 not 404) and
GET /runs/{run_id} (full row incl result, 404 if missing), backed by
list_runs_for/get_run in analysis_runs.repository (psycopg v3 CAST(:x AS type)).
Routes ordered before /{parcel_id} so /runs/{run_id} is not shadowed.
Closes#992. Closes#994. Refs #961.
NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён.
Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
Add nullable lat/lon (EPSG:4326, 6 dp) to /analyze competitors[] and
pipeline_24mo.top_objects[] so the frontend can plot Leaflet markers. Coords
come from domrf_kn_objects.latitude/longitude (same source as distance_m).
Purely additive: no existing field/shape changed. Frontend map layers follow
in a separate PR. Part of EPIC #958 (958-B4).
build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task,
not inline on the sync /analyze endpoint.
- repository: latest_run_for gains keyword-only schema_version (default None keeps
v_analysis_runs_latest behavior, backward-compat); when given, reads base
analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches
the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top
(index-served via 127's (cad_num, created_at DESC)).
- new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0,
runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via
persist_analysis_run. Graceful: no base run / compute error → logger + return None
(worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include.
Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district)
→ orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass;
code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized,
no IntegrityError). Refs #994#961.
deficit_index pins to -1.0 for every ЕКБ segment (12mo demand flow vs multi-year
supply stock → log-ratio clamps) → zero discriminating power, though the oversupply
is partly real. Add MOI (gross competing supply / demand_per_mo), the real-estate
absorption standard, as an additive non-saturating companion that DISCRIMINATES
(Уралмаш 42mo … Чермет 109mo) where deficit cannot. deficit_index math kept exactly
as-is (honest absolute: -1 = genuinely oversupplied); docstrings clarify -1 is common
and MOI is the discriminating companion (no recalibration). _gross_supply extract-method
(single source of truth; _project_supply behavior byte-identical, code-review-verified).
Surface MOI in §22 future_market (passthrough) + exec_summary key_numbers/verdict.
Guards: no demand → None, no supply → 0. Prod: MOI varies 42→109mo, deficit stays -1.
Discrimination test pins MOI separating two segments both at deficit -1. Refs #952.
/analyze passes the official ЕКБ admin district (ekb_districts polygon, e.g.
'Кировский'), but objective_lots/corpus_room_month store informal micro-districts
('Втузгородок','ЖБИ') -> admin name matched 0 rows -> silent empty forecast.
Add resolve_objective_districts() (site_finder/district_resolver.py) mapping an
admin name to its clean micros via ekb_district_alias (note IS NULL), with
None -> EKB-wide fallback and raw-micro pass-through. Wire into the objective_lots
district filters of market_metrics (§9.2 stock+sales), supply_layers L1 (§9.3),
and sales_series Sources A+B (crm shares the micro vocab, prod-verified),
switching the scalar filter to psycopg3-safe = ANY(CAST(:districts AS text[])).
supply_layers L2/L3 keep the admin name (domrf_kn_objects.district_name is admin vocab).
Prod: Кировский/Ленинский/Орджоникидзевский obj_count 0 -> 32/64/31.
Tests mutation-verified non-vacuous. 192 module tests pass; ruff clean. Refs #969#949.
_KEY_RATE_MARKET_SPREAD_PP was a 0.0 placeholder, so §7.9 affordability payments
used the bare CBR key_rate (~14.5%), understating borrower cost and OVERSTATING
affordability. Calibrate to 4.5pp from the prod anchor (macro_indicator
mortgage_rate_primary_domrf 19.125% @ 2026-04-19 vs key_rate ~14.5% -> observed
~4.6pp, rounded conservatively; inside the typical RF 3-5pp band), so
rate_used = key_rate + 4.5 ~= 19% matches the directly-observed market primary
mortgage rate. Makes affordability LESS optimistic / more accurate. Docs + tests
updated to the symbolic spread (new regression anchor pins spread==4.5 and
key_rate 14.5 -> ~19.0); rate_kind/graceful-fallback semantics unchanged.
Forecasting suite 841 passed; ruff clean.
Backtest (OOS directional hit-rate): single-best-lag compute_rate_sensitivity
is directionally noise (0.148 Source B EKB-wide, lag-unstable); the Almon
distributed-lag estimator (compute_district_rate_regression) is strictly less
noisy on every tier (0.407 Source B / 0.60 survivorship-free Source A,
lag-stable). Add a thin adapter compute_rate_regime_sensitivity mapping
DistributedLagFit onto the existing RateSensitivity contract (beta=long-run
sum-beta, confidence regression->medium / fallback->low, district=None->low and
no call) and repoint the three consumers (demand_normalization, product_scoring,
demand_supply_forecast). Magnitude bounded by the existing [0.5,1.2] clamp.
Reversible; compute_rate_sensitivity kept for the backtest. Consumer tests
repointed to the real Almon path (mutation-verified genuine) + adapter unit
tests + end-to-end fallback degradation. Forecasting suite 840 passed; ruff clean.
weighted_avg_velocity was a naive mean despite the name — a 500-flat ЖК weighed
the same as a 20-flat one. Now count-weighted by flats_total (sql.md AVG
principle): Σ(velocity*flats_total)/Σ(flats_total). Competitors with unknown
flats_total are excluded from weights; if sizes are unknown for ALL, graceful
fallback to the simple mean (den>0 guard). Field name + API contract UNCHANGED
(zero consumer ripple — traced: only CompetitorsSummary, no frontend ref).
Tests: equal sizes → weighted==naive (existing 6.0 stays); NEW test with
500-flat@40 + 20-flat@2 → 38.54 (not naive 21.0), proving the weighting.
Found by read-only services audit.
- recommendation._usp_from_deficits: skip di<=0 so «стройте его» is never emitted
for OVERSUPPLIED formats; all-surplus top-K → [] (no white-space niches).
Aligns with product_scoring._count_positive_usp (di>0). Was: «Дефицит формата
X — стройте его» for a surplus format, reaching PDF/Excel USP-ниши.
- report_assembler._domrf_coverage: drop ambiguous >1.0 percent-guess; normalize
per-branch (analyze pct /100, supply_layers fraction as-is). Sub-1% coverage
(0.8%) no longer read as 80% → no inflated confidence in the near-zero-coverage
case §15 flags. tests for both + end-to-end no-inflation. 241+148 pass.
REAL code-level cause (prod-confirmed, 2 parcels: "Geometry type (Polygon) does
not match column type (MultiPolygon)") of single-contour parcels stuck "fetching"
— complements migrations 129/130 (which fixed the cad_parcels NOT NULL chain).
F1: _save_parcel/_save_quarter now ST_Multi() the geom so single-contour Polygons
fit the MultiPolygon columns (migr 93/58), mirroring the bulk_harvest paths.
F2: _save_building rewritten to the wide cad_buildings schema (migr 92):
quarter_cad_number (was quarter_cad_num → UndefinedColumn), _safe_int(floors)
(range strings → NULL), source='nspd' NOT NULL.
F3: upsert_features wraps each feature in begin_nested() SAVEPOINT so one bad
feature no longer aborts the whole quarter snapshot (backend.md rule; mirrors
grid-walk). New tests/workers/test_nspd_geo.py + savepoint tests. 89 passed.
FOLLOW-UP (Leha: not a bug, follow-up).
get_macro_series / get_latest_macro defaulted region='rf', but CBR mortgage_*
series live under 'sverdl' in macro_indicator (key_rate under 'rf'). Only caller
passes region explicitly so no active leak, but a future caller omitting region
would silently get None — latent footgun. region now str|None; when None →
_canonical_region (mortgage set sourced from _CBR_SERIES_MAP, single source of
truth). Explicit region always wins → existing callers unchanged. Region-binding
tests added. 159 tests pass.
Documented (NOT code-fixed — upstream data-quality): mortgage_debt /
mortgage_overdue stay empty in macro_indicator because cbr_mortgage_series.period
holds value-like garbage (e.g. '10054588.0') for the Debt series — corruption
from the upstream CBR-XLSX scraper that built domrf.db (imported verbatim by
44_import_anton_db.py), NOT an ingest bug. 123_macro_indicator.sql correctly
skips unparseable periods. Needs a separate scraper re-ingest (idempotent
backfill once period is valid). Refs #945
REOPENED 951-B §9.6.
PART A: fix look-ahead leakage in backtest_rate_sensitivity --detrend. The
ln(units) trend was fit over train+test then split, so test data shaped the
detrend and inflated the OOS hit-rate. _detrend_log now takes fit_n; backtest_tier
fits the trend on TRAIN months only (same split evaluate_oos uses) and projects
(a,b) point-in-time onto test. Default fit_n=None preserves prior behaviour.
PART B (DoD): new app/services/forecasting/regression.py — Almon polynomial
distributed-lag (deg 2) of Δln(district demand) on Δkey_rate lags 0..6 via
OLS-on-Almon-regressors (numpy lstsq) + per-lag reconstruction + manual
Newey-West HAC SEs (NO statsmodels). Output {best_lag_months, coef=long-run
multiplier, x_pct, r2, n, per_lag_coef, hac_se,...}; gate mirrors _elasticity_coef
(n<30 OR R²<0.1 OR Σβ≥0 → fallback); §9.6 phrase from the lag shape. ADVISORY,
shipped standalone (integration point documented), NOT wired — protects the live
compute_rate_sensitivity consumers.
125+31 tests (synthetic known-lag recovery, HAC computed/differs-from-OLS,
fallback gating, no-leakage detrend). ruff clean. Refs #978
REOPENED — normalize.py was never created; only rate-regime discount existed.
New backend/app/services/forecasting/normalize.py with normalize_demand(series):
multiplicative month-of-year deseasonalization of the raw monthly demand
SalesSeries (§9.4). Pure/deterministic; min-data guard (<2 full years / empty
month / overall_mean<=0 → factor 1.0, no divide-by-zero, no thin-data noise).
Exposes seasonal factors for explainability. Synthetic unit test: seasonality
removed (month means equalised), flat unchanged, thin/empty/all-zero safe.
DoD (module + doc + test) MET. Production wiring into
rate_sensitivity._align_sales_deltas DEFERRED (documented TODO): deseasonalizing
the short rate-driven series perturbs the recovered β/lag on current data —
needs a points-per-month gate / joint seasonal+rate estimation + backtest before
wiring. Forecast stack is advisory regardless. Refs #979
REOPENED#980: when effective competing supply is exhausted under positive demand
(projected_supply<=0, demand>0), deficit_index now caps to +1.0 (peak of [-1,+1])
instead of None. balance_ratio stays None (demand/0 undefined), but the strongest
build signal no longer reads downstream as thin data (market_fit fell to 0.5,
what_to_build dropped the cell). No-signal (supply<=0 AND demand<=0) stays None.
REOPENED#981: MAI now uses CBR key rate (macro_indicator key_rate/rf via
get_monthly_macro) as the market borrowing-cost proxy (~16-21%) instead of the
subsidized weighted rate (~7.83%), per §7.9 DoD. rate_kind='key_rate_proxy'.
If key_rate absent → rate_kind='market_unavailable' (no silent subsidy fallback).
Income (#946) still missing → payment_to_income None, confidence low.
778 forecasting tests green. Refs #980#981
REOPENED. PDF + Excel exporters read non-existent dict keys, so demand/supply/
scenario columns silently rendered "—". Tests passed only because the fixtures
were stale (hand-typed the same wrong keys → fixture agreed with buggy exporter).
- future_market: demand/supply → projected_demand_units/projected_supply_units
- scenarios: drop non-existent per-scenario "overall"; show primary-horizon
deficit_index from ScenarioForecast.forecasts (scoring.overall was NOT broken)
- Excel #991: add missing future_supply (index + breakdown) + confidence.factors
sections; add future_supply to PDF for parity
- tests: rebuild forecast/scenario fixtures from real DemandSupplyForecast /
ScenarioForecast as_dict(); contract-key regression guards fail on key-drift
(verified: reintroducing old keys fails the new tests). 28 passed.
Refs #989#991
REOPENED. _SALES_WINDOW_SQL derived "sales in window" from objective_lots_history
snapshots, but history is only ~17 days deep — every currently-sold lot had a
sold-snapshot in the window, so window-sales collapsed into the entire cumulative
sold stock (Автовокзал 6mo: 33,245 vs real ~2,308). Inflated absorption_rate
(~235%/mo with confidence=high), months_of_supply, unit_velocity, liquidity,
demand_concentration → contaminated forecast #950/#952.
Count window sales directly from objective_lots by contract_date in the window
(the real sale date — present on 100% of sold lots: 41,091/41,091). Return
contract of _query_sales_window unchanged (units/area/by-room ROLLUP); downstream
formulas untouched. Removed the now-dead objective_lots_history JOIN/CTE.
Regression test: lots sold outside window (contract_date out of range) not counted
(41,091 cumulative vs 2,308 window → absorption 2.35→0.04). 288 tests green.
Verification = prod compute_market_metrics(Автовокзал) post-deploy. Refs #949
REOPENED. L3 future-supply rows are computed per (district_name, dev_group_name)
but dev_group_name was never a key column — only embedded in method text. With
complex_id/obj_class NULL for L3, every dev_group of a district collapsed to one
upsert key → ~95.6% loss. Ground-truth (Академический, prod): should be 13,808
units / 15 dev_groups / 54 objects; only 1 row / 607 units survived.
Migration 128: ADD COLUMN supply_layers.dev_group_name TEXT + rebuild
uq_supply_layers_logical to (layer, district_name, complex_id, obj_class,
dev_group_name, source, snapshot_date) NULLS NOT DISTINCT (L1/L2 dev_group_name
NULL stays transparent → their dedup unchanged; L3 distinct groups no longer
collapse). Dry-run-verified vs prod catalog (applies clean, ROLLBACK clean).
Worker: SupplyLayerRow gains dev_group_name (L1/L2=None, L3=group); _UPSERT_SQL
adds it to INSERT/VALUES (CAST(:dev_group_name AS text)) + ON CONFLICT (key col,
not in DO UPDATE SET). Service+worker regression tests assert same-district/
different-dev_group → distinct keys (no collapse). 234 supply tests pass.
Deploy applies migration before container restart; collapsed data self-heals on
next supply_layers_refresh. Verification = prod re-measure post-deploy.
Refs #970
Complements the NULL-geom 500 fix. cad_exists_in_db (docstring: "is there
GEOMETRY") checked only row existence, not geom IS NOT NULL — so for the ~964
meta-but-NULL-geom parcels it returned True. Consequence after the 500 fix:
such a parcel fell into the analyze fallback, find_or_enqueue_fetch step 2 saw
cad_exists_in_db=True → returned ("ready", None) → NO NSPD fetch enqueued →
analyze looped to a 202 with job_id=null and the parcel was stuck "fetching"
forever (never pulled real geometry, never resolved).
Fix: add `AND geom IS NOT NULL` to all three EXISTS branches (aligns the
function with its docstring). Now a NULL-geom parcel → cad_exists_in_db=False →
a real NSPD fetch is enqueued (202 + real job_id) → geometry populates →
re-poll → analyze succeeds (or 404 not_in_nspd if NSPD lacks it). No more
stuck-202. Valid-geom parcels unaffected. All 3 callers want geometry-presence
semantics. 37 analyze/fetch/by-bbox tests green. Refs #944.
POST /analyze 500'd (TypeError: float(None)) for ~964 cad_parcels_geom rows
that have a meta row but geom IS NULL: the resolution query returned the row
(no NULL filter), geom_wkt became NULL → ST_Centroid(NULL) → NULL lat/lon →
float(centroid_row["lat"]) crashed (the `if centroid_row else` guard checked
row existence, not NULL values). A user entering any such "random parcel" hit
a 500.
Fix:
- Add `AND geom IS NOT NULL` to all three geometry-resolution queries (initial
UNION, inline-refetch UNION, geom_row WKT subquery) so a NULL-geometry parcel
is treated as "no usable geometry" → falls into the #93 graceful fallback
(enqueue NSPD on-demand fetch → 202 polling), exactly like a parcel absent
from the DB. The fetch then populates real geometry and analyze succeeds.
- Defensive: centroid_lat/lon now check the VALUE is not None (not just row
existence) before float(), falling back to EKB center.
Verified: badformat→400, not-in-DB→202(fetching), valid parcel→200 unaffected.
39 analyze/by-bbox tests green. Refs #944.
The whole backend api/v1 test suite hits `app` via TestClient mimo Caddy, so
rbac_guard (app/main.py) 401'd every authed request (no X-Authenticated-User)
— 112 tests red, the suite effectively dead (which is how #994's district 500
shipped uncaught). Add `settings.testing` (default False — prod RBAC untouched)
+ a strictly-gated bypass at the top of rbac_guard; tests/conftest.py enables it
(env TESTING=1 + settings.testing=True before app import).
Security: bypass fires ONLY when settings.testing is True, which is set NOWHERE
in prod (default False; only tests/conftest.py flips it). RBAC's 401/403 logic
stays covered by tests/test_rbac.py (its own middleware copy, ignores the flag).
Effect: 112 → 42 failed, 1590 → 1660 passed. Remaining 42 (+5 mv_layout env
errors) are stale-mock/assertion drift + env-required tests — fixed in CI-rehab
2/3. Foundation for a real Forgejo pytest gate (3/3).
Refs #944.
LIVE BUG (from #994, already merged): the analysis_runs persist call-site in
analyze_parcel extracted district via result_payload["district"]["district_name"]
— but a district dict without that key (real data: partial district lookup)
raised KeyError OUTSIDE the best-effort SAVEPOINT (extraction is at the call
site, not inside persist_analysis_run) → 500 on the LIVE /analyze endpoint.
Fix: .get("district_name") → None instead of raising. Caught by reviving the
analyze test suite (below).
Test-infra (the suite was un-runnable, which is why the bug shipped):
- Lazy-import WeasyPrint in layout_tz_pdf.py + trade_in_pdf.py (matches the
existing report_pdf.py / snapshot_pdf.py pattern). The eager top-level imports
made `app.main` (→ parcels/trade_in routers) un-importable on hosts without
WeasyPrint native libs (e.g. macOS dev), breaking pytest COLLECTION of the
whole api/v1 suite. WeasyPrint is still imported when a PDF is actually rendered.
- tests/conftest.py: autouse fixture clears app.dependency_overrides after each
test (anti-leak — a leaked get_db override caused real-DB connection attempts).
- test_parcel_by_bbox.py: rewrite get_db mocking from patch("...get_db") (a no-op
— FastAPI Depends holds the original ref) to app.dependency_overrides[get_db],
add explicit X-Authenticated-User header (RBAC gate), patch latest_run_dates,
+ a new test asserting last_analysis_date from a latest run (#994). 5/5 green.
NOTE: reviving collectability exposes PRE-EXISTING rot in other api/v1 suites
(analyze/admin/best_layouts: RBAC-header + stale-assertion/mock drift) — those
are NOT regressions from this PR (they were uncollectable before) and are
tracked separately for a deliberate suite-rehab + CI test-gate effort.
Refs #994.
Persist each successful POST /analyze into analysis_runs (migration 127):
best-effort SAVEPOINT-wrapped INSERT in a thin analysis_runs repository, then
explicit db.commit() (get_db has no commit-on-success; SAVEPOINT RELEASE alone
does not persist — without the commit the row rolls back on request teardown,
a silent no-op caught in code review). A persist failure never breaks the
response or poisons the session. Result serialized via jsonable_encoder→
json.dumps (handles embedded Pydantic models/dates); confidence/status
normalized to satisfy the 127 CHECK constraints (unknown→NULL/'complete').
Populate by-bbox last_analysis_date from v_analysis_runs_latest via a single
batch query (no N+1), replacing the #307 placeholder. The read is best-effort
wrapped too — a view-missing deploy window or future drift must not 500 the
map (falls back to last_analysis_date=None).
Additive only — analyze response shape unchanged. Tests: by-bbox suite patches
latest_run_dates (mock-db returns same rows for any query) + new test asserts
ISO last_analysis_date from a latest run. Analyze mock-suite unaffected
(graceful side_effect overflow + best-effort persist absorb the extra INSERT).
Incidental: ruff-format fixed one pre-existing f-string spacing (line ~2391)
the format hook flags on touch.
Closes#994. Refs #961.
