render_report_docx (python-docx) mirrors report_md/report_pdf section order &
content, reuses report_pdf pure helpers (DRY), graceful on thin/empty report.
Widen /forecast/export format Literal to include docx → Word attachment.
Add python-docx dep + regenerate uv.lock (uv sync --frozen passes). Part of #959.
«Скачать отчёт .md / .json» + «Скопировать для Telegram» on the ready §22
forecast, wired to GET /parcels/{cad}/forecast/export. triggerDownload extracted
to lib/download.ts (reused by ExportButtons). Clipboard fallback for insecure
contexts; 404 → inline «Отчёт ещё не готов». Part of EPIC #959.
_http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест.
Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
Миграция 100_enable_deactivate_stale_avito.sql — идемпотентный UPDATE scrape_schedules SET enabled=true, next_run_at=NOW() WHERE source='deactivate_stale_avito' AND enabled=false. Чинит QA-находку #759: 090 ON CONFLICT DO NOTHING оставил pre-seed disabled-строку → task ни разу не отработала. Non-destructive, guard на повторный прогон. +7 тестов.
Closes#759.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
render_report_telegram_summary (pure, no new deps, DRY-reuses report_pdf str
helpers) + `tg` format on GET /{cad}/forecast/export → inline text/plain snippet
(no attachment, copy-paste-ready). md/json unchanged; no-run 404, bad format 422.
Graceful on thin/empty reports. Part of EPIC #959.
render_report_markdown (pure, no new runtime deps) reuses report_pdf's str
helpers (DRY), + GET /{cad_num}/forecast/export?format=md|json. No forecast run
→ 404; graceful on thin/empty reports; GFM-safe table escaping. PDF/XLSX already
existed; this adds the cheapest no-dep formats. Part of EPIC #959.
#992: attach response_model=AnalyzeResponse to POST /{cad_num}/analyze. Model
uses extra="allow" so any result_payload key not explicitly modeled is preserved
in the 200 response (no silent drops that would break Site Finder), and ALL
fields are Optional so the #93 202 fetch-stub serializes without a 500.
#994: add GET /{cad_num}/runs (light summary list, empty 200 not 404) and
GET /runs/{run_id} (full row incl result, 404 if missing), backed by
list_runs_for/get_run in analysis_runs.repository (psycopg v3 CAST(:x AS type)).
Routes ordered before /{parcel_id} so /runs/{run_id} is not shadowed.
Closes#992. Closes#994. Refs #961.
NSPDClient.list_layers() парсил _walk_layer_tree по id/children, но НСПД layers-theme-tree отдаёт слои в плоском layers[] с ключом layerId → list_layers() молча возвращал [] для любой темы. Fix: сначала плоский data["layers"], иначе fallback на _walk_layer_tree; defensive None-guard на missing id. +тест на flat-форму, backward-compat сохранён.
Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
Docker auto-sets HOSTNAME to the container id; Next standalone's server.js
binds process.env.HOSTNAME, so it listened on the container hostname and the
compose healthcheck (TCP 127.0.0.1:3000) was refused → frontend container
chronically "(unhealthy)" (cosmetic: Caddy reaches it by service name and the
site serves 200). Pin HOSTNAME=0.0.0.0 — the canonical Next.js Docker fix.
Diagnosed on prod 2026-06-05.
Forgejo runs only .forgejo/workflows/* — the .github pytest gate never
executed here, so backend changes merged + deployed untested (live bug
#994 district 500 shipped uncaught). Add a real gate: ruff check + the full
backend pytest suite (1687 passed) on PRs to main and feature-branch pushes,
scoped to backend/** + data/sql/** via paths-filter. Mock-only lane (no
postgres service): the one real-DB test self-skips via a connectivity probe;
WeasyPrint PDF tests RUN thanks to installed libpango. uv via official
installer (not setup-uv, PEP 668), uv sync --frozen against the committed lock,
TESTING=1 to activate the test-mode RBAC bypass.
NOTE: making this a HARD required check + having the auto-merge bot consult
check status needs Forgejo branch-protection config (human action) — until
then the gate is visible but advisory.
Refs #944.
Add nullable lat/lon (EPSG:4326, 6 dp) to /analyze competitors[] and
pipeline_24mo.top_objects[] so the frontend can plot Leaflet markers. Coords
come from domrf_kn_objects.latitude/longitude (same source as distance_m).
Purely additive: no existing field/shape changed. Frontend map layers follow
in a separate PR. Part of EPIC #958 (958-B4).
Critical demo-blocker: POST /analyze returns 202 {status:fetching} for parcels
whose geometry isn't cached yet (#93 on-demand НСПД fetch). useParcelAnalyzeQuery
treated 202 as success → the stub (no score/competitors) reached render → Section 3/4
threw TypeError → white screen (no error boundary). Repro confirmed on prod.
- useParcelAnalyzeQuery: 202-aware — poll /fetch-status (2s, 2-min cap), re-POST
on ready (status-checked, no stub on 202 race), throw HTTPError on
not_in_nspd/failed/invalid_format; horizon preserved in both POSTs; retry: false
- error boundaries: app/site-finder/analysis/[cad]/error.tsx + app/error.tsx —
no white screen ever; calm RU message in prod, error detail in dev
- Section3/Section4: guard competitors/score against partial payloads
- CadInput: drop hardcoded default cad (empty + placeholder), raw hex → tokens
Map parcel-select already fixed by 3cea915. Part of EPIC #958.
Segmented control «Горизонт прогноза» driving the analyze query (?horizon=)
and Section 6's target highlight. Default 12.
- HorizonSelector: radiogroup + a11y radios, tokens-only, tabular-nums,
disabled while a re-analyze is in flight
- useParcelAnalyzeQuery(cad, horizon=12): horizon in queryKey + ?horizon=;
keepPreviousData so switching horizon doesn't blank the page to loading
- AnalysisPageContent: horizon state + forecast-poll invalidation on change
- Section6Forecast/ForecastHorizonsBlock: selectedHorizon → target row («Целевой»)
Part of EPIC #958.
build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task,
not inline on the sync /analyze endpoint.
- repository: latest_run_for gains keyword-only schema_version (default None keeps
v_analysis_runs_latest behavior, backward-compat); when given, reads base
analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches
the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top
(index-served via 127's (cad_num, created_at DESC)).
- new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0,
runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via
persist_analysis_run. Graceful: no base run / compute error → logger + return None
(worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include.
Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district)
→ orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass;
code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized,
no IntegrityError). Refs #994#961.
deficit_index pins to -1.0 for every ЕКБ segment (12mo demand flow vs multi-year
supply stock → log-ratio clamps) → zero discriminating power, though the oversupply
is partly real. Add MOI (gross competing supply / demand_per_mo), the real-estate
absorption standard, as an additive non-saturating companion that DISCRIMINATES
(Уралмаш 42mo … Чермет 109mo) where deficit cannot. deficit_index math kept exactly
as-is (honest absolute: -1 = genuinely oversupplied); docstrings clarify -1 is common
and MOI is the discriminating companion (no recalibration). _gross_supply extract-method
(single source of truth; _project_supply behavior byte-identical, code-review-verified).
Surface MOI in §22 future_market (passthrough) + exec_summary key_numbers/verdict.
Guards: no demand → None, no supply → 0. Prod: MOI varies 42→109mo, deficit stays -1.
Discrimination test pins MOI separating two segments both at deficit -1. Refs #952.
/analyze passes the official ЕКБ admin district (ekb_districts polygon, e.g.
'Кировский'), but objective_lots/corpus_room_month store informal micro-districts
('Втузгородок','ЖБИ') -> admin name matched 0 rows -> silent empty forecast.
Add resolve_objective_districts() (site_finder/district_resolver.py) mapping an
admin name to its clean micros via ekb_district_alias (note IS NULL), with
None -> EKB-wide fallback and raw-micro pass-through. Wire into the objective_lots
district filters of market_metrics (§9.2 stock+sales), supply_layers L1 (§9.3),
and sales_series Sources A+B (crm shares the micro vocab, prod-verified),
switching the scalar filter to psycopg3-safe = ANY(CAST(:districts AS text[])).
supply_layers L2/L3 keep the admin name (domrf_kn_objects.district_name is admin vocab).
Prod: Кировский/Ленинский/Орджоникидзевский obj_count 0 -> 32/64/31.
Tests mutation-verified non-vacuous. 192 module tests pass; ruff clean. Refs #969#949.
§9.x forecast metrics filter objective_lots.district (informal micro-districts)
but /analyze resolves the official admin district from ekb_districts → admin
name matched 0 rows → silent empty forecast (market_metrics('Кировский')→0).
Add a curated micro→admin alias table seeding all 35 micro values to their
primary admin (8 EKB admins). Curated not spatial: complex_id join hits only
~28% of rows and complexes is multi-city, so ST_Contains scatters micros
uniformly. 4 cross-city values (Н.Тагил/Каменск) contained in one noted bucket.
Coverage: 8/8 admins resolve, 35/35 mapped, 0 unmapped. Idempotent + additive
(CREATE TABLE IF NOT EXISTS + INSERT ON CONFLICT DO NOTHING); applied to prod
+ registered in _schema_migrations. Resolver/fallback documented in header;
market_metrics wiring is Step 2 (separate PR). Refs #969#949.
_KEY_RATE_MARKET_SPREAD_PP was a 0.0 placeholder, so §7.9 affordability payments
used the bare CBR key_rate (~14.5%), understating borrower cost and OVERSTATING
affordability. Calibrate to 4.5pp from the prod anchor (macro_indicator
mortgage_rate_primary_domrf 19.125% @ 2026-04-19 vs key_rate ~14.5% -> observed
~4.6pp, rounded conservatively; inside the typical RF 3-5pp band), so
rate_used = key_rate + 4.5 ~= 19% matches the directly-observed market primary
mortgage rate. Makes affordability LESS optimistic / more accurate. Docs + tests
updated to the symbolic spread (new regression anchor pins spread==4.5 and
key_rate 14.5 -> ~19.0); rate_kind/graceful-fallback semantics unchanged.
Forecasting suite 841 passed; ruff clean.
The OOS verdict flagged a variant 'candidate to promote' on hit-rate >= 0.5+margin
+ lag_stable alone. On thin data this over-claims: Source A Almon-ADL scored 6/10
(0.60) lag-stable and was flagged as signal, but P(X>=6|10,0.5)~=0.377 -- a coin
flip. Live ground-truth confirmed no signal (full-sample R2~=0.003, wrong sign).
Add exact stdlib-only one-sided binomial _binom_sf_ge + _VERDICT_ALPHA=0.05 and
require P(X>=hits|n_test,0.5) < alpha in both verdict() and cross_source_verdict()
on top of the effect-size margin. hits recovered exactly as round(hit_rate*n_test)
(n_test==scored invariant; no evaluator shape change). Verdict text now states
n_test + the binomial p on pass and fail. Evaluator/estimator math and the
read-only SELECT discipline untouched. Refs #978.
Backtest (OOS directional hit-rate): single-best-lag compute_rate_sensitivity
is directionally noise (0.148 Source B EKB-wide, lag-unstable); the Almon
distributed-lag estimator (compute_district_rate_regression) is strictly less
noisy on every tier (0.407 Source B / 0.60 survivorship-free Source A,
lag-stable). Add a thin adapter compute_rate_regime_sensitivity mapping
DistributedLagFit onto the existing RateSensitivity contract (beta=long-run
sum-beta, confidence regression->medium / fallback->low, district=None->low and
no call) and repoint the three consumers (demand_normalization, product_scoring,
demand_supply_forecast). Magnitude bounded by the existing [0.5,1.2] clamp.
Reversible; compute_rate_sensitivity kept for the backtest. Consumer tests
repointed to the real Almon path (mutation-verified genuine) + adapter unit
tests + end-to-end fallback degradation. Forecasting suite 840 passed; ruff clean.
Extend the read-only §9.6 rate-sensitivity OOS harness with two opt-in
candidate-method variants so any wiring decision is evidence-based:
- --almon: evaluate_oos_almon, Almon distributed-lag (regression.fit_almon_dl),
fit on TRAIN only, point-in-time sum_j beta_j*drate[t-j] prediction.
- --deseasonalize: train-only month-of-year factors (normalize.seasonal_factors)
divided out before log_diff, then the existing best_lag evaluator.
Both pin the fit to _time_ordered_split(n_train); no look-ahead leakage
(adversarial tests assert the train fit is byte-identical under test corruption).
Default path (best_lag/raw) is byte-identical to before. 88 tests pass, ruff clean.
Prod OOS findings (directional hit-rate, coin-flip 0.50, bar 0.55+lag-stable):
- #979 deseasonalize: neutral (B 0.148->0.148, A 0.40->0.40) -> keep advisory.
- #978 Almon-ADL: dominates best_lag (B 0.148->0.407 lag-stable; A 0.40->0.60,
clears coin-flip+margin) -> candidate to promote from advisory.
weighted_avg_velocity was a naive mean despite the name — a 500-flat ЖК weighed
the same as a 20-flat one. Now count-weighted by flats_total (sql.md AVG
principle): Σ(velocity*flats_total)/Σ(flats_total). Competitors with unknown
flats_total are excluded from weights; if sizes are unknown for ALL, graceful
fallback to the simple mean (den>0 guard). Field name + API contract UNCHANGED
(zero consumer ripple — traced: only CompetitorsSummary, no frontend ref).
Tests: equal sizes → weighted==naive (existing 6.0 stays); NEW test with
500-flat@40 + 20-flat@2 → 38.54 (not naive 21.0), proving the weighting.
Found by read-only services audit.
- recommendation._usp_from_deficits: skip di<=0 so «стройте его» is never emitted
for OVERSUPPLIED formats; all-surplus top-K → [] (no white-space niches).
Aligns with product_scoring._count_positive_usp (di>0). Was: «Дефицит формата
X — стройте его» for a surplus format, reaching PDF/Excel USP-ниши.
- report_assembler._domrf_coverage: drop ambiguous >1.0 percent-guess; normalize
per-branch (analyze pct /100, supply_layers fraction as-is). Sub-1% coverage
(0.8%) no longer read as 80% → no inflated confidence in the near-zero-coverage
case §15 flags. tests for both + end-to-end no-inflation. 241+148 pass.
REAL code-level cause (prod-confirmed, 2 parcels: "Geometry type (Polygon) does
not match column type (MultiPolygon)") of single-contour parcels stuck "fetching"
— complements migrations 129/130 (which fixed the cad_parcels NOT NULL chain).
F1: _save_parcel/_save_quarter now ST_Multi() the geom so single-contour Polygons
fit the MultiPolygon columns (migr 93/58), mirroring the bulk_harvest paths.
F2: _save_building rewritten to the wide cad_buildings schema (migr 92):
quarter_cad_number (was quarter_cad_num → UndefinedColumn), _safe_int(floors)
(range strings → NULL), source='nspd' NOT NULL.
F3: upsert_features wraps each feature in begin_nested() SAVEPOINT so one bad
feature no longer aborts the whole quarter snapshot (backend.md rule; mirrors
grid-walk). New tests/workers/test_nspd_geo.py + savepoint tests. 89 passed.
Marker click opens an interactive Leaflet popup with a primary "Открыть анализ"
action that navigates to /site-finder/analysis/<cad> (encodeURIComponent —
matches the [cad] route's single-decode contract) and records the visit in
RecentParcels. Hover tooltip enriched with status + actionable hint. Existing
ParcelDrawer preview kept, reachable via the popup's secondary "Подробнее".
ESLint + tsc clean.
After #129 fixed quarter_cad_number, the on-demand НСПД ingest of a real
not-pre-loaded parcel hit the NEXT cad_parcels NOT NULL column: source
(NotNullViolation) → geometry transaction rolled back → POST /analyze still
stuck. Repro 66:41:0610029:83 (job 57: quarter derived by #129, source null).
Migration 130: extend the #129 BEFORE INSERT/UPDATE trigger function to also
default source→'nspd' when NULL (column DEFAULT can't help — INSERT binds
explicit NULL; trigger overrides it). Explicit sources ("search"/"wms_grid_walk")
respected — only NULL is filled. cad_parcels NOT NULL surface now fully covered
(cad_num/quarter[#129]/category_id[literal]/source[#130]/fetched_at+updated_at[NOW]).
Dry-run-verified on prod: null quarter+source → 66:41:0610029 + nspd, INSERT ok.
PROD bug: on-demand НСПД ingest (bulk_harvest.upsert_parcel, for parcels NOT
pre-loaded) intermittently inserts quarter_cad_number=NULL at runtime →
NotNullViolation → geometry never persists → POST /analyze for a real new parcel
stuck "Геометрия загружается из НСПД…" forever (re-enqueues each call). Repro:
66:41:0610029:83 (real НСПД parcel; failing row had valid geom + props, only
quarter_cad_number null). Code derives quarter_cad but NULL still reaches the
INSERT via some runtime path (grid-walk).
Migration 129: BEFORE INSERT OR UPDATE trigger on cad_parcels — when
quarter_cad_number IS NULL and cad_num has >=3 colon-segments, derive it
(first 3 segments: 66:41:0610029:83 → 66:41:0610029). Fixes ALL write paths at
the DB level (quarter_cad_number is a pure derivation of cad_num). <3-segment
cad_num intentionally left NULL → NOT NULL still rejects genuine garbage (no
masking). Coexists with m.92 set_updated_at trigger. Idempotent.
Dry-run-verified on prod catalog: null-quarter insert → derived 66:41:0610029;
garbage <3seg still rejected. Verification = re-fetch 0610029:83 post-deploy.
FOLLOW-UP (Leha: not a bug, follow-up).
get_macro_series / get_latest_macro defaulted region='rf', but CBR mortgage_*
series live under 'sverdl' in macro_indicator (key_rate under 'rf'). Only caller
passes region explicitly so no active leak, but a future caller omitting region
would silently get None — latent footgun. region now str|None; when None →
_canonical_region (mortgage set sourced from _CBR_SERIES_MAP, single source of
truth). Explicit region always wins → existing callers unchanged. Region-binding
tests added. 159 tests pass.
Documented (NOT code-fixed — upstream data-quality): mortgage_debt /
mortgage_overdue stay empty in macro_indicator because cbr_mortgage_series.period
holds value-like garbage (e.g. '10054588.0') for the Debt series — corruption
from the upstream CBR-XLSX scraper that built domrf.db (imported verbatim by
44_import_anton_db.py), NOT an ingest bug. 123_macro_indicator.sql correctly
skips unparseable periods. Needs a separate scraper re-ingest (idempotent
backfill once period is valid). Refs #945
