Add the LLM prose-composition path for the parcel-forecast chat, layered
over the deterministic Step-1 fallback which stays the safety net.
- chat/tools.py: 5 read-only section tools (exec_summary, product_recommendation,
forecast, risks, scenarios) — pure slices of the loaded report dict, no DB/
recompute, graceful on missing sections. market_now (raw analyze blob) and meta
are deliberately NOT exposed -> highest-PII data cannot reach the LLM.
- chat/safe_payload.py: the §19 gate — single place that builds the outbound
SafePayload from a section-aggregate allowlist; honors is_confidential hard-block.
- chat/orchestrator.py: manual tool-call loop with call-cap/termination, real
grounded_in provenance; any LLMResult.ok=False (disabled/timeout/rate_limited/
redaction_refused/call_cap/provider_error/empty) degrades to the deterministic answer.
- llm/prompts.py: versioned chat_system@v1 — answer only from sections, never
fabricate numbers, advisory tone, decline out-of-scope.
- api/v1/chat.py: branch on settings.llm_enabled; sync complete bridged via
run_in_threadpool. Default-off -> deterministic path, no provider built.
- Tests: fake provider only (no network), planted-secret redaction-boundary +
per-reason fallback + call-cap + numbers-from-report coverage.
Refs #957
Step 1 of #957. Answers parcel questions by reading the already-persisted
§22 SiteFinderReport (latest_run_for, schema "1.0", read-only) and returning
templated RU answers with engine numbers verbatim (§16, never fabricated).
Intent routing (explicit or RU keyword match) -> per-section renderers;
graceful on partial/missing sections and pending (no run / DB error) without
500s. Works with llm_enabled=False (llm_used always False); LLM composition
is Step 2. Mounted off /api/v1/chat so rbac_guard auto-requires an authed
known user.
Refs #957
Upgrade the §25.3 cannibalization index from a same-class-competitor proxy to
true own-portfolio overlap: score the recommended candidate segment against the
developer's own portfolio (get_own_portfolio, #1169 PR1) across axes —
audience/class (ordinal distance), price ₽/м² (interval overlap), unit-mix
(L1 similarity), timing (half-life decay) — geo-weighted by haversine proximity
to the parcel, aggregated by geo-weighted soft-max (the strongest nearby
cannibalizer dominates, not a diluting mean). Empty portfolio -> labelled proxy
fallback, confidence forced low, never presented as the true index.
Pure scoring fns unit-tested without DB; None-not-0 on missing axes; thin/
only-current portfolio -> low confidence + §26 note; deterministic (sorted
tie-breaks, no RNG). class+price+geo active now; unit-mix+timing plumbed via
optional params for a follow-up that wires them from the orchestrator horizon.
ruff + mypy clean; 151 special-index tests pass (964 forecasting dir, no regr).
Refs #1169
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
demand_index used a fixed clamp01(velocity/50); on a live prod refresh
all 8 ЕКБ districts sold ≥50/mo so it saturated to 1.0 everywhere — zero
discrimination between districts. Redesign to mirror infra_index:
normalize each district's unit_velocity against the city reference (MAX
district velocity per refresh run), so demand always discriminates and
self-calibrates as the market grows (no magic constant to rot).
- normalize_demand(velocity, *, city_reference_velocity), pure + graceful
(None stays None; reference<=0 -> honest 0.0, no ZeroDivisionError)
- refresh_locations now two-pass: collect velocities (one
compute_market_metrics per district, no O(n^2)), derive city reference,
normalize + upsert; SAVEPOINT-per-row and counters preserved
- remove _DEMAND_SATURATION_UPM constant; log city_reference_velocity
- tests: rewrite demand normalization + add end-to-end city-relative
suite incl. discrimination regression guarding the all-1.0 prod bug
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add
fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934,
monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert
indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series
guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK).
Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the
reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target
nudge, non-positive to avoid double-counting f_rate, excluded from
_RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed;
_CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved
0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting
tests green. No migration, no new deps.
Refs #946.
Cold §22 forecast measured ~215-233s on prod: §9.x layers re-execute the same
horizon/segment-invariant DB loads with identical args hundreds of times per
report (profiled: get_competitors x69, market_metrics x124, get_monthly_macro
x290). Add a per-report ContextVar cache (forecast_cache(), opened once in the
orchestrator) + @cached(key_builder) on the expensive §9.x loaders so each
unique load runs ONCE and reuses the same frozen, read-only instance.
Output is byte-identical (memoized producers are frozen dataclasses / read-only
Pydantic, callers never mutate; cache is per-report, discarded on exit; no-op
outside the report build). No concurrency, no signature changes.
- forecast_request_cache.py: ContextVar cache + cached() decorator (no-op
outside context, reentrant, _MISS sentinel for cached None)
- @cached on competitors/future_supply/market_metrics/macro_series/
sales_series/macro_coefficient/demand_normalization/regression loaders
- orchestrator: wrap build_site_finder_report in forecast_cache()
- 58 tests: key discrimination (call-counting regression guard), no-op-outside,
per-report isolation, reentrancy, frozen-producer canary, amplification proof
(real get_monthly_macro xN->1)
code-reviewer APPROVE (keys correct, mutation-safe, output identical). 1265
forecast/cache tests green. No new deps. Refs #1129.
render_report_docx (python-docx) mirrors report_md/report_pdf section order &
content, reuses report_pdf pure helpers (DRY), graceful on thin/empty report.
Widen /forecast/export format Literal to include docx → Word attachment.
Add python-docx dep + regenerate uv.lock (uv sync --frozen passes). Part of #959.
render_report_telegram_summary (pure, no new deps, DRY-reuses report_pdf str
helpers) + `tg` format on GET /{cad}/forecast/export → inline text/plain snippet
(no attachment, copy-paste-ready). md/json unchanged; no-run 404, bad format 422.
Graceful on thin/empty reports. Part of EPIC #959.
render_report_markdown (pure, no new runtime deps) reuses report_pdf's str
helpers (DRY), + GET /{cad_num}/forecast/export?format=md|json. No forecast run
→ 404; graceful on thin/empty reports; GFM-safe table escaping. PDF/XLSX already
existed; this adds the cheapest no-dep formats. Part of EPIC #959.
#992: attach response_model=AnalyzeResponse to POST /{cad_num}/analyze. Model
uses extra="allow" so any result_payload key not explicitly modeled is preserved
in the 200 response (no silent drops that would break Site Finder), and ALL
fields are Optional so the #93 202 fetch-stub serializes without a 500.
#994: add GET /{cad_num}/runs (light summary list, empty 200 not 404) and
GET /runs/{run_id} (full row incl result, 404 if missing), backed by
list_runs_for/get_run in analysis_runs.repository (psycopg v3 CAST(:x AS type)).
Routes ordered before /{parcel_id} so /runs/{run_id} is not shadowed.
Closes#992. Closes#994. Refs #961.
build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task,
not inline on the sync /analyze endpoint.
- repository: latest_run_for gains keyword-only schema_version (default None keeps
v_analysis_runs_latest behavior, backward-compat); when given, reads base
analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches
the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top
(index-served via 127's (cad_num, created_at DESC)).
- new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0,
runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via
persist_analysis_run. Graceful: no base run / compute error → logger + return None
(worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include.
Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district)
→ orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass;
code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized,
no IntegrityError). Refs #994#961.
deficit_index pins to -1.0 for every ЕКБ segment (12mo demand flow vs multi-year
supply stock → log-ratio clamps) → zero discriminating power, though the oversupply
is partly real. Add MOI (gross competing supply / demand_per_mo), the real-estate
absorption standard, as an additive non-saturating companion that DISCRIMINATES
(Уралмаш 42mo … Чермет 109mo) where deficit cannot. deficit_index math kept exactly
as-is (honest absolute: -1 = genuinely oversupplied); docstrings clarify -1 is common
and MOI is the discriminating companion (no recalibration). _gross_supply extract-method
(single source of truth; _project_supply behavior byte-identical, code-review-verified).
Surface MOI in §22 future_market (passthrough) + exec_summary key_numbers/verdict.
Guards: no demand → None, no supply → 0. Prod: MOI varies 42→109mo, deficit stays -1.
Discrimination test pins MOI separating two segments both at deficit -1. Refs #952.
/analyze passes the official ЕКБ admin district (ekb_districts polygon, e.g.
'Кировский'), but objective_lots/corpus_room_month store informal micro-districts
('Втузгородок','ЖБИ') -> admin name matched 0 rows -> silent empty forecast.
Add resolve_objective_districts() (site_finder/district_resolver.py) mapping an
admin name to its clean micros via ekb_district_alias (note IS NULL), with
None -> EKB-wide fallback and raw-micro pass-through. Wire into the objective_lots
district filters of market_metrics (§9.2 stock+sales), supply_layers L1 (§9.3),
and sales_series Sources A+B (crm shares the micro vocab, prod-verified),
switching the scalar filter to psycopg3-safe = ANY(CAST(:districts AS text[])).
supply_layers L2/L3 keep the admin name (domrf_kn_objects.district_name is admin vocab).
Prod: Кировский/Ленинский/Орджоникидзевский obj_count 0 -> 32/64/31.
Tests mutation-verified non-vacuous. 192 module tests pass; ruff clean. Refs #969#949.
_KEY_RATE_MARKET_SPREAD_PP was a 0.0 placeholder, so §7.9 affordability payments
used the bare CBR key_rate (~14.5%), understating borrower cost and OVERSTATING
affordability. Calibrate to 4.5pp from the prod anchor (macro_indicator
mortgage_rate_primary_domrf 19.125% @ 2026-04-19 vs key_rate ~14.5% -> observed
~4.6pp, rounded conservatively; inside the typical RF 3-5pp band), so
rate_used = key_rate + 4.5 ~= 19% matches the directly-observed market primary
mortgage rate. Makes affordability LESS optimistic / more accurate. Docs + tests
updated to the symbolic spread (new regression anchor pins spread==4.5 and
key_rate 14.5 -> ~19.0); rate_kind/graceful-fallback semantics unchanged.
Forecasting suite 841 passed; ruff clean.
Backtest (OOS directional hit-rate): single-best-lag compute_rate_sensitivity
is directionally noise (0.148 Source B EKB-wide, lag-unstable); the Almon
distributed-lag estimator (compute_district_rate_regression) is strictly less
noisy on every tier (0.407 Source B / 0.60 survivorship-free Source A,
lag-stable). Add a thin adapter compute_rate_regime_sensitivity mapping
DistributedLagFit onto the existing RateSensitivity contract (beta=long-run
sum-beta, confidence regression->medium / fallback->low, district=None->low and
no call) and repoint the three consumers (demand_normalization, product_scoring,
demand_supply_forecast). Magnitude bounded by the existing [0.5,1.2] clamp.
Reversible; compute_rate_sensitivity kept for the backtest. Consumer tests
repointed to the real Almon path (mutation-verified genuine) + adapter unit
tests + end-to-end fallback degradation. Forecasting suite 840 passed; ruff clean.
Found by read-only services audit.
- recommendation._usp_from_deficits: skip di<=0 so «стройте его» is never emitted
for OVERSUPPLIED formats; all-surplus top-K → [] (no white-space niches).
Aligns with product_scoring._count_positive_usp (di>0). Was: «Дефицит формата
X — стройте его» for a surplus format, reaching PDF/Excel USP-ниши.
- report_assembler._domrf_coverage: drop ambiguous >1.0 percent-guess; normalize
per-branch (analyze pct /100, supply_layers fraction as-is). Sub-1% coverage
(0.8%) no longer read as 80% → no inflated confidence in the near-zero-coverage
case §15 flags. tests for both + end-to-end no-inflation. 241+148 pass.
REAL code-level cause (prod-confirmed, 2 parcels: "Geometry type (Polygon) does
not match column type (MultiPolygon)") of single-contour parcels stuck "fetching"
— complements migrations 129/130 (which fixed the cad_parcels NOT NULL chain).
F1: _save_parcel/_save_quarter now ST_Multi() the geom so single-contour Polygons
fit the MultiPolygon columns (migr 93/58), mirroring the bulk_harvest paths.
F2: _save_building rewritten to the wide cad_buildings schema (migr 92):
quarter_cad_number (was quarter_cad_num → UndefinedColumn), _safe_int(floors)
(range strings → NULL), source='nspd' NOT NULL.
F3: upsert_features wraps each feature in begin_nested() SAVEPOINT so one bad
feature no longer aborts the whole quarter snapshot (backend.md rule; mirrors
grid-walk). New tests/workers/test_nspd_geo.py + savepoint tests. 89 passed.
FOLLOW-UP (Leha: not a bug, follow-up).
get_macro_series / get_latest_macro defaulted region='rf', but CBR mortgage_*
series live under 'sverdl' in macro_indicator (key_rate under 'rf'). Only caller
passes region explicitly so no active leak, but a future caller omitting region
would silently get None — latent footgun. region now str|None; when None →
_canonical_region (mortgage set sourced from _CBR_SERIES_MAP, single source of
truth). Explicit region always wins → existing callers unchanged. Region-binding
tests added. 159 tests pass.
Documented (NOT code-fixed — upstream data-quality): mortgage_debt /
mortgage_overdue stay empty in macro_indicator because cbr_mortgage_series.period
holds value-like garbage (e.g. '10054588.0') for the Debt series — corruption
from the upstream CBR-XLSX scraper that built domrf.db (imported verbatim by
44_import_anton_db.py), NOT an ingest bug. 123_macro_indicator.sql correctly
skips unparseable periods. Needs a separate scraper re-ingest (idempotent
backfill once period is valid). Refs #945
