Closes the last #946 gap (СМР). Fetches «Индексы цен производителей на
строительную продукцию по РФ» from the OPEN Rosstat xlsx
(rosstat.gov.ru/storage/mediabank/Invest_ind_stroitel_MM-YYYY.xlsx),
parses the monthly «к предыдущему месяцу» section → macro_indicator
(indicator_type='construction_price_index', region='rf', monthly, %).
The reCAPTCHA-gated fedstat dataGrid.do path is DELIBERATELY NOT used —
the open Rosstat source needs no captcha. EMISS id=31108 documented as a
prod-only SDMX alternative (not wired).
URL is date-derived with bounded look-back (filename changes monthly;
_fetch_construction_latest walks current month back to 6, first 200 wins;
year-boundary handled, exhaustion raises, 404 breaks fast — no hang).
Wired into rosstat_macro_sync with per-source try/except (can't break
demography/income) + SAVEPOINT-per-row. None-not-0 on blank months.
33 tests (incl look-back + year-boundary + graceful), ruff clean.
Note: no construction-cost channel in macro_coefficient yet (demand-side
only) — series is ingested/available for future cost-side use + display.
Refs #946
Add the LLM prose-composition path for the parcel-forecast chat, layered
over the deterministic Step-1 fallback which stays the safety net.
- chat/tools.py: 5 read-only section tools (exec_summary, product_recommendation,
forecast, risks, scenarios) — pure slices of the loaded report dict, no DB/
recompute, graceful on missing sections. market_now (raw analyze blob) and meta
are deliberately NOT exposed -> highest-PII data cannot reach the LLM.
- chat/safe_payload.py: the §19 gate — single place that builds the outbound
SafePayload from a section-aggregate allowlist; honors is_confidential hard-block.
- chat/orchestrator.py: manual tool-call loop with call-cap/termination, real
grounded_in provenance; any LLMResult.ok=False (disabled/timeout/rate_limited/
redaction_refused/call_cap/provider_error/empty) degrades to the deterministic answer.
- llm/prompts.py: versioned chat_system@v1 — answer only from sections, never
fabricate numbers, advisory tone, decline out-of-scope.
- api/v1/chat.py: branch on settings.llm_enabled; sync complete bridged via
run_in_threadpool. Default-off -> deterministic path, no provider built.
- Tests: fake provider only (no network), planted-secret redaction-boundary +
per-reason fallback + call-cap + numbers-from-report coverage.
Refs #957
Step 1 of #957. Answers parcel questions by reading the already-persisted
§22 SiteFinderReport (latest_run_for, schema "1.0", read-only) and returning
templated RU answers with engine numbers verbatim (§16, never fabricated).
Intent routing (explicit or RU keyword match) -> per-section renderers;
graceful on partial/missing sections and pending (no run / DB error) without
500s. Works with llm_enabled=False (llm_used always False); LLM composition
is Step 2. Mounted off /api/v1/chat so rbac_guard auto-requires an authed
known user.
Refs #957
Upgrade the §25.3 cannibalization index from a same-class-competitor proxy to
true own-portfolio overlap: score the recommended candidate segment against the
developer's own portfolio (get_own_portfolio, #1169 PR1) across axes —
audience/class (ordinal distance), price ₽/м² (interval overlap), unit-mix
(L1 similarity), timing (half-life decay) — geo-weighted by haversine proximity
to the parcel, aggregated by geo-weighted soft-max (the strongest nearby
cannibalizer dominates, not a diluting mean). Empty portfolio -> labelled proxy
fallback, confidence forced low, never presented as the true index.
Pure scoring fns unit-tested without DB; None-not-0 on missing axes; thin/
only-current portfolio -> low confidence + §26 note; deterministic (sorted
tie-breaks, no RNG). class+price+geo active now; unit-mix+timing plumbed via
optional params for a follow-up that wires them from the orchestrator horizon.
ruff + mypy clean; 151 special-index tests pass (964 forecasting dir, no regr).
Refs #1169
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
demand_index used a fixed clamp01(velocity/50); on a live prod refresh
all 8 ЕКБ districts sold ≥50/mo so it saturated to 1.0 everywhere — zero
discrimination between districts. Redesign to mirror infra_index:
normalize each district's unit_velocity against the city reference (MAX
district velocity per refresh run), so demand always discriminates and
self-calibrates as the market grows (no magic constant to rot).
- normalize_demand(velocity, *, city_reference_velocity), pure + graceful
(None stays None; reference<=0 -> honest 0.0, no ZeroDivisionError)
- refresh_locations now two-pass: collect velocities (one
compute_market_metrics per district, no O(n^2)), derive city reference,
normalize + upsert; SAVEPOINT-per-row and counters preserved
- remove _DEMAND_SATURATION_UPM constant; log city_reference_velocity
- tests: rewrite demand normalization + add end-to-end city-relative
suite incl. discrimination regression guarding the all-1.0 prod bug
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
Insight = аналитик пишет free-form intel про локацию/участок с пометкой
«непублично» (is_confidential, §7.13/§8.9). CRUD под /api/v1/insights;
created_by из X-Authenticated-User (не из тела — спуфинг автора невозможен).
Зеркало custom_pois: raw-SQL service + Pydantic v2 schemas, sync def handlers
(threadpool, off event loop).
- data/sql/145_insight.sql: table insight (idempotent, geom GENERATED из lon/lat,
индексы district/cad_num/is_confidential/created_by/GIST); аддитивно расширяет
CHECK audit_log.action += insight_write (тот же constraint ck_audit_log_action,
to_regclass-guarded, superset — не ломает #962-аудит; НЕ правит applied 144).
- audit_middleware.classify_path: method-aware insight_write для POST/PUT/DELETE
/api/v1/insights (GET-чтения не аудируются); обратно совместимо с parcels.
- ACL: backend rbac_guard hard-блокирует только /api/v1/admin/*; pilot отсекается
frontend RouteGuard + Caddy (как parcels/custom_pois). is_confidential — стораемая
пометка без per-record backend-гейта (analyst видит, per #962-политике).
- tests: 28 insight (CRUD+filters+required+#261 commit) + 5 audit. 86 зелёных, ruff.
Part B (Location first-class, §8.2) — отдельный follow-up.
Refs #948.
EPIC18/§19. analyst sees everything (deals, insights, exports, site-finder,
analytics, concept) EXCEPT admin/data-management. Enforcement is backend-hard
(the existing rbac_guard already 403s any non-admin role on /api/v1/admin/*, so
adding the role auto-blocks it) + frontend (deny_paths via /me) + audit.
§19 audit: new best-effort HTTP middleware logs the sensitive actions
(analyze / forecast / forecast-export) to a new audit_log table after the
response. Audit failures never break or delay-fail the request (2-layer
try/except + finally close). Registered INNER to rbac_guard so only authorized
requests are audited (a 403 short-circuits before audit). classify_path matches
export before forecast (anchored).
- auth/roles.yaml: analyst role (paths /**, deny admin-mgmt) + analysttest QA user
- core/auth.py (+ tradein mirror): Role Literal += analyst
- core/audit_middleware.py (new) + main.py registration
- data/sql/144_audit_log.sql (idempotent; auto-applies)
- tests: analyst rbac (403 admin / 200 parcels) + 11 audit cases
No data-level ACL (analyst sees data per policy) -> no #948 dependency. No new
deps. parcels.py untouched. Real analyst logins still need adding to
caddy/users.caddy.snippet (devops).
Refs #962.
fedstat ИПЦ is reCAPTCHA-blocked; CBR publishes inflation openly. Add
fetch_inflation + parse_inflation_xlsx (CBR UniDbQuery DownloadExcel/132934,
monthly % г/г, region=rf, source=cbr) to cbr_macro.py; upsert
indicator_type=inflation_yoy via the existing cbr_macro_sync task (per-series
guard, SAVEPOINT-per-row, CAST not ::, ON CONFLICT on the PK).
Surface inflation_yoy in MonthlyMacro (frozen, carry-forward) and ACTIVATE the
reserved §9.5 inflation channel (macro_coefficient f_inflation: level-vs-4%-target
nudge, non-positive to avoid double-counting f_rate, excluded from
_RATE_DRIVEN_FACTORS). Channel was DEGRADED (no data) -> now BACKED + consumed;
_CONF_HIGH_MIN_BACKED 4->5. Deterministic (§16/§26); renorm claims the reserved
0.08 slice as designed. Live-verified (2026-04 5.58%); 194 macro + 902 forecasting
tests green. No migration, no new deps.
Refs #946.
Cold §22 forecast measured ~215-233s on prod: §9.x layers re-execute the same
horizon/segment-invariant DB loads with identical args hundreds of times per
report (profiled: get_competitors x69, market_metrics x124, get_monthly_macro
x290). Add a per-report ContextVar cache (forecast_cache(), opened once in the
orchestrator) + @cached(key_builder) on the expensive §9.x loaders so each
unique load runs ONCE and reuses the same frozen, read-only instance.
Output is byte-identical (memoized producers are frozen dataclasses / read-only
Pydantic, callers never mutate; cache is per-report, discarded on exit; no-op
outside the report build). No concurrency, no signature changes.
- forecast_request_cache.py: ContextVar cache + cached() decorator (no-op
outside context, reentrant, _MISS sentinel for cached None)
- @cached on competitors/future_supply/market_metrics/macro_series/
sales_series/macro_coefficient/demand_normalization/regression loaders
- orchestrator: wrap build_site_finder_report in forecast_cache()
- 58 tests: key discrimination (call-counting regression guard), no-op-outside,
per-report isolation, reentrancy, frozen-producer canary, amplification proof
(real get_monthly_macro xN->1)
code-reviewer APPROVE (keys correct, mutation-safe, output identical). 1265
forecast/cache tests green. No new deps. Refs #1129.
render_report_docx (python-docx) mirrors report_md/report_pdf section order &
content, reuses report_pdf pure helpers (DRY), graceful on thin/empty report.
Widen /forecast/export format Literal to include docx → Word attachment.
Add python-docx dep + regenerate uv.lock (uv sync --frozen passes). Part of #959.
_http_get_json/_http_post_json → verify=False (const _VERIFY_TLS): геопортал ЕКБ отдаёт цепочку с росс-гос-CA, которого нет в trust-store контейнера → verify=True падал CERTIFICATE_VERIFY_FAILED, весь geoportal-слой (C8b/D9b/#1085) не работал бы с прода. Данные публичные open-data, секреты не передаются. Прецедент: nspd_lite/ekburg_permits/pzz_loader. Прод-probe подтвердил. +регрессия-тест.
Refs #1067.
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
