One layer's caught DB error left the shared §22 report Session in Postgres's
"aborted transaction" state with no rollback, silently emptying every later
layer on the same report. Wrap each failure point in db.begin_nested()
(SAVEPOINT): orchestrator._safe_call, macro_series _query_key_rate_monthly /
_query_inflation_monthly / _query_mortgage_monthly (per indicator),
special_indices._run (per index), and sales_series _query_source_a /
_query_source_b — so a sibling builder's failure no longer cascades.
Also close the RELEASE-SAVEPOINT trap: inner helpers that swallowed a
db.execute failure without their own savepoint left the tx aborted, so the
outer builder's savepoint failed at RELEASE (illegal in an aborted tx) and
poisoning still cascaded. Add savepoints at the inner catch sites —
special_indices._query_parcel_centroid and district_resolver._admin_names /
resolve_objective_districts.
Refs #2464 (cluster A session-poisoning).
_SUPPLY_BATCH_SQL джойнил domrf_kn_flats по ОДНОЙ глобальной дате
(f.snapshot_date = MAX(snapshot_date) по всей таблице). Но domrf_kn_flats —
ПО-ОБЪЕКТНЫЙ time-series: каждый ЖК скрейпится в свой день. На единственной
глобал-max дате присутствует обычно 1 объект → у остальных 0 квартир →
supply_units_in_radius=0 для всех строк 4.2 Планировки → frontend показывал
«Срок продажи 0 мес» и «% продано —». Регрессия от #1944 (objects-first
дедуп snapshot'ов объектов, который сам по себе корректен).
Фикс: flats_latest CTE (DISTINCT ON (obj_id) ... ORDER BY obj_id,
snapshot_date DESC, id DESC) берёт для КАЖДОГО obj_id его собственный
последний снимок и джойнится к nearby. objects-first MATERIALIZED дедуп
(#1944) сохранён → fan-out по снимкам не возвращается. Глобальный
db.scalar(MAX(snapshot_date)) + :latest_snap bind удалены.
Прод (66:41:0205010:287, r=1км, 9 объектов): supply 0 (global-max) → 2675
(per-object, 4 объекта имеют flats на разных датах 2026-05-17/05-05; ни один
не на глобал-max 2026-06-22). Данные flats частично сломаны (#1945, отдельно),
но фикс корректно двигает supply с 0 к реальным per-object числам.
Тесты: новый guard test_supply_joins_flats_per_object_latest_snapshot;
обновлены mock-фабрики (db.scalar больше не вызывается).
Корень «−1.00 везде» (эпик #1953): compute_demand_supply_forecast брал
district-wide unit_velocity (847.5/мес, ВСЕ классы/комнаты) как спрос и
весь district-сток (~63k доступных) как предложение для КАЖДОЙ ячейки
what_to_build → один и тот же ratio во всех ячейках → все deficit_index
прижаты к −1.0. Плюс objective_lots — append-per-snapshot (~2.9× инфляция
строк), что симметрично раздувало обе базы → даже сегментация без дедупа
осталась бы вырожденной.
Фикс (blast radius — ТОЛЬКО forecast/deficit calc; platform-wide dedup = #1964):
- market_metrics.compute_market_metrics: +obj_class/+room_bucket (+cache key).
_STOCK_SQL и _SALES_WINDOW_SQL дедуплят до ПОСЛЕДНЕГО снапшота на физлот
(DISTINCT ON project_name,corpus_name,section,floor,lot_number ORDER BY …
snapshot_date DESC,id DESC), затем агрегируют. Class-фильтр (LOWER=LOWER,
class lowercase) + room-bucket (Source-B room_area-вокабуляр, зеркало
sales_series.room_area_bucket_of → what_to_build фильтрует без перевода).
ROLLUP/GROUPING сохранён; confidence считается на дедуплицированных counts.
- demand_supply_forecast: base_pace и open-сток теперь ПОСЕГМЕНТНЫЕ
(market_metrics(obj_class,room_bucket)). При заданном сегменте L2/L3
(hidden/future) ИСКЛЮЧЕНЫ из баланса — они класс/формат-агностичны, иначе
двоились бы по всем ячейкам. +_market_room_bucket VOCAB-мост (валидирующий
pass-through Source-B меток; неизвестное → None = без фильтра, не тихий 0-rows).
- what_to_build/_DEFAULT_CLASSES и recommendation Economy-маппинг: «эконом»→
«стандарт» (в objective_lots эконома НЕТ, стандарт=483k → раньше ячейка
матчила 0 строк и молча выпадала).
- report_assembler honesty-guard: если ВСЯ сетка прижата к ±1.0
(degenerate-fallback) — не эмитим «строить»/«избегать», показываем
«недостаточно гранулярных данных для посегментного вывода».
- data/sql/173_objective_lots_physflat_idx.sql: partial index под DISTINCT ON
(Index Only Scan + Unique, без Sort на 1.75M строк; idempotent, BEGIN/COMMIT).
Prod-verify (parcel 66:41:0205010:287, Железнодорожный, h=24): ячейки
ДИФФЕРЕНЦИРУЮТ (12 measured, 7 distinct) вместо all −1.0; MOI комфорт/студия
38.5 vs стандарт/студия 244.3 (точное совпадение с ожидаемым).
Тесты: регрессия «ячейки различаются (не all −1.0)» + vocab-translation +
honesty-guard + посегментное предложение. ruff clean; no :name::type.
domrf_kn_objects is a snapshot dimension (UNIQUE (obj_id, snapshot_date), ~8
snapshots/obj_id). _SUPPLY_BATCH_SQL joined flats to ALL object-snapshot rows
(no o.snapshot_date filter), counting each flat ~8.5x → supply_units_in_radius
inflated ~8.5x, sold_pct_of_supply deflated ~8.5x, is_oversold under-fired
(all user-facing, best_layouts.py:571-611; sold_pct=deals/supply is a raw
ratio so no canceling).
Fix: dedup objects to one row per obj_id (latest-snapshot coords) via
DISTINCT ON in an objects-first MATERIALIZED CTE, then join domrf_kn_flats via
idx_kn_flats_obj. units now = one count per flat (prod cross-check at radius
1.5km: units == count(*) == count(DISTINCT f.id) == 9612 for 65 objects;
correction factor 8.56x at 1.5km, 9.13x at 1.0km). This also aligns the supply
denominator with the deals numerator (_COMPETITORS_IN_RADIUS_SQL already uses
DISTINCT ON latest snapshot).
Perf bonus: objects-first avoids the parallel seq scan of the ~376k-row flats
snapshot. radius 1.5km / snapshot 2026-05-17: 240ms/~28k buffers/6712 disk
reads -> 49ms/1554 buffers/0 disk reads (~5x).
Tests: add SQL-text fan-out guard (DISTINCT ON + MATERIALIZED, no bare
flats->objects join); update stale EXPLAIN mirror in test_phantom_columns.
USER-FACING: best-layouts supply/sold_pct/is_oversold/sell-out-months shift
~8.5x toward correct (frontend BestLayoutsBlock only; ТЗ recommendation + PDF
unchanged — they derive from sum_deals, not supply). Deep-reviewed (APPROVE).
New price tier objective_geo_radius: ST_DWithin median of Objective new-build prices (objective_lots ⋈ complexes) within 3km of parcel centroid, between quarter-MV and district_reference. Closes the name-match gap (5 of 9 EKB districts had no Objective name-match). data/sql/168 functional GIST index (prod EXPLAIN 114ms→28ms). Degenerate-centroid guard + honest RU price_source captions. Deep-review ✅.
Refs #1881
Add `velocity_by_room: dict[str, float] | None` to `MarketMetrics` — per-bucket
unit velocity (ед./мес) derived from the existing `sold_by_room` ROLLUP data that
`_query_sales_window` already returns. No new SQL required.
Thread per-bucket velocity through `_demand_only_overlay` via the new
`_FORECAST_TO_METRIC_BUCKETS` constant that maps each forecast bucket to its
market_metrics room-bucket keys. "80+ м²" sums "4" + "5+" keys. Fallback to
aggregate `unit_velocity` when `velocity_by_room` is None (thin-data path).
Previously `base_pace` was identical for all 5 room-buckets, so §9.4 norm and §9.2
base_pace cancelled out in pace/max_pace and ranking was driven purely by §9.5
macro_coef (segment steepness proxy). Now §9.2 reflects real per-bucket observed
demand from objective_lots.contract_date data.
Callers of `compute_market_metrics` that don't use `velocity_by_room` are unaffected
(the new field is additive to the frozen dataclass). All existing callers verified —
none construct `MarketMetrics` directly except the one production site.
_price_sensitivity передавал сырое admin-имя ('Кировский') в _elasticity_coef,
который фильтрует objective_corpus_room_month.district по МИКРО-вокабуляру
(Втузгородок, ЖБИ, …) → регрессия получала 0 точек → всегда FALLBACK_ELASTICITY.
§9.2 district-level эластичность молча НЕ считалась в /analyze-пути (только
'Академический' совпадал в обоих вокабулярах случайно).
Fix: вызываем resolve_objective_districts() в _price_sensitivity и передаём
список микро через новый kwarg districts=[…] в _elasticity_coef. Резолвер
None ('не определён' / нет чистых алиасов) → пустой список → EKB-wide
регрессия. _elasticity_coef расширен с back-compat: districts=None →
legacy путь по district_name (другой caller в analytics_queries —
отдельный bug class, вне scope).
5 новых юнит-тестов TestPriceSensitivityDistrictResolution: admin→micros в
SQL bind, None→EKB-wide, regression preserved post-resolve, graceful.
76/76 market_metrics + 156/156 elasticity/sensitivity тестов зелёные.
ruff + psycopg v3 grep clean.
Closes#1211
_SALES_WINDOW_SQL делал GROUP BY ROLLUP (rooms_int), rooms_int nullable
(ETL пишет NULL для «неопределённого типа», sales_series.py:399 явно
обрабатывает None). Проданный лот с rooms_int IS NULL даёт ДВЕ строки
rooms_int IS NULL (NULL-группа + grand-total итог), неразличимые в
Python (оба if r["rooms_int"] is None).
MixedAggregate-план PG16 эмитит grand-total ПЕРВЫМ (среди hash-строк),
NULL-группа после → loop затирает units_total частичным счётом (живой
тест на PG16: 2000 → 200). Эффект: unit_velocity / absorption_rate
занижены, months_of_supply завышен → base_pace в demand_supply_forecast
неверный (recommendation.py:586) → reports/scoring врёт.
Patch:
- SQL: добавить GROUPING(rooms_int) AS is_total (=1 для grand-total).
- Python: ветвить по is_total, NULL-комнатную группу класть в
by_room['unknown'] (отдельный бакет), аккумулировать через +=
вместо assign (защита от будущих NULL-вариантов).
- Тесты: моки получили "is_total" поле (1 для grand-total, 0 иначе).
71/71 market_metrics тестов зелёные. ruff clean.
Closes#1214
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
demand_index used a fixed clamp01(velocity/50); on a live prod refresh
all 8 ЕКБ districts sold ≥50/mo so it saturated to 1.0 everywhere — zero
discrimination between districts. Redesign to mirror infra_index:
normalize each district's unit_velocity against the city reference (MAX
district velocity per refresh run), so demand always discriminates and
self-calibrates as the market grows (no magic constant to rot).
- normalize_demand(velocity, *, city_reference_velocity), pure + graceful
(None stays None; reference<=0 -> honest 0.0, no ZeroDivisionError)
- refresh_locations now two-pass: collect velocities (one
compute_market_metrics per district, no O(n^2)), derive city reference,
normalize + upsert; SAVEPOINT-per-row and counters preserved
- remove _DEMAND_SATURATION_UPM constant; log city_reference_velocity
- tests: rewrite demand normalization + add end-to-end city-relative
suite incl. discrimination regression guarding the all-1.0 prod bug
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
/analyze passes the official ЕКБ admin district (ekb_districts polygon, e.g.
'Кировский'), but objective_lots/corpus_room_month store informal micro-districts
('Втузгородок','ЖБИ') -> admin name matched 0 rows -> silent empty forecast.
Add resolve_objective_districts() (site_finder/district_resolver.py) mapping an
admin name to its clean micros via ekb_district_alias (note IS NULL), with
None -> EKB-wide fallback and raw-micro pass-through. Wire into the objective_lots
district filters of market_metrics (§9.2 stock+sales), supply_layers L1 (§9.3),
and sales_series Sources A+B (crm shares the micro vocab, prod-verified),
switching the scalar filter to psycopg3-safe = ANY(CAST(:districts AS text[])).
supply_layers L2/L3 keep the admin name (domrf_kn_objects.district_name is admin vocab).
Prod: Кировский/Ленинский/Орджоникидзевский obj_count 0 -> 32/64/31.
Tests mutation-verified non-vacuous. 192 module tests pass; ruff clean. Refs #969#949.
FOLLOW-UP (Leha: not a bug, follow-up).
get_macro_series / get_latest_macro defaulted region='rf', but CBR mortgage_*
series live under 'sverdl' in macro_indicator (key_rate under 'rf'). Only caller
passes region explicitly so no active leak, but a future caller omitting region
would silently get None — latent footgun. region now str|None; when None →
_canonical_region (mortgage set sourced from _CBR_SERIES_MAP, single source of
truth). Explicit region always wins → existing callers unchanged. Region-binding
tests added. 159 tests pass.
Documented (NOT code-fixed — upstream data-quality): mortgage_debt /
mortgage_overdue stay empty in macro_indicator because cbr_mortgage_series.period
holds value-like garbage (e.g. '10054588.0') for the Debt series — corruption
from the upstream CBR-XLSX scraper that built domrf.db (imported verbatim by
44_import_anton_db.py), NOT an ingest bug. 123_macro_indicator.sql correctly
skips unparseable periods. Needs a separate scraper re-ingest (idempotent
backfill once period is valid). Refs #945
REOPENED. _SALES_WINDOW_SQL derived "sales in window" from objective_lots_history
snapshots, but history is only ~17 days deep — every currently-sold lot had a
sold-snapshot in the window, so window-sales collapsed into the entire cumulative
sold stock (Автовокзал 6mo: 33,245 vs real ~2,308). Inflated absorption_rate
(~235%/mo with confidence=high), months_of_supply, unit_velocity, liquidity,
demand_concentration → contaminated forecast #950/#952.
Count window sales directly from objective_lots by contract_date in the window
(the real sale date — present on 100% of sold lots: 41,091/41,091). Return
contract of _query_sales_window unchanged (units/area/by-room ROLLUP); downstream
formulas untouched. Removed the now-dead objective_lots_history JOIN/CTE.
Regression test: lots sold outside window (contract_date out of range) not counted
(41,091 cumulative vs 2,308 window → absorption 2.35→0.04). 288 tests green.
Verification = prod compute_market_metrics(Автовокзал) post-deploy. Refs #949
REOPENED. L3 future-supply rows are computed per (district_name, dev_group_name)
but dev_group_name was never a key column — only embedded in method text. With
complex_id/obj_class NULL for L3, every dev_group of a district collapsed to one
upsert key → ~95.6% loss. Ground-truth (Академический, prod): should be 13,808
units / 15 dev_groups / 54 objects; only 1 row / 607 units survived.
Migration 128: ADD COLUMN supply_layers.dev_group_name TEXT + rebuild
uq_supply_layers_logical to (layer, district_name, complex_id, obj_class,
dev_group_name, source, snapshot_date) NULLS NOT DISTINCT (L1/L2 dev_group_name
NULL stays transparent → their dedup unchanged; L3 distinct groups no longer
collapse). Dry-run-verified vs prod catalog (applies clean, ROLLBACK clean).
Worker: SupplyLayerRow gains dev_group_name (L1/L2=None, L3=group); _UPSERT_SQL
adds it to INSERT/VALUES (CAST(:dev_group_name AS text)) + ON CONFLICT (key col,
not in DO UPDATE SET). Service+worker regression tests assert same-district/
different-dev_group → distinct keys (no collapse). 234 supply tests pass.
Deploy applies migration before container restart; collapsed data self-heals on
next supply_layers_refresh. Verification = prod re-measure post-deploy.
Refs #970
Round 1 (commit bcd7dc8) был broken: на 2-bucket входах surplus уходил в free
полностью без учёта capacity → free превышал cap → следующая итерация
clamp'ировала его и наоборот. Infinite oscillation в FastAPI handler.
Round 2 fix per review BLOCK (#282 comment):
- Surplus распределяется пропорционально available capacity (cap - v),
не текущему v. Free никогда не вылетит выше cap.
- free = строго < cap (не <=) — иначе деление на 0 capacity.
- Hard guard `for _ in range(N+1)` — гарантированно завершается.
- Pathological (surplus > total_capacity): возвращаем оригинальный pct_map
+ cap_skipped=True (sum=100 invariant сохранён).
- Hamilton round вынесен в _hamilton_round() helper.
Tests:
- 2-bucket cases (90/10, 70/30, 99/1) expected cap_skipped=True
- test_cap_iteration_count_bounded — все pathological завершаются < 100ms
- All 13 cases verified standalone (3 fast-path + 7 reproduced + 3 pathological)
Раньше _VELOCITY_DIVISORS делил агрегаты mv_layout_velocity (24 мес)
на 4/12 для quarter/year, не меняя реальное окно данных. Теперь
inline SQL из objective_corpus_room_month с CAST(:window_interval AS interval).
velocity_per_month = deals_window / months_in_window (1.0/3.0/12.0).
Разные time_window → разные строки из БД → разный mix/velocity/jk_count.
Closes (epic part) #271 item 1
