Three #1953 audit follow-ups:
- perf: add migration 174 — composite index idx_kn_flats_obj_snap on
domrf_kn_flats (obj_id, snapshot_date DESC). Serves the #1956 flats_latest
CTE in best_layouts.py (DISTINCT ON (obj_id) ORDER BY obj_id, snapshot_date
DESC + self-join on (obj_id, snapshot_date)); previously only (obj_id)
existed so Postgres sorted per object. Prod: 789 569 rows, idx ~5.7 MB,
dry-run instant. Idempotent, self-wrapped BEGIN/COMMIT.
- frontend: route every max_height_m read through coerceFloat (same string-bug
class as max_far #1962). max_height_m is NUMERIC → arrives as a string on the
wire; ptica-adapt.ts read it raw at 4 sites and relied on formatInt/Math.round
coercion. Widen the type in nspd.ts and fix the stale "real number" comment in
nspd-regulation.ts.
- frontend: hide the «Дата регистрации» EGRN row entirely when
registration_date is null (~97% of parcels) instead of rendering a bare «—».
Корень «−1.00 везде» (эпик #1953): compute_demand_supply_forecast брал
district-wide unit_velocity (847.5/мес, ВСЕ классы/комнаты) как спрос и
весь district-сток (~63k доступных) как предложение для КАЖДОЙ ячейки
what_to_build → один и тот же ratio во всех ячейках → все deficit_index
прижаты к −1.0. Плюс objective_lots — append-per-snapshot (~2.9× инфляция
строк), что симметрично раздувало обе базы → даже сегментация без дедупа
осталась бы вырожденной.
Фикс (blast radius — ТОЛЬКО forecast/deficit calc; platform-wide dedup = #1964):
- market_metrics.compute_market_metrics: +obj_class/+room_bucket (+cache key).
_STOCK_SQL и _SALES_WINDOW_SQL дедуплят до ПОСЛЕДНЕГО снапшота на физлот
(DISTINCT ON project_name,corpus_name,section,floor,lot_number ORDER BY …
snapshot_date DESC,id DESC), затем агрегируют. Class-фильтр (LOWER=LOWER,
class lowercase) + room-bucket (Source-B room_area-вокабуляр, зеркало
sales_series.room_area_bucket_of → what_to_build фильтрует без перевода).
ROLLUP/GROUPING сохранён; confidence считается на дедуплицированных counts.
- demand_supply_forecast: base_pace и open-сток теперь ПОСЕГМЕНТНЫЕ
(market_metrics(obj_class,room_bucket)). При заданном сегменте L2/L3
(hidden/future) ИСКЛЮЧЕНЫ из баланса — они класс/формат-агностичны, иначе
двоились бы по всем ячейкам. +_market_room_bucket VOCAB-мост (валидирующий
pass-through Source-B меток; неизвестное → None = без фильтра, не тихий 0-rows).
- what_to_build/_DEFAULT_CLASSES и recommendation Economy-маппинг: «эконом»→
«стандарт» (в objective_lots эконома НЕТ, стандарт=483k → раньше ячейка
матчила 0 строк и молча выпадала).
- report_assembler honesty-guard: если ВСЯ сетка прижата к ±1.0
(degenerate-fallback) — не эмитим «строить»/«избегать», показываем
«недостаточно гранулярных данных для посегментного вывода».
- data/sql/173_objective_lots_physflat_idx.sql: partial index под DISTINCT ON
(Index Only Scan + Unique, без Sort на 1.75M строк; idempotent, BEGIN/COMMIT).
Prod-verify (parcel 66:41:0205010:287, Железнодорожный, h=24): ячейки
ДИФФЕРЕНЦИРУЮТ (12 measured, 7 distinct) вместо all −1.0; MOI комфорт/студия
38.5 vs стандарт/студия 244.3 (точное совпадение с ожидаемым).
Тесты: регрессия «ячейки различаются (не all −1.0)» + vocab-translation +
honesty-guard + посегментное предложение. ruff clean; no :name::type.
FIX A (#1955) «Что хорошо продаётся»: убран фантомный класс 'Comfort'.
- Миграция 172: v_bucket_success_score COALESCE(obj_class,'Comfort')
→ COALESCE(obj_class,'не указан'). Английский литерал заполнял 397 NULL
и сливался отдельным классом от русского 'Комфорт' → визуальные дубли
бакетов в UI. Источник уже канонически-русский (проверено на проде),
synonym-mapping не нужен.
- parcels.py: obj_class протаскивается в success-ranking query + dict.
- TS SuccessRankingBucket.obj_class добавлен.
FIX B (#1960) «Медиана рынка» = 64k (квартальная росреестровская n=1 ДКП):
- district.median_price_per_m2 больше не COALESCE(median_12m, ekb_ref) в SQL.
Basis-приоритет (newbuild-first): Objective по имени района →
geo_radius (Objective в 3км) → ekb_districts reference →
квартальная росреестровская медиана ТОЛЬКО при deals_count≥5.
Для 66:41:0205010:287: 64k → 132690 (geo_radius, newbuild-consistent).
- median_price_basis добавлен в payload + TS type (nullable median).
- Frontend null-guards для нового nullable median.
Tests: +4 (geo_radius basis, objective-приоритет, deals-guard, obj_class
passthrough); обновлены district-моки в 9 analyze-тестах под новую
SQL-сигнатуру.
The objective_corpus_room_month index migration (#1942) was committed to
backend/data/sql/171_*, but the main deploy runner applies migrations from
repo-root data/sql/*.sql (deploy.yml:280) and the path trigger is data/sql/**.
So the file never ran — prod still seq-scans (verified post-deploy: index
absent, query 265ms). Move it to data/sql/171_* (alongside 170) so it deploys.
No SQL change; the index itself was dry-run-verified on prod (BEGIN/ROLLBACK).
New price tier objective_geo_radius: ST_DWithin median of Objective new-build prices (objective_lots ⋈ complexes) within 3km of parcel centroid, between quarter-MV and district_reference. Closes the name-match gap (5 of 9 EKB districts had no Objective name-match). data/sql/168 functional GIST index (prod EXPLAIN 114ms→28ms). Degenerate-centroid guard + honest RU price_source captions. Deep-review ✅.
Refs #1881
Adds period_type TEXT NOT NULL DEFAULT 'unknown' to macro_indicator and
widens the PRIMARY KEY to (indicator_type, region, obs_date, period_type).
Before: yearly aggregate ('год'→obs_date YYYY-01-01) and Q1 ('I квартал'→
same date) shared the same PK slot → ON CONFLICT DO UPDATE made them
overwrite each other despite the in-memory dedup fix in #1687.
After: each EmissRow carries period_type from _emiss_period_granularity
('year'/'quarter'/'month'); DB ON CONFLICT targets include it, so
yearly+Q1 rows genuinely coexist.
Non-EMISS sources (CBR, rosstat open-data, domrf) use period_type='unknown'
(literal in SQL) — they are disambiguated by obs_date already, so the
wider PK is backward-compatible. All ON CONFLICT clauses in
cbr_macro_sync.py and rosstat_macro_sync.py updated accordingly.
Migration: 163_emiss_pk_period_type.sql (idempotent BEGIN/COMMIT).
Tests: 49 passed (emiss + rosstat suite), ruff clean, py_compile OK.
sat_factor = 1 + ((sold_pct-50)/100)*0.30 was computed in 09_macro_and_trend.py,
written to district_economics.sat_factor, and fetched in server.py and 10_score_v2.py
— but never multiplied into any score. The live market sub-score uses a separate
sat_score = min(100, sold_pct*100/70) directly, so sat_factor was dead code that
would double-count absorption if ever wired in.
- 09_macro_and_trend.py: remove sat_factor computation, ALTER TABLE column, UPDATE
binding, and debug print column
- 10_score_v2.py: remove sat_factor from SELECT and unpacking
- server.py: remove sat_factor variable assignment and from macro_factors response
- static/index.html: remove sat_factor documentation row
- data/sql/162_drop_district_economics_sat_factor.sql: DROP COLUMN IF EXISTS
Replace ST_Intersects(district.geom, quarter.geom) with ST_Within(ST_Centroid(quarter.geom), district.geom)
so each cadastral quarter is assigned to exactly one district. Eliminates 85 double-counted quarter-district
pairs and 307 spurious deal-district rows that were skewing PERCENTILE_CONT medians. Function signature
unchanged; no backend code changes required.
Правки уже-применённых миграций (85/062/080) не долетают до прода через
_schema_migrations. Дописаны отдельные миграции на живых таблицах:
- 158: pzz_zones_ekb UNIQUE NULLS NOT DISTINCT (rosreestr_id) (#1361)
- tradein 108: повторный backfill Avito-адресов улучшенным regexp (#1419)
- tradein 109: пересчёт asking_to_sold_ratios secondary-only (#1364/#1186)
Все три dry-run (BEGIN..ROLLBACK) против прод-схемы: PSQL_RC=0.
P0 goldmine: objective_lots (1.12M, 81% цен) связывается с domrf для velocity/
layout-аналитики. Mapping было 148 → +115 (107 high + 8 mid) = 263.
Fuzzy-match objective(project_name) ↔ domrf_kn_objects(comm_name) LATEST snapshot
region_cd=66 через pg_trgm на НОРМАЛИЗОВАННЫХ именах (strip ЖК-boilerplate с
\m\M word-boundary). Правило приёмки dev-corroborated: nsim≥0.62 AND dsim≥0.5 —
precision over recall для P0-цен (generic-имена с разным застройщиком, напр.
Солнечный/Прекрасный/Стрелки dsim=0.0, отклонены). match_method='auto_fuzzy_v2',
is_reviewed=false (для human-review топ-кандидатов), confidence-tier в note.
Idempotent ON CONFLICT (objective_complex_name, objective_group). REFRESH
CONCURRENTLY mv_layout_velocity после COMMIT.
Refs #307
UNIQUE (obj_id, doc_type, doc_num, file_url) в м.112 без NULLS NOT
DISTINCT, doc_num nullable. В PG NULL != NULL в уникальном индексе →
для документов с docNum=null (документированный частый вход:
extract_documents:125-134 → doc_num=None) ON CONFLICT-арбитр в
scrapers/documents.py:59 НИКОГДА не матчился → каждый ежедневный sweep
domrf_kn.py:1903 молча вставлял новую строку-дубль вместо UPDATE.
Таблица росла ×N прогонов; дубли уходили в analytics_queries.py:1068.
Миграция 140:
1. Дедуп существующих NULL-дублей: ROW_NUMBER() PARTITION BY все 4 ключа
ORDER BY scraped_at DESC, id DESC → оставляем самую свежую, удаляем
остальные WHERE rn>1 AND doc_num IS NULL.
2. DROP старого autonamed constraint'а IF EXISTS.
3. ADD uq_domrf_kn_documents_logical UNIQUE NULLS NOT DISTINCT (...) с
именованным guard по pg_constraint.conname (паттерн м.125 → idempotent).
Caller scrapers/documents.py НЕ меняется — после миграции ON CONFLICT-
арбитр начнёт матчить NULL-doc_num строки автоматически. Прецедент того
же класса: м.110 (uq_infra_dedupe), м.125 (uq_supply_layers_logical).
Closes#1206
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
The demand_index DB column comment still said "насыщающее преобразование"
(saturating) after #1167 switched it to city-relative normalization —
exactly the misleading-comment class that hid the original bug. Migration
147 updates COMMENT ON COLUMN (idempotent, comment-only, no data DDL).
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
Insight = аналитик пишет free-form intel про локацию/участок с пометкой
«непублично» (is_confidential, §7.13/§8.9). CRUD под /api/v1/insights;
created_by из X-Authenticated-User (не из тела — спуфинг автора невозможен).
Зеркало custom_pois: raw-SQL service + Pydantic v2 schemas, sync def handlers
(threadpool, off event loop).
- data/sql/145_insight.sql: table insight (idempotent, geom GENERATED из lon/lat,
индексы district/cad_num/is_confidential/created_by/GIST); аддитивно расширяет
CHECK audit_log.action += insight_write (тот же constraint ck_audit_log_action,
to_regclass-guarded, superset — не ломает #962-аудит; НЕ правит applied 144).
- audit_middleware.classify_path: method-aware insight_write для POST/PUT/DELETE
/api/v1/insights (GET-чтения не аудируются); обратно совместимо с parcels.
- ACL: backend rbac_guard hard-блокирует только /api/v1/admin/*; pilot отсекается
frontend RouteGuard + Caddy (как parcels/custom_pois). is_confidential — стораемая
пометка без per-record backend-гейта (analyst видит, per #962-политике).
- tests: 28 insight (CRUD+filters+required+#261 commit) + 5 audit. 86 зелёных, ruff.
Part B (Location first-class, §8.2) — отдельный follow-up.
Refs #948.
EPIC18/§19. analyst sees everything (deals, insights, exports, site-finder,
analytics, concept) EXCEPT admin/data-management. Enforcement is backend-hard
(the existing rbac_guard already 403s any non-admin role on /api/v1/admin/*, so
adding the role auto-blocks it) + frontend (deny_paths via /me) + audit.
§19 audit: new best-effort HTTP middleware logs the sensitive actions
(analyze / forecast / forecast-export) to a new audit_log table after the
response. Audit failures never break or delay-fail the request (2-layer
try/except + finally close). Registered INNER to rbac_guard so only authorized
requests are audited (a 403 short-circuits before audit). classify_path matches
export before forecast (anchored).
- auth/roles.yaml: analyst role (paths /**, deny admin-mgmt) + analysttest QA user
- core/auth.py (+ tradein mirror): Role Literal += analyst
- core/audit_middleware.py (new) + main.py registration
- data/sql/144_audit_log.sql (idempotent; auto-applies)
- tests: analyst rbac (403 admin / 200 parcels) + 11 audit cases
No data-level ACL (analyst sees data per policy) -> no #948 dependency. No new
deps. parcels.py untouched. Real analyst logins still need adding to
caddy/users.caddy.snippet (devops).
Refs #962.
