Цена продажи в финмодели калибровалась по name-match админ-района Objective —
но только 4 из 9 районов ЕКБ совпадают по имени, 5 проваливались в class_norm.
Добавлен ценовой tier "objective_geo_radius": ST_DWithin-медиана цен новостроек
Objective (objective_lots ⋈ complexes по lat/lon) в радиусе 3 км от центроида
участка, между квартальной MV и district_reference. Порог n≥10 лотов и ≥2 ЖК
(один ЖК не задаёт цену). Прод: 5 «гео»-районов теперь дают реальную медиану
(129–158k ₽/м²) вместо NULL.
- data/sql/168: функциональный GIST-индекс complexes((ST_MakePoint(lon,lat))::geography)
— без него ST_DWithin делает Seq Scan (+~90мс на /analyze, у которого уже
таймаут-давление #1850). Прод EXPLAIN: Seq Scan 114мс → Index Scan 28мс.
- guard: при вырожденном центроиде (fallback на центр ЕКБ) гео-радиус
пропускается (source=no_data_centroid_fallback) — не выдаём центр-города
медиану за калиброванную цену участка.
- price_source="objective_geo_radius" (plain str, без schema-change).
- ptica-adapt: price_source → честные RU-подписи (не сырой токен).
- +4 теста (4-tier priority chain). deep-review ✅.
Refs #1881
Adds period_type TEXT NOT NULL DEFAULT 'unknown' to macro_indicator and
widens the PRIMARY KEY to (indicator_type, region, obs_date, period_type).
Before: yearly aggregate ('год'→obs_date YYYY-01-01) and Q1 ('I квартал'→
same date) shared the same PK slot → ON CONFLICT DO UPDATE made them
overwrite each other despite the in-memory dedup fix in #1687.
After: each EmissRow carries period_type from _emiss_period_granularity
('year'/'quarter'/'month'); DB ON CONFLICT targets include it, so
yearly+Q1 rows genuinely coexist.
Non-EMISS sources (CBR, rosstat open-data, domrf) use period_type='unknown'
(literal in SQL) — they are disambiguated by obs_date already, so the
wider PK is backward-compatible. All ON CONFLICT clauses in
cbr_macro_sync.py and rosstat_macro_sync.py updated accordingly.
Migration: 163_emiss_pk_period_type.sql (idempotent BEGIN/COMMIT).
Tests: 49 passed (emiss + rosstat suite), ruff clean, py_compile OK.
sat_factor = 1 + ((sold_pct-50)/100)*0.30 was computed in 09_macro_and_trend.py,
written to district_economics.sat_factor, and fetched in server.py and 10_score_v2.py
— but never multiplied into any score. The live market sub-score uses a separate
sat_score = min(100, sold_pct*100/70) directly, so sat_factor was dead code that
would double-count absorption if ever wired in.
- 09_macro_and_trend.py: remove sat_factor computation, ALTER TABLE column, UPDATE
binding, and debug print column
- 10_score_v2.py: remove sat_factor from SELECT and unpacking
- server.py: remove sat_factor variable assignment and from macro_factors response
- static/index.html: remove sat_factor documentation row
- data/sql/162_drop_district_economics_sat_factor.sql: DROP COLUMN IF EXISTS
Replace ST_Intersects(district.geom, quarter.geom) with ST_Within(ST_Centroid(quarter.geom), district.geom)
so each cadastral quarter is assigned to exactly one district. Eliminates 85 double-counted quarter-district
pairs and 307 spurious deal-district rows that were skewing PERCENTILE_CONT medians. Function signature
unchanged; no backend code changes required.
Правки уже-применённых миграций (85/062/080) не долетают до прода через
_schema_migrations. Дописаны отдельные миграции на живых таблицах:
- 158: pzz_zones_ekb UNIQUE NULLS NOT DISTINCT (rosreestr_id) (#1361)
- tradein 108: повторный backfill Avito-адресов улучшенным regexp (#1419)
- tradein 109: пересчёт asking_to_sold_ratios secondary-only (#1364/#1186)
Все три dry-run (BEGIN..ROLLBACK) против прод-схемы: PSQL_RC=0.
P0 goldmine: objective_lots (1.12M, 81% цен) связывается с domrf для velocity/
layout-аналитики. Mapping было 148 → +115 (107 high + 8 mid) = 263.
Fuzzy-match objective(project_name) ↔ domrf_kn_objects(comm_name) LATEST snapshot
region_cd=66 через pg_trgm на НОРМАЛИЗОВАННЫХ именах (strip ЖК-boilerplate с
\m\M word-boundary). Правило приёмки dev-corroborated: nsim≥0.62 AND dsim≥0.5 —
precision over recall для P0-цен (generic-имена с разным застройщиком, напр.
Солнечный/Прекрасный/Стрелки dsim=0.0, отклонены). match_method='auto_fuzzy_v2',
is_reviewed=false (для human-review топ-кандидатов), confidence-tier в note.
Idempotent ON CONFLICT (objective_complex_name, objective_group). REFRESH
CONCURRENTLY mv_layout_velocity после COMMIT.
Refs #307
UNIQUE (obj_id, doc_type, doc_num, file_url) в м.112 без NULLS NOT
DISTINCT, doc_num nullable. В PG NULL != NULL в уникальном индексе →
для документов с docNum=null (документированный частый вход:
extract_documents:125-134 → doc_num=None) ON CONFLICT-арбитр в
scrapers/documents.py:59 НИКОГДА не матчился → каждый ежедневный sweep
domrf_kn.py:1903 молча вставлял новую строку-дубль вместо UPDATE.
Таблица росла ×N прогонов; дубли уходили в analytics_queries.py:1068.
Миграция 140:
1. Дедуп существующих NULL-дублей: ROW_NUMBER() PARTITION BY все 4 ключа
ORDER BY scraped_at DESC, id DESC → оставляем самую свежую, удаляем
остальные WHERE rn>1 AND doc_num IS NULL.
2. DROP старого autonamed constraint'а IF EXISTS.
3. ADD uq_domrf_kn_documents_logical UNIQUE NULLS NOT DISTINCT (...) с
именованным guard по pg_constraint.conname (паттерн м.125 → idempotent).
Caller scrapers/documents.py НЕ меняется — после миграции ON CONFLICT-
арбитр начнёт матчить NULL-doc_num строки автоматически. Прецедент того
же класса: м.110 (uq_infra_dedupe), м.125 (uq_supply_layers_logical).
Closes#1206
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
The demand_index DB column comment still said "насыщающее преобразование"
(saturating) after #1167 switched it to city-relative normalization —
exactly the misleading-comment class that hid the original bug. Migration
147 updates COMMENT ON COLUMN (idempotent, comment-only, no data DDL).
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
Insight = аналитик пишет free-form intel про локацию/участок с пометкой
«непублично» (is_confidential, §7.13/§8.9). CRUD под /api/v1/insights;
created_by из X-Authenticated-User (не из тела — спуфинг автора невозможен).
Зеркало custom_pois: raw-SQL service + Pydantic v2 schemas, sync def handlers
(threadpool, off event loop).
- data/sql/145_insight.sql: table insight (idempotent, geom GENERATED из lon/lat,
индексы district/cad_num/is_confidential/created_by/GIST); аддитивно расширяет
CHECK audit_log.action += insight_write (тот же constraint ck_audit_log_action,
to_regclass-guarded, superset — не ломает #962-аудит; НЕ правит applied 144).
- audit_middleware.classify_path: method-aware insight_write для POST/PUT/DELETE
/api/v1/insights (GET-чтения не аудируются); обратно совместимо с parcels.
- ACL: backend rbac_guard hard-блокирует только /api/v1/admin/*; pilot отсекается
frontend RouteGuard + Caddy (как parcels/custom_pois). is_confidential — стораемая
пометка без per-record backend-гейта (analyst видит, per #962-политике).
- tests: 28 insight (CRUD+filters+required+#261 commit) + 5 audit. 86 зелёных, ruff.
Part B (Location first-class, §8.2) — отдельный follow-up.
Refs #948.
EPIC18/§19. analyst sees everything (deals, insights, exports, site-finder,
analytics, concept) EXCEPT admin/data-management. Enforcement is backend-hard
(the existing rbac_guard already 403s any non-admin role on /api/v1/admin/*, so
adding the role auto-blocks it) + frontend (deny_paths via /me) + audit.
§19 audit: new best-effort HTTP middleware logs the sensitive actions
(analyze / forecast / forecast-export) to a new audit_log table after the
response. Audit failures never break or delay-fail the request (2-layer
try/except + finally close). Registered INNER to rbac_guard so only authorized
requests are audited (a 403 short-circuits before audit). classify_path matches
export before forecast (anchored).
- auth/roles.yaml: analyst role (paths /**, deny admin-mgmt) + analysttest QA user
- core/auth.py (+ tradein mirror): Role Literal += analyst
- core/audit_middleware.py (new) + main.py registration
- data/sql/144_audit_log.sql (idempotent; auto-applies)
- tests: analyst rbac (403 admin / 200 parcels) + 11 audit cases
No data-level ACL (analyst sees data per policy) -> no #948 dependency. No new
deps. parcels.py untouched. Real analyst logins still need adding to
caddy/users.caddy.snippet (devops).
Refs #962.
