Adds period_type TEXT NOT NULL DEFAULT 'unknown' to macro_indicator and
widens the PRIMARY KEY to (indicator_type, region, obs_date, period_type).
Before: yearly aggregate ('год'→obs_date YYYY-01-01) and Q1 ('I квартал'→
same date) shared the same PK slot → ON CONFLICT DO UPDATE made them
overwrite each other despite the in-memory dedup fix in #1687.
After: each EmissRow carries period_type from _emiss_period_granularity
('year'/'quarter'/'month'); DB ON CONFLICT targets include it, so
yearly+Q1 rows genuinely coexist.
Non-EMISS sources (CBR, rosstat open-data, domrf) use period_type='unknown'
(literal in SQL) — they are disambiguated by obs_date already, so the
wider PK is backward-compatible. All ON CONFLICT clauses in
cbr_macro_sync.py and rosstat_macro_sync.py updated accordingly.
Migration: 163_emiss_pk_period_type.sql (idempotent BEGIN/COMMIT).
Tests: 49 passed (emiss + rosstat suite), ruff clean, py_compile OK.
sat_factor = 1 + ((sold_pct-50)/100)*0.30 was computed in 09_macro_and_trend.py,
written to district_economics.sat_factor, and fetched in server.py and 10_score_v2.py
— but never multiplied into any score. The live market sub-score uses a separate
sat_score = min(100, sold_pct*100/70) directly, so sat_factor was dead code that
would double-count absorption if ever wired in.
- 09_macro_and_trend.py: remove sat_factor computation, ALTER TABLE column, UPDATE
binding, and debug print column
- 10_score_v2.py: remove sat_factor from SELECT and unpacking
- server.py: remove sat_factor variable assignment and from macro_factors response
- static/index.html: remove sat_factor documentation row
- data/sql/162_drop_district_economics_sat_factor.sql: DROP COLUMN IF EXISTS
Replace ST_Intersects(district.geom, quarter.geom) with ST_Within(ST_Centroid(quarter.geom), district.geom)
so each cadastral quarter is assigned to exactly one district. Eliminates 85 double-counted quarter-district
pairs and 307 spurious deal-district rows that were skewing PERCENTILE_CONT medians. Function signature
unchanged; no backend code changes required.
Правки уже-применённых миграций (85/062/080) не долетают до прода через
_schema_migrations. Дописаны отдельные миграции на живых таблицах:
- 158: pzz_zones_ekb UNIQUE NULLS NOT DISTINCT (rosreestr_id) (#1361)
- tradein 108: повторный backfill Avito-адресов улучшенным regexp (#1419)
- tradein 109: пересчёт asking_to_sold_ratios secondary-only (#1364/#1186)
Все три dry-run (BEGIN..ROLLBACK) против прод-схемы: PSQL_RC=0.
P0 goldmine: objective_lots (1.12M, 81% цен) связывается с domrf для velocity/
layout-аналитики. Mapping было 148 → +115 (107 high + 8 mid) = 263.
Fuzzy-match objective(project_name) ↔ domrf_kn_objects(comm_name) LATEST snapshot
region_cd=66 через pg_trgm на НОРМАЛИЗОВАННЫХ именах (strip ЖК-boilerplate с
\m\M word-boundary). Правило приёмки dev-corroborated: nsim≥0.62 AND dsim≥0.5 —
precision over recall для P0-цен (generic-имена с разным застройщиком, напр.
Солнечный/Прекрасный/Стрелки dsim=0.0, отклонены). match_method='auto_fuzzy_v2',
is_reviewed=false (для human-review топ-кандидатов), confidence-tier в note.
Idempotent ON CONFLICT (objective_complex_name, objective_group). REFRESH
CONCURRENTLY mv_layout_velocity после COMMIT.
Refs #307
UNIQUE (obj_id, doc_type, doc_num, file_url) в м.112 без NULLS NOT
DISTINCT, doc_num nullable. В PG NULL != NULL в уникальном индексе →
для документов с docNum=null (документированный частый вход:
extract_documents:125-134 → doc_num=None) ON CONFLICT-арбитр в
scrapers/documents.py:59 НИКОГДА не матчился → каждый ежедневный sweep
domrf_kn.py:1903 молча вставлял новую строку-дубль вместо UPDATE.
Таблица росла ×N прогонов; дубли уходили в analytics_queries.py:1068.
Миграция 140:
1. Дедуп существующих NULL-дублей: ROW_NUMBER() PARTITION BY все 4 ключа
ORDER BY scraped_at DESC, id DESC → оставляем самую свежую, удаляем
остальные WHERE rn>1 AND doc_num IS NULL.
2. DROP старого autonamed constraint'а IF EXISTS.
3. ADD uq_domrf_kn_documents_logical UNIQUE NULLS NOT DISTINCT (...) с
именованным guard по pg_constraint.conname (паттерн м.125 → idempotent).
Caller scrapers/documents.py НЕ меняется — после миграции ON CONFLICT-
арбитр начнёт матчить NULL-doc_num строки автоматически. Прецедент того
же класса: м.110 (uq_infra_dedupe), м.125 (uq_supply_layers_logical).
Closes#1206
Foundation PR: unified "our projects" source the §25.3 overlap engine (PR2)
will consume. Two origins normalized to OwnProject (class/timing/price/unit-mix):
- current <- domrf_kn_objects filtered by settings.own_developer_ids (numeric
prefix of composite dev_id; empty -> [] graceful, no DB hit, no hardcoded id)
- future <- new manual-entry own_planned_project entity (migration 148)
Adds OWN_DEVELOPER_IDS config (comma-sep -> list[int], default []),
own_planned_project table (range/unit_mix CHECKs via IMMUTABLE helper, generated
geom), /api/v1/own-projects CRUD (created_by from X-Authenticated-User), and
get_own_portfolio(db). Per-source graceful degradation; psycopg-v3 CAST clean.
Does not touch special_indices.py or parcels.py (out of scope).
Refs #1169
The demand_index DB column comment still said "насыщающее преобразование"
(saturating) after #1167 switched it to city-relative normalization —
exactly the misleading-comment class that hid the original bug. Migration
147 updates COMMENT ON COLUMN (idempotent, comment-only, no data DDL).
Refs #948
Promote district to a first-class `location` entity (ТЗ §8.2), ADDITIVE — no
district->FK refactor. New `location` table keyed by district_name (joinable by
string), carrying 4 normalized [0,1] indices (NULL when no data, never 0):
- infra: _district_poi_score / _city_avg_poi_score (per-district POI aggregate)
- competition: market_metrics.overstock_index (available/stuck competing supply
— orthogonal to demand; NOT sell-through, which is market-heat correlated w/ demand)
- demand: market_metrics.unit_velocity (saturating /50)
- future_supply: future_supply_pressure.index (passthrough, already 0..1)
- data/sql/146_location.sql: idempotent table + UNIQUE(district_name) + range
CHECK + centroid GIST
- services/site_finder/locations.py: compute_location_indices (reuses forecast
per-district fns) + refresh_locations (SAVEPOINT per-row, CAST, ON CONFLICT)
- workers/tasks/location_refresh.py + beat (Mon 07:00 MSK, after supply-layers)
- api/v1/locations.py: read-only GET list + GET by name (analyst+admin via rbac,
frontend pilot-gated)
- tests: 34 (normalization 0..1/null, competition⊥demand orthogonality, idempotent
upsert, read API list/by-name/404)
Part of #948 (Part A insight shipped #1164).
Insight = аналитик пишет free-form intel про локацию/участок с пометкой
«непублично» (is_confidential, §7.13/§8.9). CRUD под /api/v1/insights;
created_by из X-Authenticated-User (не из тела — спуфинг автора невозможен).
Зеркало custom_pois: raw-SQL service + Pydantic v2 schemas, sync def handlers
(threadpool, off event loop).
- data/sql/145_insight.sql: table insight (idempotent, geom GENERATED из lon/lat,
индексы district/cad_num/is_confidential/created_by/GIST); аддитивно расширяет
CHECK audit_log.action += insight_write (тот же constraint ck_audit_log_action,
to_regclass-guarded, superset — не ломает #962-аудит; НЕ правит applied 144).
- audit_middleware.classify_path: method-aware insight_write для POST/PUT/DELETE
/api/v1/insights (GET-чтения не аудируются); обратно совместимо с parcels.
- ACL: backend rbac_guard hard-блокирует только /api/v1/admin/*; pilot отсекается
frontend RouteGuard + Caddy (как parcels/custom_pois). is_confidential — стораемая
пометка без per-record backend-гейта (analyst видит, per #962-политике).
- tests: 28 insight (CRUD+filters+required+#261 commit) + 5 audit. 86 зелёных, ruff.
Part B (Location first-class, §8.2) — отдельный follow-up.
Refs #948.
EPIC18/§19. analyst sees everything (deals, insights, exports, site-finder,
analytics, concept) EXCEPT admin/data-management. Enforcement is backend-hard
(the existing rbac_guard already 403s any non-admin role on /api/v1/admin/*, so
adding the role auto-blocks it) + frontend (deny_paths via /me) + audit.
§19 audit: new best-effort HTTP middleware logs the sensitive actions
(analyze / forecast / forecast-export) to a new audit_log table after the
response. Audit failures never break or delay-fail the request (2-layer
try/except + finally close). Registered INNER to rbac_guard so only authorized
requests are audited (a 403 short-circuits before audit). classify_path matches
export before forecast (anchored).
- auth/roles.yaml: analyst role (paths /**, deny admin-mgmt) + analysttest QA user
- core/auth.py (+ tradein mirror): Role Literal += analyst
- core/audit_middleware.py (new) + main.py registration
- data/sql/144_audit_log.sql (idempotent; auto-applies)
- tests: analyst rbac (403 admin / 200 parcels) + 11 audit cases
No data-level ACL (analyst sees data per policy) -> no #948 dependency. No new
deps. parcels.py untouched. Real analyst logins still need adding to
caddy/users.caddy.snippet (devops).
Refs #962.
§9.x forecast metrics filter objective_lots.district (informal micro-districts)
but /analyze resolves the official admin district from ekb_districts → admin
name matched 0 rows → silent empty forecast (market_metrics('Кировский')→0).
Add a curated micro→admin alias table seeding all 35 micro values to their
primary admin (8 EKB admins). Curated not spatial: complex_id join hits only
~28% of rows and complexes is multi-city, so ST_Contains scatters micros
uniformly. 4 cross-city values (Н.Тагил/Каменск) contained in one noted bucket.
Coverage: 8/8 admins resolve, 35/35 mapped, 0 unmapped. Idempotent + additive
(CREATE TABLE IF NOT EXISTS + INSERT ON CONFLICT DO NOTHING); applied to prod
+ registered in _schema_migrations. Resolver/fallback documented in header;
market_metrics wiring is Step 2 (separate PR). Refs #969#949.
After #129 fixed quarter_cad_number, the on-demand НСПД ingest of a real
not-pre-loaded parcel hit the NEXT cad_parcels NOT NULL column: source
(NotNullViolation) → geometry transaction rolled back → POST /analyze still
stuck. Repro 66:41:0610029:83 (job 57: quarter derived by #129, source null).
Migration 130: extend the #129 BEFORE INSERT/UPDATE trigger function to also
default source→'nspd' when NULL (column DEFAULT can't help — INSERT binds
explicit NULL; trigger overrides it). Explicit sources ("search"/"wms_grid_walk")
respected — only NULL is filled. cad_parcels NOT NULL surface now fully covered
(cad_num/quarter[#129]/category_id[literal]/source[#130]/fetched_at+updated_at[NOW]).
Dry-run-verified on prod: null quarter+source → 66:41:0610029 + nspd, INSERT ok.
PROD bug: on-demand НСПД ingest (bulk_harvest.upsert_parcel, for parcels NOT
pre-loaded) intermittently inserts quarter_cad_number=NULL at runtime →
NotNullViolation → geometry never persists → POST /analyze for a real new parcel
stuck "Геометрия загружается из НСПД…" forever (re-enqueues each call). Repro:
66:41:0610029:83 (real НСПД parcel; failing row had valid geom + props, only
quarter_cad_number null). Code derives quarter_cad but NULL still reaches the
INSERT via some runtime path (grid-walk).
Migration 129: BEFORE INSERT OR UPDATE trigger on cad_parcels — when
quarter_cad_number IS NULL and cad_num has >=3 colon-segments, derive it
(first 3 segments: 66:41:0610029:83 → 66:41:0610029). Fixes ALL write paths at
the DB level (quarter_cad_number is a pure derivation of cad_num). <3-segment
cad_num intentionally left NULL → NOT NULL still rejects genuine garbage (no
masking). Coexists with m.92 set_updated_at trigger. Idempotent.
Dry-run-verified on prod catalog: null-quarter insert → derived 66:41:0610029;
garbage <3seg still rejected. Verification = re-fetch 0610029:83 post-deploy.
REOPENED. L3 future-supply rows are computed per (district_name, dev_group_name)
but dev_group_name was never a key column — only embedded in method text. With
complex_id/obj_class NULL for L3, every dev_group of a district collapsed to one
upsert key → ~95.6% loss. Ground-truth (Академический, prod): should be 13,808
units / 15 dev_groups / 54 objects; only 1 row / 607 units survived.
Migration 128: ADD COLUMN supply_layers.dev_group_name TEXT + rebuild
uq_supply_layers_logical to (layer, district_name, complex_id, obj_class,
dev_group_name, source, snapshot_date) NULLS NOT DISTINCT (L1/L2 dev_group_name
NULL stays transparent → their dedup unchanged; L3 distinct groups no longer
collapse). Dry-run-verified vs prod catalog (applies clean, ROLLBACK clean).
Worker: SupplyLayerRow gains dev_group_name (L1/L2=None, L3=group); _UPSERT_SQL
adds it to INSERT/VALUES (CAST(:dev_group_name AS text)) + ON CONFLICT (key col,
not in DO UPDATE SET). Service+worker regression tests assert same-district/
different-dev_group → distinct keys (no collapse). 234 supply tests pass.
Deploy applies migration before container restart; collapsed data self-heals on
next supply_layers_refresh. Verification = prod re-measure post-deploy.
Refs #970
Migration 127: durable, versioned, re-openable persistence for Site Finder
analysis runs (the §22 API result + forecast/recommendation/report snapshot).
One row = one completed analysis of a parcel (frozen result jsonb =
SiteFinderReport.as_dict() #987, schema_version for versioned re-open).
Append-mostly (history/versioning — never overwrites prior runs → TABLE not MV).
Replaces the last_analysis_date=None placeholder (parcels.py:1026,1092):
"date of last analysis" + "current analysis of parcel" now come from
v_analysis_runs_latest (DISTINCT ON cad_num, max created_at, deterministic
id-DESC tiebreak — mirrors v_supply_layers_latest m.125).
created_by = X-Authenticated-User identity, nullable TEXT, NO FK (no users
table; precedent m.90/m.119; background/Celery runs have no author). result/
params/segment as JSONB (report shape evolves); district/confidence/status
denormalized out for filter/sort. status + confidence CHECK-guarded.
Idempotent (CREATE TABLE/INDEX IF NOT EXISTS + CREATE OR REPLACE VIEW).
Dry-run-verified vs prod catalog (DDL applies clean in rolled-back txn;
DISTINCT ON latest-wins confirmed). Foundation for #994 persist + #992
contract. Closes#993. Refs #961.
