84 commits

Author	SHA1	Message	Date
Light1YT	c06dc0f5d6	feat(db): Dom.РФ housing/escrow → macro_indicator §7.10 (#946, partial) ... Migration 126: aggregate already-scraped domrf_kn_sale_graph + domrf_kn_objects into monthly macro_indicator series (source=domrf): housing_price_per_m2 (area-weighted, latest-snapshot deduped — naive SUM double-counts 2.7x), new_contracts_volume (Σ contracted ДДУ/mo), escrow_objects_count (escrow FLAG, not balances — caveat documented). Regions sverdl (region_cd=66) + rf (all scraped — currently = sverdl, domrf_kn covers region 66 only). Idempotent ON CONFLICT DO UPDATE; dry-run-verified vs prod (price ~156k ₽/m², dedup 45792→16853, escrow 460). Pure SQL, data already scraped. EMISS CPI/income (§7.11) DEFERRED: fedstat.ru WAF-blocks non-browser clients (JA3) — needs stealth.py browser-fetch generalization (separate effort, low priority while §9.6 forecast is advisory). Refs #946.	2026-06-03 17:05:33 +05:00
Light1YT	33ecbda689	feat(db): supply_layers store + latest view (#950 PR B) (#1003)	2026-06-02 20:21:04 +00:00
Light1YT	420cda2831	feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)	2026-06-02 20:06:29 +00:00
Light1YT	dbae4b0bda	feat(site-finder): macro_indicator table + backfill + reader (#945 PR A) (#963)	2026-06-02 18:41:28 +00:00
bot-backend	87a5de0cae	feat(db): quarter_price_index FDW foreign table + monthly refresh (Refs #762) (#797) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:16:28 +00:00
bot-backend	a4057a5b28	feat(db): mv_quarter_price_index — quarter price-index из ДКП (Refs #760) (#792) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:05:51 +00:00
Light1YT	e72cf87e35	feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563) (#630)	2026-05-28 15:31:24 +00:00
Light1YT	4e4762f234	fix(tradein): backfill kn obj_class (parser + 3-pass) (#572) (#628)	2026-05-28 15:04:32 +00:00
Light1YT	609ac8ea9d	fix(tradein): backfill kn price_per_m2 via sale_graph + peer median (#573) (#625)	2026-05-28 13:50:45 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	c61c8c86af	feat(sf-b1): GET /parcels/by-bbox — map entry endpoint + parcel_user_status table (#336)	2026-05-17 21:25:48 +00:00
lekss361	99e55c0bc2	feat(sf-b3): POST /pilot/request + pilot_requests table ... Добавляет endpoint для приёма заявок на пилот (lead-gen). INSERT в pilot_requests, response {id, created_at, status}. Telegram-уведомление — TODO (creds не настроены, #307 SF-B3). Migration: data/sql/118_pilot_requests.sql	2026-05-18 00:01:35 +03:00
lekss361	0c4d7f1ffb	feat(obj-2): mapping backfill via pre-materialized temp tables (migration 117) (#327)	2026-05-17 20:34:42 +00:00
lekss361	2bd80c5719	fix(obj-2): trim migration 116 to GIST index only — INSERT CTE timed out in deploy (#326)	2026-05-17 20:21:58 +00:00
lekss361	79c9a8a034	feat(obj-2): multi-feature mapping backfill (name+dev+district) — 142 → 200+ (#323)	2026-05-17 19:53:50 +00:00
lekss361	c80d79e2d4	fix(objective parser): use_float=True in ijson + default=str in json.dumps (#321)	2026-05-17 19:11:25 +00:00
lekss361	bb9a003080	fix(objective parser): wrap bytes-iter in file-like reader for ijson (#318)	2026-05-17 17:52:55 +00:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	b37e011aaa	feat(22c): domrf_kn_flat_plans new table + parser stub (#308)	2026-05-17 15:39:34 +00:00
lekss361	6612c1cd2b	feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)	2026-05-17 15:26:59 +00:00
lekss361	171a029997	feat(22i): domrf_kn_documents new table + parser (#305)	2026-05-17 15:21:31 +00:00
lekss361	3183abe41b	fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)	2026-05-17 15:15:03 +00:00
lekss361	16e3d5c86d	feat(22f): domrf_obj_checks table — 6 "Проверено на наш.дом.рф" checks (#303)	2026-05-17 15:07:24 +00:00
lekss361	0b12a55eb6	fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)	2026-05-17 14:59:02 +00:00
lekss361	bfe16ae71b	fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)	2026-05-17 14:32:23 +00:00
lekss361	e054395464	Merge pull request 'fix(sf-15): extract obj_class from aiDescription (DOM.РФ list API never returns objClass)' (#293) from fix/sf-15-obj-class-scraping into main	2026-05-17 14:18:49 +00:00
lekss361	33f44b59cb	feat(sf-18): is_sales_started flag для domrf_kn_objects (#292)	2026-05-17 14:09:21 +00:00
lekss361	440bba99cc	fix(sf-15): rename migration 105 → 106 (collision with PR #292 sf-18)	2026-05-17 17:03:16 +03:00
lekss361	3daf2a1fb1	fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field) ... Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт. _norm_object() uses it as fallback; objClass from API still takes priority if ever returned. Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads. Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).	2026-05-17 17:02:07 +03:00
lekss361	913cdcfcef	fix(sf-21): mv_layout_velocity weighted AVG — P0 data corruption ... AVG(deals_total_avg_area_m2) averaged pre-aggregated monthly values including months with 0 deals (area=0, price=0), pulling results down 2-14x. Replace with SUM(x * count) / NULLIF(SUM(count), 0). Verified on prod: obj 65307 room 4+ area 7.4->105.8 m2, price 7.9->112 k/m2. Migration 100_ includes REFRESH — no manual step needed post-deploy.	2026-05-17 16:53:36 +03:00
lekss361	989d4e5b7d	fix(sf-20): adaptive district threshold 30→15 with data_confidence flag ... - v_bucket_success_score HAVING >= 30 → >= 15 so sparse districts (Кировский etc.) are no longer silently excluded from the block - Backend: SUCCESS_REC_MIN_DEALS=15, SUCCESS_REC_STRONG_DEALS=30; data_confidence='weak' (15-29 deals) or 'strong' (≥30) in response - TS type ParcelSuccessRecommendation: add data_confidence literal - SuccessRecommendationBlock: amber badge for weak, all hex → CSS tokens, emoji star removed (ui-ux.md), empty-state text min 30→15	2026-05-17 16:34:52 +03:00
lekss361	da1c619a9c	fix(sf-04b): bulk seed mappings для всех ЖК с активностью в Objective ... Audit 2026-05-17 (extension PR #273): 13 high-confidence (sim > 0.85) auto-inserted, 25 medium-confidence (0.6-0.85) commented for manual review. 118 unmapped projects scanned via pg_trgm vs domrf_kn_objects EKB. Closes (epic part) #271 item 4 extension	2026-05-17 13:36:34 +03:00
lekss361	31b13c18b1	fix(sf-04): seed PRINZIP project mappings в objective_complex_mapping ... Audit 2026-05-17: objective_corpus_room_month содержал активные сделки для Малевич (98/12m) и Ньютон парк (30/12m), но записей в objective_complex_mapping не было → mv_layout_velocity пропускал оба проекта в velocity calculations. Добавлены маппинги: - Малевич → domrf_obj_id 64701 (Строящиеся, 244кв, сдача 2027-09) - Ньютон парк → domrf_obj_id 47390 (Сданные, 411кв, крупнейший corpus) ON CONFLICT DO NOTHING — идемпотентно. REFRESH mv_layout_velocity выполняется вручную post-deploy (runbooks/Refresh_MV_Layout_Velocity.md). Closes (epic part) #271 item 4	2026-05-17 11:57:06 +03:00
lekss361	8e377059a4	fix(#265 review): CRS 3857→4326 + stable zone_id hash + rename 102 + tx comment ... - #1 (BLOCK): ST_GeogFromGeoJSON → ST_Transform(ST_SetSRID(..., 3857), 4326)::geography so EPSG:3857 features from get_territorial_zones_in_bbox are correctly stored in GEOGRAPHY(POLYGON, 4326) column (matches cad_parcels/cad_buildings pattern) - #2: zone_id fallback changed from counter-based f"{quarter_cad}_{inserted}" to md5(json(props))[:12] — stable between runs, prevents wrong-row ON CONFLICT - #3: rename 101_cad_nspd_overlay_tables.sql → 102_* (101_user_custom_pois.sql exists) - #4: add comment clarifying SQLAlchemy autobegin guarantees outer tx before begin_nested() - Tests: updated test_feature_without_zone_id_uses_fallback to assert md5 format + idempotency	2026-05-17 10:45:52 +03:00
lekss361	f08db43145	feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ) ... Phase 1 (migration 101): создаёт 5 новых таблиц cad_territorial_zones, cad_red_lines, cad_engineering_structures, cad_risk_zones, cad_opportunity_parcels. Idempotent (IF NOT EXISTS), GIST-индексы. Phase 2 partial (ПЗЗ only): - NSPDBulkClient.get_features_in_bbox_grid — async grid-walk WMS GetFeatureInfo - NSPDBulkClient.get_territorial_zones_in_bbox — convenience wrapper layer 875838 - harvest_quarter Phase 2.5 — вызов get_territorial_zones_in_bbox после grid-walk - _save_territorial_zones — UPSERT loop с SAVEPOINT per-row, ON CONFLICT zone_id 9 unit tests (mock DB): insert/upsert/no-geom/no-id/error-isolation/savepoint pattern.	2026-05-17 10:28:22 +03:00
lekss361	485f489bd2	feat(#254): user_custom_pois — backend schema + CRUD + scoring integration ... - SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index - Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut - Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST) - API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth - Scoring in analyze_parcel: custom POI block after OSM POI loop, decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response - Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based) - Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section	2026-05-17 09:34:33 +03:00
lekss361	a6e4ff0407	feat(#114): seed 3 default weight presets + include_system API param ... Add SQL migration 100_user_weight_profiles_default_seed.sql with system presets Эконом/Комфорт/Бизнес (user_id='__system__'). Migration is idempotent via ON CONFLICT DO UPDATE. Backend: - weight_profiles.py: add SYSTEM_USER_ID constant + list_profiles_with_system() - admin_weight_profiles.py: add include_system query param to GET list endpoint Tests: 3 new tests covering include_system flag and service sentinel behaviour.	2026-05-16 22:49:56 +03:00
lekss361	f704785428	fix(sql): escape apostrophe in migration 99 COMMENT (deploy hotfix) ... PR #222 deploy failed на 99_nspd_entities_denorm.sql: ERROR: syntax error at or near "а" LINE 2: '3-сегментный квартал последнего harvest'а. Используется... Unescaped apostrophe в COMMENT string closes literal early. Fix: harvest'а → harvest''а (SQL escape). Других unescaped апострофов в SQL литералах нет (другие 3 — в SQL комментариях '--' , безопасно).	2026-05-16 21:02:00 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	fdb54834e5	fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220) ... - data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin - _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4 tolerance error and returns correct non-zero length for LINESTRING intersections - Rename intersection_area_sqm → intersection_length_m across schema, TS types, frontend component, and tests; add test_get_red_lines_db_exception_returns_empty Addresses review-bot feedback on PR #220.	2026-05-16 19:33:14 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	efa66a9f68	feat(analytics): mv_quarter_price_per_m2 MV + refresh helper (#33 D1 PR A) (#209)	2026-05-16 11:12:01 +00:00
lekss361	6aa22ef8ba	feat(analytics): mv_layout_velocity MV + refresh helper (#113 PR B) (#194)	2026-05-16 08:26:59 +00:00
lekss361	5f8df655cc	fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) ... NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4) which crashed upsert_parcel — schema was strict POLYGON. Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom), DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom. upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to MultiPolygon. NSPD returns both types (simple + multi-contour parcels). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:52:38 +03:00
lekss361	37e4f854c5	fix(db): retire ALL legacy cad_buildings index names before rename (#168 hotfix2) (#174) ... PR #173 fixed cad_buildings_pkey but missed 5 other indexes. Same root cause: PG does NOT auto-rename indexes on table rename → all 6 backing indexes remain on cad_buildings_old_apr26 with original names. Add 5 more `ALTER INDEX IF EXISTS ... RENAME TO ..._old_apr26_*` statements for: geom_gist, quarter_idx, objdoc_idx, complex_idx, purpose_idx. Migration 92 still NOT in _schema_migrations (4× rollback now). Next deploy will re-apply cleanly. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 14:09:40 +03:00
lekss361	7b2d1782e9	fix(db): retire legacy cad_buildings_pkey before rename — fix migration 92 (#168 hotfix) (#173) ... Migration 92_cad_bulk_layers.sql fails on deploy with: ERROR: relation "cad_buildings_pkey" already exists CONTEXT: ALTER INDEX cad_buildings_new_pkey RENAME TO cad_buildings_pkey Root cause: K.1 renamed cad_buildings → cad_buildings_old_apr26, but PG does NOT auto-rename the backing PK index on table rename. The legacy cad_buildings_pkey index is still attached to cad_buildings_old_apr26. K.3 then tries to rename cad_buildings_new_pkey to the same name → collision. Fix: add an explicit `ALTER INDEX IF EXISTS cad_buildings_pkey RENAME TO cad_buildings_old_apr26_pkey` BEFORE the new rename. Idempotent (IF EXISTS). Migration 92 was rolled back in last 3 deploy attempts (not in _schema_migrations), so this edit will re-apply cleanly on next deploy. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:59:53 +03:00