83 commits

Author	SHA1	Message	Date
Light1YT	33ecbda689	feat(db): supply_layers store + latest view (#950 PR B) (#1003)	2026-06-02 20:21:04 +00:00
Light1YT	420cda2831	feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)	2026-06-02 20:06:29 +00:00
Light1YT	dbae4b0bda	feat(site-finder): macro_indicator table + backfill + reader (#945 PR A) (#963)	2026-06-02 18:41:28 +00:00
bot-backend	87a5de0cae	feat(db): quarter_price_index FDW foreign table + monthly refresh (Refs #762) (#797) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:16:28 +00:00
bot-backend	a4057a5b28	feat(db): mv_quarter_price_index — quarter price-index из ДКП (Refs #760) (#792) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:05:51 +00:00
Light1YT	e72cf87e35	feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563) (#630)	2026-05-28 15:31:24 +00:00
Light1YT	4e4762f234	fix(tradein): backfill kn obj_class (parser + 3-pass) (#572) (#628)	2026-05-28 15:04:32 +00:00
Light1YT	609ac8ea9d	fix(tradein): backfill kn price_per_m2 via sale_graph + peer median (#573) (#625)	2026-05-28 13:50:45 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0567ad2130	feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)	2026-05-17 21:26:22 +00:00
lekss361	c61c8c86af	feat(sf-b1): GET /parcels/by-bbox — map entry endpoint + parcel_user_status table (#336)	2026-05-17 21:25:48 +00:00
lekss361	99e55c0bc2	feat(sf-b3): POST /pilot/request + pilot_requests table ... Добавляет endpoint для приёма заявок на пилот (lead-gen). INSERT в pilot_requests, response {id, created_at, status}. Telegram-уведомление — TODO (creds не настроены, #307 SF-B3). Migration: data/sql/118_pilot_requests.sql	2026-05-18 00:01:35 +03:00
lekss361	0c4d7f1ffb	feat(obj-2): mapping backfill via pre-materialized temp tables (migration 117) (#327)	2026-05-17 20:34:42 +00:00
lekss361	2bd80c5719	fix(obj-2): trim migration 116 to GIST index only — INSERT CTE timed out in deploy (#326)	2026-05-17 20:21:58 +00:00
lekss361	79c9a8a034	feat(obj-2): multi-feature mapping backfill (name+dev+district) — 142 → 200+ (#323)	2026-05-17 19:53:50 +00:00
lekss361	c80d79e2d4	fix(objective parser): use_float=True in ijson + default=str in json.dumps (#321)	2026-05-17 19:11:25 +00:00
lekss361	bb9a003080	fix(objective parser): wrap bytes-iter in file-like reader for ijson (#318)	2026-05-17 17:52:55 +00:00
lekss361	29dbbee2ac	feat(trade-in): TI-1 mock /trade-in/estimate endpoint + Pydantic + SQL migration (#316)	2026-05-17 16:45:06 +00:00
lekss361	389e141ec9	fix(scrape-objective): stream-parse lots_pf via ijson to avoid OOM (#315)	2026-05-17 16:37:30 +00:00
lekss361	b37e011aaa	feat(22c): domrf_kn_flat_plans new table + parser stub (#308)	2026-05-17 15:39:34 +00:00
lekss361	6612c1cd2b	feat(22begh): schema extension — flats+6 / objects+30 / developers+4 / metro_top3 (#306)	2026-05-17 15:26:59 +00:00
lekss361	171a029997	feat(22i): domrf_kn_documents new table + parser (#305)	2026-05-17 15:21:31 +00:00
lekss361	3183abe41b	fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) (#304)	2026-05-17 15:15:03 +00:00
lekss361	16e3d5c86d	feat(22f): domrf_obj_checks table — 6 "Проверено на наш.дом.рф" checks (#303)	2026-05-17 15:07:24 +00:00
lekss361	0b12a55eb6	fix(bug-22a): map missing kn-API fields (flat_number/living_area/wall_type/energy_eff) + backfill (#302)	2026-05-17 14:59:02 +00:00
lekss361	bfe16ae71b	fix(sf-16): derive price_per_m2 from price_rub in _norm_flat + backfill SQL (#298)	2026-05-17 14:32:23 +00:00
lekss361	e054395464	Merge pull request 'fix(sf-15): extract obj_class from aiDescription (DOM.РФ list API never returns objClass)' (#293) from fix/sf-15-obj-class-scraping into main	2026-05-17 14:18:49 +00:00
lekss361	33f44b59cb	feat(sf-18): is_sales_started flag для domrf_kn_objects (#292)	2026-05-17 14:09:21 +00:00
lekss361	440bba99cc	fix(sf-15): rename migration 105 → 106 (collision with PR #292 sf-18)	2026-05-17 17:03:16 +03:00
lekss361	3daf2a1fb1	fix(sf-15): extract obj_class from aiDescription (DOM.RF list API never returns objClass field) ... Add _extract_obj_class_from_ai() with regex patterns for Комфорт/Бизнес/Премиум/Элит/Стандарт. _norm_object() uses it as fallback; objClass from API still takes priority if ever returned. Add backfill SQL 105 to retroactively populate existing rows from raw JSON payloads. Coverage: ~1806/4548 rows will be filled; remainder NULL is expected (Нежилое/no AI desc).	2026-05-17 17:02:07 +03:00
lekss361	913cdcfcef	fix(sf-21): mv_layout_velocity weighted AVG — P0 data corruption ... AVG(deals_total_avg_area_m2) averaged pre-aggregated monthly values including months with 0 deals (area=0, price=0), pulling results down 2-14x. Replace with SUM(x * count) / NULLIF(SUM(count), 0). Verified on prod: obj 65307 room 4+ area 7.4->105.8 m2, price 7.9->112 k/m2. Migration 100_ includes REFRESH — no manual step needed post-deploy.	2026-05-17 16:53:36 +03:00
lekss361	989d4e5b7d	fix(sf-20): adaptive district threshold 30→15 with data_confidence flag ... - v_bucket_success_score HAVING >= 30 → >= 15 so sparse districts (Кировский etc.) are no longer silently excluded from the block - Backend: SUCCESS_REC_MIN_DEALS=15, SUCCESS_REC_STRONG_DEALS=30; data_confidence='weak' (15-29 deals) or 'strong' (≥30) in response - TS type ParcelSuccessRecommendation: add data_confidence literal - SuccessRecommendationBlock: amber badge for weak, all hex → CSS tokens, emoji star removed (ui-ux.md), empty-state text min 30→15	2026-05-17 16:34:52 +03:00
lekss361	da1c619a9c	fix(sf-04b): bulk seed mappings для всех ЖК с активностью в Objective ... Audit 2026-05-17 (extension PR #273): 13 high-confidence (sim > 0.85) auto-inserted, 25 medium-confidence (0.6-0.85) commented for manual review. 118 unmapped projects scanned via pg_trgm vs domrf_kn_objects EKB. Closes (epic part) #271 item 4 extension	2026-05-17 13:36:34 +03:00
lekss361	31b13c18b1	fix(sf-04): seed PRINZIP project mappings в objective_complex_mapping ... Audit 2026-05-17: objective_corpus_room_month содержал активные сделки для Малевич (98/12m) и Ньютон парк (30/12m), но записей в objective_complex_mapping не было → mv_layout_velocity пропускал оба проекта в velocity calculations. Добавлены маппинги: - Малевич → domrf_obj_id 64701 (Строящиеся, 244кв, сдача 2027-09) - Ньютон парк → domrf_obj_id 47390 (Сданные, 411кв, крупнейший corpus) ON CONFLICT DO NOTHING — идемпотентно. REFRESH mv_layout_velocity выполняется вручную post-deploy (runbooks/Refresh_MV_Layout_Velocity.md). Closes (epic part) #271 item 4	2026-05-17 11:57:06 +03:00
lekss361	8e377059a4	fix(#265 review): CRS 3857→4326 + stable zone_id hash + rename 102 + tx comment ... - #1 (BLOCK): ST_GeogFromGeoJSON → ST_Transform(ST_SetSRID(..., 3857), 4326)::geography so EPSG:3857 features from get_territorial_zones_in_bbox are correctly stored in GEOGRAPHY(POLYGON, 4326) column (matches cad_parcels/cad_buildings pattern) - #2: zone_id fallback changed from counter-based f"{quarter_cad}_{inserted}" to md5(json(props))[:12] — stable between runs, prevents wrong-row ON CONFLICT - #3: rename 101_cad_nspd_overlay_tables.sql → 102_* (101_user_custom_pois.sql exists) - #4: add comment clarifying SQLAlchemy autobegin guarantees outer tx before begin_nested() - Tests: updated test_feature_without_zone_id_uses_fallback to assert md5 format + idempotency	2026-05-17 10:45:52 +03:00
lekss361	f08db43145	feat(#263 sub-A): cad_* overlay schemas + cadastre territorial_zones (ПЗЗ) ... Phase 1 (migration 101): создаёт 5 новых таблиц cad_territorial_zones, cad_red_lines, cad_engineering_structures, cad_risk_zones, cad_opportunity_parcels. Idempotent (IF NOT EXISTS), GIST-индексы. Phase 2 partial (ПЗЗ only): - NSPDBulkClient.get_features_in_bbox_grid — async grid-walk WMS GetFeatureInfo - NSPDBulkClient.get_territorial_zones_in_bbox — convenience wrapper layer 875838 - harvest_quarter Phase 2.5 — вызов get_territorial_zones_in_bbox после grid-walk - _save_territorial_zones — UPSERT loop с SAVEPOINT per-row, ON CONFLICT zone_id 9 unit tests (mock DB): insert/upsert/no-geom/no-id/error-isolation/savepoint pattern.	2026-05-17 10:28:22 +03:00
lekss361	485f489bd2	feat(#254): user_custom_pois — backend schema + CRUD + scoring integration ... - SQL migration 101_user_custom_pois.sql: table with GEOGRAPHY(POINT) + GIST index - Pydantic schemas: CustomPoiCreate / CustomPoiUpdate / CustomPoiOut - Service custom_pois.py: CRUD + get_overlaps_for_scoring (1km radius, psycopg v3 CAST) - API /api/v1/custom-pois: POST(201)/GET/PATCH/DELETE, X-Session-Id header auth - Scoring in analyze_parcel: custom POI block after OSM POI loop, decay weight * max(0, 1 - dist/1000), custom_poi_score_items in response - Tests: CRUD + validation (weight/lon/lat 422) + scoring integration (mock-based) - Vault: Schema_User_Custom_Pois.md + Endpoint_Parcel_Analyze v3.9 section	2026-05-17 09:34:33 +03:00
lekss361	a6e4ff0407	feat(#114): seed 3 default weight presets + include_system API param ... Add SQL migration 100_user_weight_profiles_default_seed.sql with system presets Эконом/Комфорт/Бизнес (user_id='__system__'). Migration is idempotent via ON CONFLICT DO UPDATE. Backend: - weight_profiles.py: add SYSTEM_USER_ID constant + list_profiles_with_system() - admin_weight_profiles.py: add include_system query param to GET list endpoint Tests: 3 new tests covering include_system flag and service sentinel behaviour.	2026-05-16 22:49:56 +03:00
lekss361	f704785428	fix(sql): escape apostrophe in migration 99 COMMENT (deploy hotfix) ... PR #222 deploy failed на 99_nspd_entities_denorm.sql: ERROR: syntax error at or near "а" LINE 2: '3-сегментный квартал последнего harvest'а. Используется... Unescaped apostrophe в COMMENT string closes literal early. Fix: harvest'а → harvest''а (SQL escape). Других unescaped апострофов в SQL литералах нет (другие 3 — в SQL комментариях '--' , безопасно).	2026-05-16 21:02:00 +03:00
lekss361	db1ff5b1fc	feat(nspd): denorm nspd_parcels + nspd_buildings tables (#94 PR3 of 4) ... - data/sql/99_nspd_entities_denorm.sql: 2 tables (cad_num PK) + 6 indexes (quarter_cad, GIST geom, partial WHERE purpose ILIKE '%многокв%') - nspd_denorm.py: denorm_parcel_feature / denorm_building_feature / denorm_dump - SAVEPOINT per row via with db.begin_nested() - ST_Transform(ST_SetSRID(ST_GeomFromGeoJSON(:geom), 3857), 4326) - json.dumps(geom, ensure_ascii=False) for psycopg bind - All CAST(:x AS type) — no ❌:type - nspd_sync.py: inline denorm after _upsert_dump in harvest_quarter (non-fatal) - nspd_denorm_backfill.py: Celery one-shot task (per-quarter commit) - admin_etl.py: POST /api/v1/admin/etl/nspd-denorm-backfill (AdminTokenAuth) - 19 tests: coerce helpers, denorm parcel/building, aggregate counts Foundation for downstream features (МКД neighbors lookup, parcel attrs). Part of #94	2026-05-16 20:20:03 +03:00
lekss361	fdb54834e5	fix(nspd): rename migration 89→98, fix red lines ST_Area→ST_Length (#220) ... - data/sql/98_*: rename from 89 to avoid collision with existing 89_drop_dead_brin - _get_red_lines: ST_Length(ST_Intersection(planar)::geography) instead of ST_Area(ST_Intersection(::geography, ::geography)) — fixes PostGIS 3.4 tolerance error and returns correct non-zero length for LINESTRING intersections - Rename intersection_area_sqm → intersection_length_m across schema, TS types, frontend component, and tests; add test_get_red_lines_db_exception_returns_empty Addresses review-bot feedback on PR #220.	2026-05-16 19:33:14 +03:00
lekss361	ed3c128528	feat(nspd): TIER 4 opportunity layers + red lines (#94 PR2 of 4) ... - NSPDClient: QUARTER_OPPORTUNITY_LAYERS (auction/scheme/free/future/oopt) + QuarterDump.opportunity field - nspd_sync: harvest_quarter accepts include_opportunity, denorm cols has_auction_parcels + opportunity_count in UPSERT - quarter_dump_lookup: - _get_opportunity_parcels (sort by distance, early-exit on count=0) - _get_red_lines (query existing dump.red_lines core path, layer='red_lines') - SQL 89_*: has_auction_parcels + opportunity_count + partial index - Pydantic: OpportunityParcel + RedLine schemas - /analyze: nspd_opportunity_parcels + nspd_red_lines fields - Frontend: NspdOpportunityBlock + NspdRedLinesBlock + LandTab integration - Tests: 28 total (11 PR1 + 17 PR2), all pass Red lines uses existing core harvest path (layer 879243 already in dump.red_lines from PR1+core) — no duplicate harvest, no red_lines_count_v2 column needed. Part of #94	2026-05-16 19:06:22 +03:00
lekss361	dbf31cc911	feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)	2026-05-16 12:47:09 +00:00
lekss361	adc45f5724	feat(permits): ЕКБ РНС/РВЭ xlsx parser + schema + ETL (#105 PR A) (#211)	2026-05-16 11:41:21 +00:00
lekss361	efa66a9f68	feat(analytics): mv_quarter_price_per_m2 MV + refresh helper (#33 D1 PR A) (#209)	2026-05-16 11:12:01 +00:00
lekss361	6aa22ef8ba	feat(analytics): mv_layout_velocity MV + refresh helper (#113 PR B) (#194)	2026-05-16 08:26:59 +00:00
lekss361	5f8df655cc	fix(cadastre): cad_parcels.geom Polygon -> MultiPolygon (migration 93) (#185) ... NSPD returns MultiPolygon for Многоконтурный участок (e.g. 66:41:0105017:4) which crashed upsert_parcel — schema was strict POLYGON. Migration 93: ALTER COLUMN geom Polygon -> MultiPolygon USING ST_Multi(geom), DROP+RECREATE GIST index + back-compat VIEW cad_parcels_geom. upsert_parcel SQL wraps ST_Transform in ST_Multi() — coerces Polygon to MultiPolygon. NSPD returns both types (simple + multi-contour parcels). Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 18:52:38 +03:00
lekss361	37e4f854c5	fix(db): retire ALL legacy cad_buildings index names before rename (#168 hotfix2) (#174) ... PR #173 fixed cad_buildings_pkey but missed 5 other indexes. Same root cause: PG does NOT auto-rename indexes on table rename → all 6 backing indexes remain on cad_buildings_old_apr26 with original names. Add 5 more `ALTER INDEX IF EXISTS ... RENAME TO ..._old_apr26_*` statements for: geom_gist, quarter_idx, objdoc_idx, complex_idx, purpose_idx. Migration 92 still NOT in _schema_migrations (4× rollback now). Next deploy will re-apply cleanly. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 14:09:40 +03:00
lekss361	7b2d1782e9	fix(db): retire legacy cad_buildings_pkey before rename — fix migration 92 (#168 hotfix) (#173) ... Migration 92_cad_bulk_layers.sql fails on deploy with: ERROR: relation "cad_buildings_pkey" already exists CONTEXT: ALTER INDEX cad_buildings_new_pkey RENAME TO cad_buildings_pkey Root cause: K.1 renamed cad_buildings → cad_buildings_old_apr26, but PG does NOT auto-rename the backing PK index on table rename. The legacy cad_buildings_pkey index is still attached to cad_buildings_old_apr26. K.3 then tries to rename cad_buildings_new_pkey to the same name → collision. Fix: add an explicit `ALTER INDEX IF EXISTS cad_buildings_pkey RENAME TO cad_buildings_old_apr26_pkey` BEFORE the new rename. Idempotent (IF EXISTS). Migration 92 was rolled back in last 3 deploy attempts (not in _schema_migrations), so this edit will re-apply cleanly on next deploy. Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:59:53 +03:00
lekss361	59f222eaf3	feat(db): bulk cadastre schema — 7 tables + cadastre_jobs (#168 PR1/5) (#169) ... * feat(db): schema migration for bulk cadastre ingest layers (#168) Add 7 cadastre layer tables + cadastre_jobs saga-state table. Tables: - cad_parcels (36368 ЗУ ЕГРН) replaces cad_parcels_geom - cad_buildings v2 wide schema (replaces existing, backed up as cad_buildings_old_apr26) - cad_constructions (36383 Сооружения + ЕГРН, flag is_egrn) - cad_oncs (36384), cad_enk (39663), cad_zouit (4 categoryIds), cad_quarter_stats - cadastre_jobs (resumable saga state, mirrors geo_jobs pattern) GIST + BTREE indexes. Migration of cad_parcels_geom data via INSERT...SELECT. Restore v_complex_full + v_complex_buildings views after table rename. * fixup(db): address PR #169 bot review — 2 blockers + 3 lessers Blocker #1: cad_parcels_geom DROP → back-compat VIEW Old DROP TABLE would break site-finder analyze fallback (per 83_*.sql COMMENT). Now: drop old BASE TABLE (guarded by table_type), create VIEW over cad_parcels with identical 4-column signature. Callers continue to work; PR2/5 will switch. Blocker #2: K.3 redundant RENAME CONSTRAINT removed ALTER INDEX RENAME on PK index already renames the backing constraint in PG16. Second ALTER TABLE RENAME CONSTRAINT would throw 'does not exist' and abort tx. Lesser #1: cadastre_jobs.status CHECK constraint Inline CHECK (status IN ('queued','running','done','failed','zombie','cancelled','paused')) per partial-index intent and admin UI plan. Lesser #2: COMMENT ON TABLE for all 8 tables + view (keeps convention from 63, 83). Lesser #3: set_updated_at() triggers on 7 tables with updated_at column. Reuses existing function (verified present). DO $$ guards on pg_trigger for idempotent re-apply. cadastre_jobs excluded (uses heartbeat_at instead). --------- Co-authored-by: lekss361 <claudestars@proton.me>	2026-05-15 13:05:25 +03:00