lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/data/sql
History
lekss361 31471bc1e7 feat(db): add nspd_quarter_dumps table for PKK harvest persistence (#94 pt.3/4) 
Sprint 1.1 item #2 из плана #94 part 2. Persistent storage для QuarterDump
snapshots (PR #109). Foundation для Celery harvest_quarter task (next PR)
и analyze_parcel integration (после).

Schema (data/sql/88_nspd_quarter_dumps.sql):
- Table `nspd_quarter_dumps`: 17 columns matching QuarterDump dataclass.
  - PK quarter_cad (natural key, one row per quarter)
  - quarter_geom (MultiPolygon, 4326) + bbox_3857 (Polygon, 3857) — dual-index
    стратегия: bbox для scheduler queries без transform, geom для joins с
    PZZ/district в 4326
  - 7 per-layer count columns + total_features (computed denorm для быстрых
    статистических queries без распаковки JSON)
  - features_json JSONB — array of ~117 NSPDFeature objects (~120KB per row,
    TOAST handles transparently). Geometry в raw EPSG:3857 — caller transforms.
  - layers_fetched TEXT[] mirrors QuarterDump.layers_fetched tuple
  - harvest_duration_ms (nullable — NULL = "not measured"), harvest_error TEXT
  - region_code SMALLINT NOT NULL (no DEFAULT — explicit per insert)
- 6 indexes: PK, GIST(bbox_3857), GIST(quarter_geom), B-tree(fetched_at_utc DESC),
  B-tree(region_code), partial B-tree(fetched_at_utc WHERE harvest_error IS NULL)
- View v_quarter_dumps_freshness — admin monitoring (AGE + is_failed flag)

Конвенции:
- BEGIN/COMMIT atomic
- IF NOT EXISTS на каждом CREATE — idempotent
- COMMENT ON TABLE + 7 COMMENT ON COLUMN + comments на каждом index
- psycopg v3 compatible (DDL only, no %s)

Применено к prod в рамках database-expert verification (DDL only, idempotent).

Vault entry: code/schemas/Schema_Nspd_Quarter_Dumps.md (upsert template +
EXPLAIN plans + cross-refs).

Code review (code-reviewer pre-push): APPROVE, 2 minor non-blocking:
- MINOR-1 quarter_geom MultiPolygon vs Polygon — resolved в Celery PR via
  ST_Multi(ST_Transform(...)) на ingest contract
- MINOR-2 layers_fetched default '{}' — documented в comment как
  partial-failure path для error rows

Next: Celery harvest_quarter task + beat schedule.

Part of #94 Sprint 1.1.
2026-05-12 18:19:30 +03:00
..
01_schema_rosreestr_deals.sql add data 2026-04-26 22:30:52 +03:00
02_load_all_quarters.sh fix 2026-04-30 22:16:44 +03:00
02b_load_via_tunnel.py fix 2026-04-30 22:16:44 +03:00
03_materialized_views.sql add data 2026-04-26 22:30:52 +03:00
10_schema_domrf.sql add data 2026-04-26 22:30:52 +03:00
11_load_domrf.py add data 2026-04-26 22:30:52 +03:00
20_schema_domrf_extras.sql add data 2026-04-26 22:30:52 +03:00
21_load_domrf_extras.py add data 2026-04-26 22:30:52 +03:00
30_scrape_domrf.py add anal 2 2026-04-27 13:05:36 +03:00
31_schema_domrf_full.sql add data 2026-04-26 22:30:52 +03:00
31_schema_domrf_raw.sql add data 2026-04-26 22:30:52 +03:00
32_load_domrf_full.py add data 2026-04-26 22:30:52 +03:00
32_load_domrf_raw.py add data 2026-04-26 22:30:52 +03:00
33_schema_domrf_lazy.sql add anal 2 2026-04-27 13:05:36 +03:00
34_load_domrf_lazy.py add anal 2 2026-04-27 13:05:36 +03:00
35_schema_domrf_stats.sql add anal 2 2026-04-27 13:05:36 +03:00
36_parse_xlsx_stats.py add anal 2 2026-04-27 13:05:36 +03:00
37_cleanup_redundant.sql add anal 2 2026-04-27 13:05:36 +03:00
38_relations_fix.sql add anal 2 2026-04-27 13:05:36 +03:00
39_territory_aliases.sql add anal 2 2026-04-27 13:05:36 +03:00
40_relations.sql add data 2026-04-26 22:30:52 +03:00
40_scrape_history.py add anal 2 2026-04-27 13:05:36 +03:00
41_load_history.py add anal 2 2026-04-27 13:05:36 +03:00
42_schema_breakdown_widen.sql add anal 2 2026-04-27 13:05:36 +03:00
43_anton_import.sql add anal 2 2026-04-27 13:05:36 +03:00
44_import_anton_db.py add anal 2 2026-04-27 13:05:36 +03:00
45_relations_v2.sql add anal 2 2026-04-27 13:05:36 +03:00
50_schema_kn_extensions.sql add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright 2026-04-27 17:58:40 +03:00
50_scrape_kn_flats.py add domrf extras parser (sale_graph + sales_agg + infra + photos) + 2026-04-27 18:35:46 +03:00
51_schema_kn_extras.sql add domrf extras parser (sale_graph + sales_agg + infra + photos) + 2026-04-27 18:35:46 +03:00
52_import_prinzip_crm.py ship #13-16: multi-target Dockerfile, per-object drill-in, CRM pipeline, 2026-04-27 19:32:37 +03:00
52_schema_prinzip_crm.sql ship #13-16: multi-target Dockerfile, per-object drill-in, CRM pipeline, 2026-04-27 19:32:37 +03:00
53_schema_kn_thumbs.sql fix ui lag 2026-04-27 20:26:55 +03:00
54_schema_scrape_log.sql add logs 2026-04-27 20:51:44 +03:00
55_schema_scrape_resume.sql add log scrapper 2026-04-27 21:50:44 +03:00
56_fetch_ekb_districts.py feat(geo): ЖК → district mapping через PostGIS-полигоны OSM (Limit_District_Geometry closed) 2026-04-29 08:03:38 +03:00
56_schema_ekb_districts_geom.sql feat(geo): ЖК → district mapping через PostGIS-полигоны OSM (Limit_District_Geometry closed) 2026-04-29 08:03:38 +03:00
57_kn_objects_district.sql feat(geo): ЖК → district mapping через PostGIS-полигоны OSM (Limit_District_Geometry closed) 2026-04-29 08:03:38 +03:00
58_fetch_cad_quarters_nspd.py sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
58_schema_cad_quarters.sql sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
59_backfill_obj_cad_quarter.sql sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
60_v_zk_rosreestr_velocity.sql sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
61_import_nspd_batch.py sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
62_scrape_nspd_full.py sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
63_schema_nspd_runs.sql fix 2026-04-30 22:16:44 +03:00
64_v_zk_rosreestr_velocity.sql fix 2026-04-30 22:16:44 +03:00
65_partitions_rosreestr_2024_h1.sql fix 2026-04-30 22:16:44 +03:00
66_indexes_recommend.sql audit-fix: 27mo окно, MODE bucket, индексы recommend_mix 2026-04-30 22:41:43 +03:00
67_refresh_ekb_districts_median.sql fix(scrape): SQLAlchemy text() парсил ':00:' regex как named-param 2026-04-30 23:16:11 +03:00
68_schema_objective.sql feat(objective): integrate api.objctv.ru — auth, schema, weekly sync 2026-05-07 21:04:00 +03:00
69_objective_smoke.py ignore 2026-05-07 21:04:38 +03:00
69b_objective_minimal.py ignore 2026-05-07 21:04:38 +03:00
70_parse_objective_raw.py feat(objective): integrate api.objctv.ru — auth, schema, weekly sync 2026-05-07 21:04:00 +03:00
71_etl_anton_sqlite_to_pg.py feat(objective): full sync pipeline + dynamic admin config 2026-05-10 19:54:15 +03:00
72_objective_sync_config.sql feat(objective): full sync pipeline + dynamic admin config 2026-05-10 19:54:15 +03:00
73_complexes_master.sql feat(db): canonical complexes + unified views + naming consistency 2026-05-10 21:11:20 +03:00
74_dedupe_unified_views.sql feat(db): canonical complexes + unified views + naming consistency 2026-05-10 21:11:20 +03:00
75_unification_naming.sql feat(db): canonical complexes + unified views + naming consistency 2026-05-10 21:11:20 +03:00
76_complex_connectivity.sql feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
77_nspd_geo_jobs.sql refactor(nspd-geo): unify on rosreestr2coord v5, drop legacy column 2026-05-11 14:13:33 +03:00
78_drop_use_rosreestr2coord.sql refactor(nspd-geo): unify on rosreestr2coord v5, drop legacy column 2026-05-11 14:13:33 +03:00
80_complexes_backfill_nulls.sql feat(db): backfill complexes nulls + add job_settings (migrations 80, 81) 2026-05-11 15:30:44 +03:00
81_job_settings.sql feat(db): backfill complexes nulls + add job_settings (migrations 80, 81) 2026-05-11 15:30:44 +03:00
82_osm_poi_ekb.sql feat(db): osm_poi_ekb table for site-finder POI analysis 2026-05-11 18:06:28 +03:00
83_cad_parcels_geom.sql feat(site-finder): cad_parcels_geom migration + POI markers on map + bulk-job UX 2026-05-11 19:54:56 +03:00
84_osm_noise_sources_ekb.sql feat(db): osm_noise_sources_ekb for site-finder noise scoring 2026-05-11 20:35:27 +03:00
85_pzz_zones_ekb.sql feat(site-finder): ПЗЗ territorial zones from Rosreestr PKK6 + zoning in analyze 2026-05-11 21:51:51 +03:00
86_v_bucket_success_score.sql feat(db): v_bucket_success_score view (success-driven mix #25) 2026-05-11 22:04:22 +03:00
87_on_demand_indexes.sql feat(site-finder): auto-fetch cadastre geometry on-demand (#93) (#95) 2026-05-12 09:02:17 +03:00
88_nspd_quarter_dumps.sql feat(db): add nspd_quarter_dumps table for PKK harvest persistence (#94 pt.3/4) 2026-05-12 18:19:30 +03:00
decode_to_file.py add data 2026-04-26 22:30:52 +03:00
upload_server.py add data 2026-04-26 22:30:52 +03:00