Commit graph

7 commits

Author SHA1 Message Date
c94810f3a7 feat(forecast): forecast_site_finder_report Celery task + schema-filtered run read (3b-i)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m21s
Deploy / build-worker (push) Successful in 2m39s
Deploy / deploy (push) Successful in 1m11s
build_site_finder_report (§22) takes ~30-180s → runs in a background Celery task,
not inline on the sync /analyze endpoint.

- repository: latest_run_for gains keyword-only schema_version (default None keeps
  v_analysis_runs_latest behavior, backward-compat); when given, reads base
  analysis_runs filtered by schema_version ORDER BY created_at DESC LIMIT 1 — fetches
  the latest analyze-1.0 site-analysis run even when newer 1.0 (§22) rows exist on top
  (index-served via 127's (cad_num, created_at DESC)).
- new workers/tasks/forecast.py::forecast_site_finder_report: reads latest analyze-1.0,
  runs the §22 orchestrator, persists SiteFinderReport.as_dict() as a 1.0 run via
  persist_analysis_run. Graceful: no base run / compute error → logger + return None
  (worker not crashed). time_limit=900/soft=840 (no global limit). Registered in include.

Prod-confirmed: analyze-1.0 result carries the full analyze dict (competitors+district)
→ orchestrator input valid. Endpoint trigger (3b-ii) + §9.x untouched. 943 tests pass;
code-review APPROVE (contracts verified vs real as_dict(); status done→complete normalized,
no IntegrityError). Refs #994 #961.
2026-06-05 09:02:15 +05:00
5f60668642 fix(nspd-geo): coerce geom to MultiPolygon + wide cad_buildings schema in on-demand ingest
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 1m8s
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m35s
Deploy / build-worker (push) Successful in 2m19s
REAL code-level cause (prod-confirmed, 2 parcels: "Geometry type (Polygon) does
not match column type (MultiPolygon)") of single-contour parcels stuck "fetching"
— complements migrations 129/130 (which fixed the cad_parcels NOT NULL chain).

F1: _save_parcel/_save_quarter now ST_Multi() the geom so single-contour Polygons
fit the MultiPolygon columns (migr 93/58), mirroring the bulk_harvest paths.
F2: _save_building rewritten to the wide cad_buildings schema (migr 92):
quarter_cad_number (was quarter_cad_num → UndefinedColumn), _safe_int(floors)
(range strings → NULL), source='nspd' NOT NULL.
F3: upsert_features wraps each feature in begin_nested() SAVEPOINT so one bad
feature no longer aborts the whole quarter snapshot (backend.md rule; mirrors
grid-walk). New tests/workers/test_nspd_geo.py + savepoint tests. 89 passed.
2026-06-04 13:11:47 +05:00
ed3b302d57 fix(supply-layers): thread dev_group_name into L3 upsert key (#970 CRITICAL)
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / build-worker (push) Successful in 2m32s
Deploy / deploy (push) Successful in 1m4s
REOPENED. L3 future-supply rows are computed per (district_name, dev_group_name)
but dev_group_name was never a key column — only embedded in method text. With
complex_id/obj_class NULL for L3, every dev_group of a district collapsed to one
upsert key → ~95.6% loss. Ground-truth (Академический, prod): should be 13,808
units / 15 dev_groups / 54 objects; only 1 row / 607 units survived.

Migration 128: ADD COLUMN supply_layers.dev_group_name TEXT + rebuild
uq_supply_layers_logical to (layer, district_name, complex_id, obj_class,
dev_group_name, source, snapshot_date) NULLS NOT DISTINCT (L1/L2 dev_group_name
NULL stays transparent → their dedup unchanged; L3 distinct groups no longer
collapse). Dry-run-verified vs prod catalog (applies clean, ROLLBACK clean).

Worker: SupplyLayerRow gains dev_group_name (L1/L2=None, L3=group); _UPSERT_SQL
adds it to INSERT/VALUES (CAST(:dev_group_name AS text)) + ON CONFLICT (key col,
not in DO UPDATE SET). Service+worker regression tests assert same-district/
different-dev_group → distinct keys (no collapse). 234 supply tests pass.

Deploy applies migration before container restart; collapsed data self-heals on
next supply_layers_refresh. Verification = prod re-measure post-deploy.

Refs #970
2026-06-04 10:06:30 +05:00
ae85f19993 feat(workers): supply-layers refresh task + weekly beat (#950 Step 5) (#1005)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / deploy (push) Successful in 1m9s
Deploy / changes (push) Successful in 6s
Deploy / build-worker (push) Successful in 2m22s
2026-06-02 20:47:05 +00:00
c78c6ec2d0 feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / deploy (push) Successful in 1m10s
Deploy / changes (push) Successful in 6s
Deploy / build-backend (push) Successful in 1m24s
Deploy / build-worker (push) Successful in 2m53s
2026-06-02 19:05:57 +00:00
f9adb5cc2f feat(22d): catalog scraper — skip-today filter + force flag (#348)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m19s
Deploy / build-worker (push) Successful in 2m26s
Deploy / deploy (push) Successful in 49s
2026-05-17 23:15:22 +00:00
lekss361
4d492ebc50
perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187)
* perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise

Three operational improvements after ekb_full v3 completed at 99.1%:

Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope

Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation

Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)

40 tests passing.

* fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause)

Job 18 (manual_list recovery 11 quarters) crashed every quarter with:
  psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023'

NSPD returns registration_date in mixed formats:
- DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ
- YYYY-MM-DD ISO for ЗУ / Здания
- YYYY-MM-DD HH:MM:SS with time

PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes.

Add _parse_nspd_date helper: handles both formats + None/empty.
Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel
for registration_date / build_record_registration_date / legal_act_date.

3 new unit tests for the helper (43 tests total).

* fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests

Bot review on PR #187 caught:
1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError
2. Missing tests for Fix H + Fix I
3. _parse_nspd_date didn't validate date ranges

Changes:
- Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column),
  no schema migration needed
- _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45
- Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py
- Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py
- Add 1 test for parse_nspd_date range validation (was: only format)

49 tests passing.

---------

Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 23:02:37 +03:00