perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction (#187)
* perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise
Three operational improvements after ekb_full v3 completed at 99.1%:
Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope
Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation
Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)
40 tests passing.
* fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause)
Job 18 (manual_list recovery 11 quarters) crashed every quarter with:
psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023'
NSPD returns registration_date in mixed formats:
- DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ
- YYYY-MM-DD ISO for ЗУ / Здания
- YYYY-MM-DD HH:MM:SS with time
PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes.
Add _parse_nspd_date helper: handles both formats + None/empty.
Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel
for registration_date / build_record_registration_date / legal_act_date.
3 new unit tests for the helper (43 tests total).
* fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests
Bot review on PR #187 caught:
1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError
2. Missing tests for Fix H + Fix I
3. _parse_nspd_date didn't validate date ranges
Changes:
- Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column),
no schema migration needed
- _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45
- Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py
- Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py
- Add 1 test for parse_nspd_date range validation (was: only format)
49 tests passing.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
4d492ebc50