ea42d29a4a
feat(site-finder): live CBR key-rate scraper → macro_indicator ( #945 PR B)
...
Site Finder v2 / GG-форсайт. Deterministic, no LLM. Fills the key_rate series
(empty after PR A) into macro_indicator from the live CBR source.
- services/scrapers/cbr_macro.py: fetch_key_rate via CBR SOAP DailyInfoWebServ
KeyRateXML(fromDate, ToDate) (POST soap+xml; ToDate capital-T + dateTime
suffix required — flat GET returns empty). Pure parse_key_rate_xml (TZ-safe
date, empty/malformed → []). Live-verified: 85 rows, current rate 14.5%.
- workers/tasks/cbr_macro_sync.py: upsert (key_rate, rf, daily, %) into
macro_indicator ON CONFLICT DO UPDATE; SAVEPOINT per row; fetch-fail → raise.
- beat: cbr-macro-sync-weekly (Mon 05:30 МСК, Europe/Moscow tz).
- 15 tests (pure parse + mocked upsert). No migration (table from #963 PR A).
2026-06-03 00:05:25 +05:00
lekss361
4d492ebc50
perf(cadastre): freshness skip + grid-walk heartbeat + log noise reduction ( #187 )
...
* perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise
Three operational improvements after ekb_full v3 completed at 99.1%:
Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope
Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation
Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)
40 tests passing.
* fix(cadastre): parse NSPD DD.MM.YYYY dates (Fix K — job 18 root cause)
Job 18 (manual_list recovery 11 quarters) crashed every quarter with:
psycopg.errors.DatetimeFieldOverflow: date/time field value out of range: '13.03.2023'
NSPD returns registration_date in mixed formats:
- DD.MM.YYYY for Сооружения / ОНС / ЕНК / ЗОУИТ
- YYYY-MM-DD ISO for ЗУ / Здания
- YYYY-MM-DD HH:MM:SS with time
PG CAST(:date AS date) accepts ISO only — DD.MM.YYYY crashes.
Add _parse_nspd_date helper: handles both formats + None/empty.
Apply in upsert_construction, upsert_onc, upsert_enk, upsert_zouit, upsert_parcel
for registration_date / build_record_registration_date / legal_act_date.
3 new unit tests for the helper (43 tests total).
* fix(cadastre): address bot blockers — pivot Fix H to raw_targets + tests
Bot review on PR #187 caught:
1. cadastre_jobs has no extra_config column → Fix H crashed on KeyError
2. Missing tests for Fix H + Fix I
3. _parse_nspd_date didn't validate date ranges
Changes:
- Pivot Fix H: read skip_fresh_hours from raw_targets JSONB (existing column),
no schema migration needed
- _parse_nspd_date: validate via date.fromisoformat() — rejects 2023-13-45
- Add 3 tests for Fix H in backend/tests/workers/test_scrape_cadastre.py
- Add 2 tests for Fix I heartbeat callbacks in test_cadastre_bulk.py
- Add 1 test for parse_nspd_date range validation (was: only format)
49 tests passing.
---------
Co-authored-by: lekss361 <claudestars@proton.me>
2026-05-15 23:02:37 +03:00
