ae04dcb4b9
Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| noise_sync.py | ||
| nspd_geo.py | ||
| nspd_sync.py | ||
| objective_etl.py | ||
| poi_sync.py | ||
| pzz_sync.py | ||
| refresh_analytics.py | ||
| scrape_cadastre.py | ||
| scrape_kn.py | ||
| scrape_objective.py | ||