ae04dcb4b9
Three operational improvements after ekb_full v3 completed at 99.1%: Fix H — per-quarter freshness skip (scrape_cadastre.py): - enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter - Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h) - extra_config.skip_fresh_hours=0 disables skip (full re-fetch) - Saves ~30k NSPD calls on repeated runs of same scope Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category): - 225 cells x 70-300ms = 15-75s heartbeat silence - Pass update_progress callback to _grid_walk_category - Call every heartbeat_every (50) cells with phase progress info - Prevents cleanup_zombies false-positive cancellation Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category): - NSPD WMS endpoint frequently returns HTTP 500 (server-side issue) - Was: logger.warning each, 200+ warnings per quarter - Now: logger.debug (visible via --log-level=DEBUG) 40 tests passing. |
||
|---|---|---|
| .. | ||
| api | ||
| core | ||
| models | ||
| schemas | ||
| scrapers | ||
| services | ||
| workers | ||
| __init__.py | ||
| main.py | ||