lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/backend/app/workers
History
lekss361 ae04dcb4b9 perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise 
Three operational improvements after ekb_full v3 completed at 99.1%:

Fix H — per-quarter freshness skip (scrape_cadastre.py):
- enqueue_cadastre_harvest queries cad_quarter_stats.fetched_at per quarter
- Skip quarters where fetched_at >= NOW() - skip_fresh_hours (default 1h)
- extra_config.skip_fresh_hours=0 disables skip (full re-fetch)
- Saves ~30k NSPD calls on repeated runs of same scope

Fix I — heartbeat in grid-walk loop (bulk_harvest.py:_grid_walk_category):
- 225 cells x 70-300ms = 15-75s heartbeat silence
- Pass update_progress callback to _grid_walk_category
- Call every heartbeat_every (50) cells with phase progress info
- Prevents cleanup_zombies false-positive cancellation

Fix J — NSPD WMS HTTP 500 noise to debug (bulk_harvest.py:_grid_walk_category):
- NSPD WMS endpoint frequently returns HTTP 500 (server-side issue)
- Was: logger.warning each, 200+ warnings per quarter
- Now: logger.debug (visible via --log-level=DEBUG)

40 tests passing.
2026-05-15 22:12:00 +03:00
..
tasks perf(cadastre): freshness skip + grid-walk heartbeat + reduce WMS log noise 2026-05-15 22:12:00 +03:00
__init__.py init 2026-04-25 13:45:19 +03:00
beat_schedule.py refactor(workers): split celery_app.py god-object into 3 modules (#135) 2026-05-15 00:09:38 +03:00
celery_app.py feat(cadastre): bulk_harvest worker + grid-walker + admin API (#168 PR3/5) (#171) 2026-05-15 13:31:32 +03:00
lifecycle.py refactor(workers): split celery_app.py god-object into 3 modules (#135) 2026-05-15 00:09:38 +03:00