Light1YT
f87b6ada4c
Root: city-sweep counters (yandex/avito/cian/domclick/newbuilding) wrote lots_fetched/lots_inserted only into scrape_runs.counters jsonb; the dedicated columns total_seen/new_count (schema 015) were never SET, so admin/observability reported total_seen=0 despite real rows saved (audit #1871/#1926). Fix (Option A): single-place extraction in scrape_runs.py via _column_counts() maps counters lots_fetched->total_seen and lots_inserted->new_count (with explicit total_seen/new_count keys taking priority). update_heartbeat, mark_done, mark_failed and mark_banned now SET both columns with COALESCE(CAST(:x AS int), col) so a missing key preserves the existing value. Covers all sweep sources at once since every pipeline routes counters through these finalizers. |
||
|---|---|---|
| .. | ||
| app | ||
| data/sql | ||
| scripts | ||
| tests | ||
| .dockerignore | ||
| Dockerfile | ||
| pyproject.toml | ||