f87b6ada4c
fix(tradein): persist scrape_runs.total_seen/new_count columns from sweep counters ( #1926 )
...
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Root: city-sweep counters (yandex/avito/cian/domclick/newbuilding) wrote
lots_fetched/lots_inserted only into scrape_runs.counters jsonb; the dedicated
columns total_seen/new_count (schema 015) were never SET, so admin/observability
reported total_seen=0 despite real rows saved (audit #1871/#1926).
Fix (Option A): single-place extraction in scrape_runs.py via _column_counts()
maps counters lots_fetched->total_seen and lots_inserted->new_count (with
explicit total_seen/new_count keys taking priority). update_heartbeat, mark_done,
mark_failed and mark_banned now SET both columns with COALESCE(CAST(:x AS int),
col) so a missing key preserves the existing value. Covers all sweep sources at
once since every pipeline routes counters through these finalizers.
2026-06-26 22:59:27 +05:00
2f88a71915
feat(admin): unified scrape/runs + scraper/health + rotate-ip API для единой scrapers-страницы
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-20 09:25:57 +03:00
80a063b482
feat(observability): sentry alert on N consecutive scraper failures (Refs #827 ) ( #833 )
...
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 24s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 20:10:29 +00:00
222389cba9
fix(tradein/avito): anti-bot hardening — sleep + abort + shared session ( #487 )
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-23 20:08:40 +00:00
2914ff2ef2
feat(tradein): city sweep — auto ЕКБ pipeline ( #477 )
...
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).
- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests
Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
