Под нагрузкой Avito отдаёт HTTP 200 с общей витриной ("Авито — Объявления
на сайте Авито", без item-view) вместо detail-страницы — мягкий анти-бот
deflect. _is_firewall_page (3 IP-block маркера) её не ловил → parse падал
ValueError("Cannot extract item_id") → generic failed, blocked=0, IP не
ротировался, enriched каскадно падал к 0 (прод runs 431→437: 115→0).
FIX 2: _is_detail_soft_block() (нет item-view + generic <title> в первых 16KB)
OR-ится в is_firewall в обоих путях fetch_detail (curl + browser) → soft-block
проходит существующую 403/firewall reconnect-машинерию (свежий backconnect
exit-IP) либо после MAX_RETRIES → AvitoBlockedError → loop-ротация.
FIX 1: snapshot ORDER BY (lat IS NULL) DESC, scraped_at DESC NULLS LAST —
сперва листинги без координат (detail-страница даёт координаты здания, #1967).
Tests: test_avito_detail_soft_block.py (reconnect-escape, exhausted->blocked,
unit-detector) + lat-priority assert в snapshot-тесте.
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
* Single snapshot SELECT at start (guarantees termination)
* Same proxy/AsyncSession path as scrape_pipeline.py step 5
* Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
* rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368)
* start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
full CI suite 1809 passed
