fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters #1661

Merged

lekss361 merged 1 commit from fix/avito-anchor-timeout-and-counters into main

2026-06-17 06:07:50 +00:00

lekss361 commented

2026-06-17 06:03:00 +00:00

Owner

Summary

Fixes the spurious "avito_city_sweep = 0 lots / 5 errors" — avito was actually working (103 new listings + 30 detail-enriched on 2026-06-17), but the run counters lied.

Root cause (same class as #1659)

Each anchor is wrapped in asyncio.wait_for(run_avito_pipeline(...), timeout=ANCHOR_TIMEOUT_SEC=240). SERP+save is fast (~30s), but detail-enrich (top-N via BrowserFetcher; avito anti-bot detail pages spin to the browser service's 60s page.goto timeout) blows the 240s budget → every anchor TimeoutErrors → the counter-aggregation block is skipped → SERP lots (already saved to DB) vanish from the run counters. Prod proof: the run was exactly 1200s = 5×240s, yet 103 listings landed and 30 enriched.

Fix

Scale the per-anchor timeout for the actual workload: ANCHOR_TIMEOUT_SEC + detail_top_n × 50s + (180s if enrich_houses) ≈ 1420s for detail_top_n=20+houses. Anchors now complete instead of being guillotined.
Durable SERP counters: refactored to an inner _avito_anchor_phases coroutine that writes lots_fetched/inserted/updated to the sweep-level counters immediately after save_listings, before the houses/detail phases — so a detail-phase timeout no longer erases the SERP counts.
Fail-fast: abort an anchor's detail phase after 5 consecutive detail timeouts (avoid wasting 20×60s on a fully-blocked anchor).
ANCHOR_TIMEOUT_SEC global unchanged (yandex/other paths keep their own logic).

Test plan

Full gate: 1847 passed, 2 deselected (added: timeout-scaling formula test + SERP-counters-durable-after-detail-timeout tests; updated anti-bot/imv tests to patch fetch_around)
ruff clean
Post-deploy smoke: trigger avito_city_sweep → run completes (not 5×240s) with counters reflecting real lots_fetched/inserted

Refs #759

## Summary Fixes the spurious "avito_city_sweep = 0 lots / 5 errors" — avito was actually working (103 new listings + 30 detail-enriched on 2026-06-17), but the run **counters** lied. ### Root cause (same class as #1659) Each anchor is wrapped in `asyncio.wait_for(run_avito_pipeline(...), timeout=ANCHOR_TIMEOUT_SEC=240)`. SERP+save is fast (~30s), but detail-enrich (top-N via BrowserFetcher; avito anti-bot detail pages spin to the browser service's 60s `page.goto` timeout) blows the 240s budget → **every anchor `TimeoutError`s** → the counter-aggregation block is skipped → SERP lots (already saved to DB) vanish from the run counters. Prod proof: the run was exactly **1200s = 5×240s**, yet 103 listings landed and 30 enriched. ### Fix - **Scale the per-anchor timeout** for the actual workload: `ANCHOR_TIMEOUT_SEC + detail_top_n × 50s + (180s if enrich_houses)` ≈ 1420s for detail_top_n=20+houses. Anchors now complete instead of being guillotined. - **Durable SERP counters**: refactored to an inner `_avito_anchor_phases` coroutine that writes `lots_fetched/inserted/updated` to the sweep-level counters **immediately after `save_listings`**, before the houses/detail phases — so a detail-phase timeout no longer erases the SERP counts. - **Fail-fast**: abort an anchor's detail phase after 5 consecutive detail timeouts (avoid wasting 20×60s on a fully-blocked anchor). - `ANCHOR_TIMEOUT_SEC` global unchanged (yandex/other paths keep their own logic). ## Test plan - [x] Full gate: **1847 passed, 2 deselected** (added: timeout-scaling formula test + SERP-counters-durable-after-detail-timeout tests; updated anti-bot/imv tests to patch `fetch_around`) - [x] ruff clean - [ ] Post-deploy smoke: trigger avito_city_sweep → run completes (not 5×240s) with counters reflecting real lots_fetched/inserted Refs #759

lekss361 added 1 commit 2026-06-17 06:03:01 +00:00

fix(avito-sweep): scale per-anchor timeout by detail budget + preserve SERP counters before detail phase

CI / changes (push) Successful in 7s

Details

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

40c044e2a6

- Compute _avito_anchor_timeout = ANCHOR_TIMEOUT_SEC + detail_top_n * 50s + 180s (houses).
  Eliminates false TimeoutError on anchors with many detail requests.
- Refactor run_avito_city_sweep to use inner _avito_anchor_phases coroutine (mirrors
  Yandex/Cian pattern): counters updated immediately after SERP+save (nonlocal), before
  detail phase. TimeoutError in detail no longer erases lots_fetched/inserted.
- Add consecutive-detail-timeout abort after 5 consecutive TimeoutErrors.
- Update test_anchor_watchdog: new tests for timeout scaling formula and counter durability.
- Update test_avito_anti_bot + test_sweep_imv_phase: patch AvitoScraper.fetch_around
  directly (run_avito_city_sweep no longer calls run_avito_pipeline); use ExitStack
  instead of starred-expression unpacking (invalid syntax in Python 3.12/3.14).