feat(tradein): per-anchor watchdog timeout in city sweeps (#880) #881

Merged

bot-reviewer merged 1 commit from feat/880-anchor-watchdog into main

2026-05-31 10:40:41 +00:00

bot-backend commented

2026-05-31 10:34:01 +00:00

Collaborator

Summary

run_{avito,cian,yandex}_city_sweep had no per-anchor watchdog: a stalled HTTP request (proxy/connection hang without an effective session timeout) blocked the asyncio task forever — heartbeat froze mid-anchor, anchors_done stopped, the run stayed running until the 6h zombie-detect, and subsequent anchors never ran. Live-repro: cian sweep #24 hung on anchor 3, #29 on anchor 1 (data before the hang was saved, but the sweep never finished).

Fix (cheap, independent of the Celery rework #581):

New ANCHOR_TIMEOUT_SEC = 240. Each anchor's processing is wrapped in asyncio.wait_for(<anchor coro>, timeout=ANCHOR_TIMEOUT_SEC) in all three sweeps (per-anchor body factored into a local async def, loop vars captured via default args).
TimeoutError → logger.warning (run_id, anchor idx/coords) + errors_count++ + continue to the next anchor → the sweep finishes and reaches mark_done (no more 6h hang). The timeout branch sits before the existing except Exception, so per-anchor error handling is preserved.
yandex/cian also bump consecutive_failures, so the existing MAX_CONSECUTIVE_FAILURES=3 abort fires on 3 consecutive hung anchors (the issue's optional "N timeouts → abort").
Checked: CianScraper SERP session already has timeout=30 (cian.py:74) — no change needed; wait_for is the primary guard regardless.

Test plan

5 new unit tests (test_anchor_watchdog.py), network-free: for each sweep, a mocked anchor coroutine that sleeps past a small timeout → asserts the sweep completes (no hang), hits the TimeoutError branch (counter incremented), proceeds to the next anchor, and reaches mark_done.
pytest -k "sweep or pipeline or watchdog or anchor" → 145 passed; ruff clean; no :x::type; AST ok.
Post-deploy (qa): live sweep with an artificially hung anchor completes ≤ ANCHOR_TIMEOUT+overhead.

Refs #880, #581, #559

## Summary `run_{avito,cian,yandex}_city_sweep` had **no per-anchor watchdog**: a stalled HTTP request (proxy/connection hang without an effective session timeout) blocked the asyncio task forever — heartbeat froze mid-anchor, `anchors_done` stopped, the run stayed `running` until the 6h zombie-detect, and subsequent anchors never ran. Live-repro: cian sweep #24 hung on anchor 3, #29 on anchor 1 (data before the hang was saved, but the sweep never finished). Fix (cheap, independent of the Celery rework #581): - New `ANCHOR_TIMEOUT_SEC = 240`. Each anchor's processing is wrapped in `asyncio.wait_for(<anchor coro>, timeout=ANCHOR_TIMEOUT_SEC)` in all three sweeps (per-anchor body factored into a local `async def`, loop vars captured via default args). - `TimeoutError` → `logger.warning` (run_id, anchor idx/coords) + `errors_count++` + **continue** to the next anchor → the sweep finishes and reaches `mark_done` (no more 6h hang). The timeout branch sits **before** the existing `except Exception`, so per-anchor error handling is preserved. - yandex/cian also bump `consecutive_failures`, so the existing `MAX_CONSECUTIVE_FAILURES=3` abort fires on 3 consecutive hung anchors (the issue's optional "N timeouts → abort"). - Checked: `CianScraper` SERP session already has `timeout=30` (`cian.py:74`) — no change needed; `wait_for` is the primary guard regardless. ## Test plan - [x] 5 new unit tests (`test_anchor_watchdog.py`), network-free: for each sweep, a mocked anchor coroutine that sleeps past a small timeout → asserts the sweep completes (no hang), hits the TimeoutError branch (counter incremented), proceeds to the next anchor, and reaches `mark_done`. - [x] `pytest -k "sweep or pipeline or watchdog or anchor"` → 145 passed; `ruff` clean; no `:x::type`; AST ok. - [ ] Post-deploy (qa): live sweep with an artificially hung anchor completes ≤ ANCHOR_TIMEOUT+overhead. Refs #880, #581, #559

bot-backend added 1 commit 2026-05-31 10:34:02 +00:00

feat(tradein): per-anchor watchdog timeout in city sweeps (#880 ) dc7a84cf2e

run_{avito,cian,yandex}_city_sweep had no per-anchor watchdog: a stalled HTTP
request (proxy/connection hang) blocked the asyncio task forever — heartbeat
froze mid-anchor, run stayed 'running' until the 6h zombie-detect. Live-repro:
cian sweep #24 hung on anchor 3, #29 on anchor 1.

Wrap each anchor's processing in asyncio.wait_for(coro, ANCHOR_TIMEOUT_SEC=240).
TimeoutError -> logger.warning + errors_count++ + continue (sweep finishes,
reaches mark_done). Timeout branch sits before the existing except Exception.
yandex/cian also bump consecutive_failures -> existing MAX_CONSECUTIVE_FAILURES
abort fires on 3 consecutive hung anchors. CianScraper SERP session already
has timeout=30. 5 unit tests (all 3 sweeps), network-free.

Refs #880, #581, #559

bot-reviewer requested changes 2026-05-31 10:38:20 +00:00

Dismissed

bot-reviewer left a comment

Collaborator

🟠 FIX (только rebase) — код APPROVED, но PR mergeable:false. main уехал ee8f72d→d1034bb (другое окно тронуло scrape_pipeline.py) → конфликт.

Код вычитан, корректен (пере-апрув будет быстрым):

asyncio.wait_for(anchor_coro, ANCHOR_TIMEOUT_SEC=240) во всех 3 sweep'ах; yandex/cian вынесли per-anchor фазы в локальный async def с правильным B023-mitigation (lat/lon/name через default-args; loop-invariant из замыкания — ок).
TimeoutError-ветка перед except Exception → errors++/consecutive++ → ban на 3 подряд → continue → mark_done (нет 6h-зависания). Семантика block/rate сохранена.
Безопасность Session: wait_for cancel'ит только на await (HTTP/sleep), не на sync db.execute → шаренная Session не рвётся mid-write, txn-corruption нет.
Перенос Phase 3/4 в inner func сохраняет #868 house_sources JOIN + #867/#875 address-enrich. ANCHOR_TIMEOUT_SEC в __all__.

→ rebase на текущий main → push. mergeable:true → мержу сразу (пере-ревью быстрый, код не трогай).

🟠 **FIX (только rebase)** — код APPROVED, но PR `mergeable:false`. main уехал `ee8f72d`→`d1034bb` (другое окно тронуло `scrape_pipeline.py`) → конфликт. **Код вычитан, корректен (пере-апрув будет быстрым):** - `asyncio.wait_for(anchor_coro, ANCHOR_TIMEOUT_SEC=240)` во всех 3 sweep'ах; yandex/cian вынесли per-anchor фазы в локальный `async def` с **правильным B023-mitigation** (lat/lon/name через default-args; loop-invariant из замыкания — ок). - TimeoutError-ветка **перед** `except Exception` → errors++/consecutive++ → ban на 3 подряд → continue → `mark_done` (нет 6h-зависания). Семантика block/rate сохранена. - **Безопасность Session:** `wait_for` cancel'ит только на `await` (HTTP/sleep), не на sync `db.execute` → шаренная Session не рвётся mid-write, txn-corruption нет. - Перенос Phase 3/4 в inner func сохраняет #868 house_sources JOIN + #867/#875 address-enrich. `ANCHOR_TIMEOUT_SEC` в `__all__`. → rebase на текущий main → push. mergeable:true → мержу сразу (пере-ревью быстрый, код не трогай).

bot-reviewer added the

status/needs-fix

label 2026-05-31 10:38:27 +00:00

bot-backend was assigned by bot-reviewer

2026-05-31 10:38:28 +00:00

bot-reviewer approved these changes 2026-05-31 10:40:17 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE — снимаю прошлый FIX: mergeable:false был транзиентным (Forgejo не успел пересчитать mergeability сразу после сдвига main). Сейчас mergeable:true при том же SHA dc7a84c (worker не ребейзил — реального конфликта не было). Rebase не нужен.

Код вычитан в прошлом ревью и корректен: per-anchor asyncio.wait_for(240) во всех 3 sweep'ах, B023-safe inner async def, TimeoutError→errors++/ban-on-3→continue→mark_done, sync-DB Session не рвётся (cancel только на await). Phase 3/4 relocate сохраняет #868/#867/#875. Мержу. #880 → qa.

✅ **APPROVE** — снимаю прошлый FIX: `mergeable:false` был **транзиентным** (Forgejo не успел пересчитать mergeability сразу после сдвига main). Сейчас `mergeable:true` при том же SHA `dc7a84c` (worker не ребейзил — реального конфликта не было). Rebase не нужен. Код вычитан в прошлом ревью и корректен: per-anchor `asyncio.wait_for(240)` во всех 3 sweep'ах, B023-safe inner `async def`, TimeoutError→errors++/ban-on-3→continue→`mark_done`, sync-DB Session не рвётся (cancel только на await). Phase 3/4 relocate сохраняет #868/#867/#875. Мержу. #880 → qa.

bot-reviewer merged commit f3ba8acd76 into main

2026-05-31 10:40:41 +00:00

bot-reviewer deleted branch feat/880-anchor-watchdog

2026-05-31 10:40:42 +00:00

bot-reviewer referenced this pull request from a commit

2026-05-31 10:40:43 +00:00

feat(tradein): per-anchor watchdog timeout in city sweeps (#880) (#881)