fix(yandex): rotate proxy IP + retry on tarpit-timeout/captcha in combos sweep #1664

Merged

lekss361 merged 1 commit from fix/yandex-rotate-on-tarpit into main

2026-06-17 07:13:48 +00:00

lekss361 commented

2026-06-17 07:13:12 +00:00

Owner

Summary

Makes the nightly yandex combos sweep resilient to yandex's bandwidth-tarpit anti-bot.

Diagnosis (prod 2026-06-17)

yandex anti-bot = bandwidth tarpit, not captcha/block: it returns HTTP 200 but throttles flagged mobile-proxy IPs to ~77 B/s for realty SERP pages → the large SERP never finishes in the 30s timeout → system curl exits 28 → status_code=0 → fetch_around returns []. Measured: tarpitted IP = 77 B/s; after _rotate_ip a fresh IP = 24807 B/s, full 459KB SERP in 18.5s with 900+ offers. So rotation defeats the tarpit — fresh IPs aren't all flagged.

The bug: the nightly combos path (fetch_around_multi_room → fetch_around) did NOT rotate/retry — on status_code != 200 it just return []. So when the current IP was tarpitted, all 30 combos returned [] and the IP was never rotated → run yields 0 (prod run 117). The exhaustive path already rotated; the nightly sweep used the non-rotating path.

Fix

fetch_around now retries up to _YANDEX_TARPIT_MAX_RETRIES=2 times (3 attempts):

status_code == 0 (tarpit/timeout) → rotate IP + sleep(2) + retry
HTTP 200 + captcha → rotate + retry
other non-200 (e.g. 404) → return [] (no rotate)
HTTP 200 + clean → parse + return
exhausted → return []

Worst case 30 combos × 3 fetches × ~17s ≈ 1530s, within the #1659 combos watchdog (2190s). Most fresh IPs succeed on attempt 0.

Test plan

Full gate: 1851 passed, 2 deselected (4 new tests: rotate-on-0, rotate-on-captcha, exhaust→empty, no-rotate-on-404)
ruff clean
Prod mechanism: fresh IP → 20 lots on attempt 0; tarpit→rotate→fresh proven (77→24807 B/s)
Post-deploy: nightly yandex run recovers from tarpitted IPs (lots > 0 consistently)

Refs #759, #1658

## Summary Makes the nightly yandex combos sweep resilient to yandex's bandwidth-tarpit anti-bot. ### Diagnosis (prod 2026-06-17) yandex anti-bot = **bandwidth tarpit**, not captcha/block: it returns HTTP 200 but throttles flagged mobile-proxy IPs to **~77 B/s** for realty SERP pages → the large SERP never finishes in the 30s timeout → system curl exits 28 → `status_code=0` → `fetch_around` returns `[]`. Measured: tarpitted IP = 77 B/s; after `_rotate_ip` a **fresh IP = 24807 B/s, full 459KB SERP in 18.5s with 900+ offers**. So rotation defeats the tarpit — fresh IPs aren't all flagged. **The bug:** the nightly combos path (`fetch_around_multi_room` → `fetch_around`) did NOT rotate/retry — on `status_code != 200` it just `return []`. So when the current IP was tarpitted, all 30 combos returned `[]` and the IP was never rotated → run yields 0 (prod run 117). The exhaustive path already rotated; the nightly sweep used the non-rotating path. ### Fix `fetch_around` now retries up to `_YANDEX_TARPIT_MAX_RETRIES=2` times (3 attempts): - `status_code == 0` (tarpit/timeout) → rotate IP + `sleep(2)` + retry - HTTP 200 + captcha → rotate + retry - other non-200 (e.g. 404) → return `[]` (no rotate) - HTTP 200 + clean → parse + return - exhausted → return `[]` Worst case 30 combos × 3 fetches × ~17s ≈ 1530s, within the #1659 combos watchdog (2190s). Most fresh IPs succeed on attempt 0. ## Test plan - [x] Full gate: **1851 passed, 2 deselected** (4 new tests: rotate-on-0, rotate-on-captcha, exhaust→empty, no-rotate-on-404) - [x] ruff clean - [x] Prod mechanism: fresh IP → 20 lots on attempt 0; tarpit→rotate→fresh proven (77→24807 B/s) - [ ] Post-deploy: nightly yandex run recovers from tarpitted IPs (lots > 0 consistently) Refs #759, #1658

lekss361 added 1 commit 2026-06-17 07:13:12 +00:00

fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience)

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 6s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI / changes (push) Successful in 8s

Details

CI / backend-tests (push) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

b19c4a0b8c

Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.