fix(avito): proactive IP rotate at sweep start + raise max_rotations 2→4 (Datadome ban recovery) #1731

Merged

lekss361 merged 1 commit from fix/avito-proactive-ip-rotate into main

2026-06-17 20:39:14 +00:00

lekss361 commented

2026-06-17 20:36:29 +00:00

Owner

Summary

Fixes the chronic avito Datadome bans (active stuck at 444; daily sweeps mark_banned / 0 inserts). Root cause: the apw mobile IP goes Datadome-stale between daily sweeps → first anchors hit the firewall → reactive _rotate_ip (capped at max_rotations=2) can't recover → 3 consecutive blocks → mark_banned.

Verified on prod (this session)

apw proxy connects + _rotate_ip/changeip works (returns fresh new_ip).
Direct proof: manually rotated the apw IP, then ran a reduced avito sweep → lots_fetched=120, lots_inserted=55, errors_count=0, 0 bans. A fresh IP passes avito Datadome; the only gap was no rotate-at-start. (NOT a proxy-pool/ops problem.)

Changes (avito.py + config.py — minimal, not base.py)

AvitoScraper.__aenter__: proactively await self._rotate_ip() once at sweep start (first step, covers both browser + curl_cffi modes), guarded on avito_proxy_rotate_url, try/except so a failed rotate logs+proceeds. _rotate_ip already asyncio.sleep(9)s so the new IP is live before the first fetch.
config.py: avito_proxy_max_rotations 2 → 4 (more reactive recovery mid-sweep).
Firewall-detect / consecutive-blocks / mark_banned logic untouched.

Test plan

pytest -k "avito or config or rotate or sweep" → 315 passed; ruff clean
rotate invoked once on aenter when rotate_url set; not when None; failed rotate doesn't abort; default max_rotations==4
Post-deploy: next daily avito sweep (06-07 UTC) → status=done, lots_inserted>0, not banned; active avito count climbs from 444

Refs #1188

## Summary Fixes the chronic avito Datadome bans (active stuck at 444; daily sweeps `mark_banned` / 0 inserts). Root cause: the apw mobile IP goes Datadome-stale between daily sweeps → first anchors hit the firewall → reactive `_rotate_ip` (capped at max_rotations=2) can't recover → 3 consecutive blocks → `mark_banned`. ## Verified on prod (this session) - apw proxy connects + `_rotate_ip`/changeip works (returns fresh new_ip). - **Direct proof:** manually rotated the apw IP, then ran a reduced avito sweep → `lots_fetched=120, lots_inserted=55, errors_count=0, 0 bans`. A fresh IP passes avito Datadome; the only gap was no rotate-at-start. (NOT a proxy-pool/ops problem.) ## Changes (avito.py + config.py — minimal, not base.py) - `AvitoScraper.__aenter__`: proactively `await self._rotate_ip()` once at sweep start (first step, covers both browser + curl_cffi modes), guarded on `avito_proxy_rotate_url`, try/except so a failed rotate logs+proceeds. `_rotate_ip` already `asyncio.sleep(9)`s so the new IP is live before the first fetch. - `config.py`: `avito_proxy_max_rotations` 2 → 4 (more reactive recovery mid-sweep). - Firewall-detect / consecutive-blocks / mark_banned logic untouched. ## Test plan - [x] `pytest -k "avito or config or rotate or sweep"` → 315 passed; ruff clean - [x] rotate invoked once on __aenter__ when rotate_url set; not when None; failed rotate doesn't abort; default max_rotations==4 - [ ] Post-deploy: next daily avito sweep (06-07 UTC) → status=done, lots_inserted>0, not banned; active avito count climbs from 444 Refs #1188

lekss361 added 1 commit 2026-06-17 20:36:30 +00:00