fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy #1764

Merged

lekss361 merged 1 commit from fix/avito-403-backconnect-reconnect-retry into main

2026-06-18 18:13:21 +00:00

lekss361 commented

2026-06-18 17:42:27 +00:00

Owner

Problem (prod run 211, banned)

avito_full_load aborts the entire 7-room×price sweep on the first HTTP 403 under the backconnect proxy.

In _fetch_serp_html (curl_cffi path), max_rot = avito_proxy_max_rotations if avito_proxy_rotate_url else 0. The backconnect proxy (mproxy.site:14619) has no avito_proxy_rotate_url → max_rot=0 → the existing changeip-rotation path is skipped → the first 403 (or HTTP-200 firewall interstitial) raises AvitoBlockedError → run_avito_full_load catches it → mark_banned → whole run aborts.

Run 211 banned at room 2/7 on a 403 at room_1_komn 13–17M, page 2 (firewall=False — not DataDome, not 429). The existing 429 short-retry didn't apply. So conc=1 is not actually "clean" — it survives only until the first flagged exit IP.

Verified: the backconnect proxy assigns a fresh exit IP per new connection (5 sequential requests → 5 distinct IPs). A 403 means only the current exit IP is flagged; opening a new connection escapes it. A keep-alive curl_cffi session reuses the same CONNECT tunnel (same IP), so the retry must rebuild the session.

Fix

Mirror the existing _AVITO_429_MAX_RETRIES pattern for 403/firewall under backconnect mode:

New constants _AVITO_403_MAX_RETRIES = 5, _AVITO_403_BACKOFF_SEC = 2.0.
Extracted _build_cffi_session() from __aenter__; added _reset_cffi() (close + rebuild → new tunnel → fresh exit IP).
In _fetch_serp_html: backconnect = bool(scraper_proxy_url) and not avito_proxy_rotate_url; on 403/firewall (NOT 429) under backconnect, _reset_cffi() + backoff + retry up to 5× before raising.

Legacy single-IP changeip rotation and browser-mode paths are unchanged; the new branch is gated off whenever a rotate_url exists.

Tests

New tests/scrapers/test_avito_403_backconnect_reconnect.py (4 tests): 403→403→200 reconnect, firewall→200 reconnect, 403 exhausted → still raises, non-backconnect (rotate_url set) → no reconnect path. Full avito suite: 274 passed. Ruff clean.

Fixes the run-211 abort; needed regardless of the backconnect-purchase decision.

## Problem (prod run 211, banned) `avito_full_load` aborts the entire 7-room×price sweep on the **first HTTP 403** under the backconnect proxy. In `_fetch_serp_html` (curl_cffi path), `max_rot = avito_proxy_max_rotations if avito_proxy_rotate_url else 0`. The backconnect proxy (`mproxy.site:14619`) has **no** `avito_proxy_rotate_url` → `max_rot=0` → the existing changeip-rotation path is skipped → the first 403 (or HTTP-200 firewall interstitial) raises `AvitoBlockedError` → `run_avito_full_load` catches it → `mark_banned` → whole run aborts. Run 211 banned at room 2/7 on a 403 at `room_1_komn` 13–17M, page 2 (`firewall=False` — **not** DataDome, **not** 429). The existing 429 short-retry didn't apply. So conc=1 is not actually "clean" — it survives only until the first flagged exit IP. **Verified:** the backconnect proxy assigns a fresh exit IP per new connection (5 sequential requests → 5 distinct IPs). A 403 means only the *current* exit IP is flagged; opening a new connection escapes it. A keep-alive curl_cffi session reuses the same CONNECT tunnel (same IP), so the retry must rebuild the session. ## Fix Mirror the existing `_AVITO_429_MAX_RETRIES` pattern for 403/firewall under backconnect mode: - New constants `_AVITO_403_MAX_RETRIES = 5`, `_AVITO_403_BACKOFF_SEC = 2.0`. - Extracted `_build_cffi_session()` from `__aenter__`; added `_reset_cffi()` (close + rebuild → new tunnel → fresh exit IP). - In `_fetch_serp_html`: `backconnect = bool(scraper_proxy_url) and not avito_proxy_rotate_url`; on 403/firewall (NOT 429) under backconnect, `_reset_cffi()` + backoff + retry up to 5× before raising. Legacy single-IP changeip rotation and browser-mode paths are unchanged; the new branch is gated off whenever a `rotate_url` exists. ## Tests New `tests/scrapers/test_avito_403_backconnect_reconnect.py` (4 tests): 403→403→200 reconnect, firewall→200 reconnect, 403 exhausted → still raises, non-backconnect (rotate_url set) → no reconnect path. Full avito suite: 274 passed. Ruff clean. Fixes the run-211 abort; needed regardless of the backconnect-purchase decision.

lekss361 added 1 commit 2026-06-18 17:42:27 +00:00

fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

9dc3be56e3

Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or
firewall interstitial aborted the entire avito_full_load sweep via
AvitoBlockedError -> mark_banned. Each new connection through backconnect
gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to
_AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip
rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.