fix(avito): gate 403 reconnect-retry on proxy presence only (was dead code in prod) #1765

Merged

lekss361 merged 1 commit from fix/avito-403-reconnect-gate into main

2026-06-18 18:32:54 +00:00

lekss361 commented

2026-06-18 18:32:12 +00:00

Owner

Problem — #1764 was dead code in production (prod run 212 banned again)

PR #1764 added a 403/firewall reconnect-retry in _fetch_serp_html, gated by:

backconnect = bool(settings.scraper_proxy_url) and not settings.avito_proxy_rotate_url

Prod has both SCRAPER_PROXY_URL (the backconnect proxy — what the curl_cffi session actually egresses through) and a leftover AVITO_PROXY_ROTATE_URL (a changeip URL for the separate auv/browser proxy). So not avito_proxy_rotate_url → False → backconnect=False → the reconnect-retry never executed. With AVITO_PROXY_MAX_ROTATIONS=0 the changeip path is a no-op too, so a single SERP 403 still aborted the whole sweep.

Run 212 banned on a 403 at room_1_komn page 20 with zero reconnect-retries logged — confirming the branch was dead.

The two proxies are independent: _rotate_ip()/avito_proxy_rotate_url is a changeip API for the auv proxy; the curl_cffi session in __aenter__ builds proxies from settings.scraper_proxy_url only. Rebuilding that session (_reset_cffi) yields a fresh backconnect exit IP regardless of any changeip config. (Verified: the backconnect proxy hands a distinct exit IP per new connection — 5 requests → 5 IPs. We don't control which IP; we only control when to grab a new connection.)

Fix

One logic line — gate the reconnect-retry on proxy presence only:

backconnect = bool(settings.scraper_proxy_url)

Block ordering unchanged: the changeip path (rot_done < max_rot and _rotate_ip()) is still tried first when a rotate_url + max_rotations>0 are configured; reconnect-retry remains the fallback before the final raise. Under prod's max_rotations=0, changeip is skipped and the reconnect path now fires.

Tests

Replaced the bug-encoding test_non_backconnect_with_rotate_url_no_reconnect_path with test_reconnect_fires_even_when_rotate_url_set — the exact run-212 prod config (proxy + rotate_url + max_rotations=0): 403→403→200 reconnects (_reset_cffi awaited 2×, _rotate_ip not awaited).
Added test_changeip_path_tried_first_when_rotations_available (proxy + rotate_url + max_rotations=1 → _rotate_ip first, no reconnect) to guard changeip precedence.
Kept the 3 original tests (403→403→200, firewall→200, 403-exhausted→raise).

Target file: 5 passed. tests/scrapers/ -k avito: 57 passed. Ruff clean. (Pre-existing psycopg2 collection errors in unrelated modules are environment issues, not from this change.)

Follow-up to #1764; needed to make the reconnect-retry actually run in prod.

## Problem — #1764 was dead code in production (prod run 212 banned again) PR #1764 added a 403/firewall reconnect-retry in `_fetch_serp_html`, gated by: ```python backconnect = bool(settings.scraper_proxy_url) and not settings.avito_proxy_rotate_url ``` Prod has **both** `SCRAPER_PROXY_URL` (the backconnect proxy — what the curl_cffi session actually egresses through) **and** a leftover `AVITO_PROXY_ROTATE_URL` (a changeip URL for the *separate* auv/browser proxy). So `not avito_proxy_rotate_url` → False → `backconnect=False` → the reconnect-retry **never executed**. With `AVITO_PROXY_MAX_ROTATIONS=0` the changeip path is a no-op too, so a single SERP 403 still aborted the whole sweep. Run 212 banned on a 403 at `room_1_komn` page 20 with **zero** reconnect-retries logged — confirming the branch was dead. The two proxies are independent: `_rotate_ip()`/`avito_proxy_rotate_url` is a changeip API for the auv proxy; the curl_cffi session in `__aenter__` builds `proxies` from `settings.scraper_proxy_url` only. Rebuilding that session (`_reset_cffi`) yields a fresh backconnect exit IP regardless of any changeip config. (Verified: the backconnect proxy hands a distinct exit IP per new connection — 5 requests → 5 IPs. We don't control *which* IP; we only control *when* to grab a new connection.) ## Fix One logic line — gate the reconnect-retry on proxy presence only: ```python backconnect = bool(settings.scraper_proxy_url) ``` Block ordering unchanged: the changeip path (`rot_done < max_rot and _rotate_ip()`) is still tried first when a rotate_url + `max_rotations>0` are configured; reconnect-retry remains the fallback before the final `raise`. Under prod's `max_rotations=0`, changeip is skipped and the reconnect path now fires. ## Tests - Replaced the bug-encoding `test_non_backconnect_with_rotate_url_no_reconnect_path` with **`test_reconnect_fires_even_when_rotate_url_set`** — the exact run-212 prod config (proxy + rotate_url + `max_rotations=0`): 403→403→200 reconnects (`_reset_cffi` awaited 2×, `_rotate_ip` not awaited). - Added **`test_changeip_path_tried_first_when_rotations_available`** (proxy + rotate_url + `max_rotations=1` → `_rotate_ip` first, no reconnect) to guard changeip precedence. - Kept the 3 original tests (403→403→200, firewall→200, 403-exhausted→raise). Target file: 5 passed. `tests/scrapers/ -k avito`: 57 passed. Ruff clean. (Pre-existing `psycopg2` collection errors in unrelated modules are environment issues, not from this change.) Follow-up to #1764; needed to make the reconnect-retry actually run in prod.

lekss361 added 1 commit 2026-06-18 18:32:13 +00:00

fix(avito): gate 403 reconnect-retry on proxy presence only, not rotate_url absence

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

44e316c514

PR #1764's reconnect-retry was dead code in prod: the gate excluded configs
with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL
(what curl_cffi egresses through) AND a leftover changeip URL for the separate
auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403
still aborted the sweep (run 212). The curl session always uses scraper_proxy_url,
so session-rebuild rotates the exit IP regardless of changeip config.