fix(avito): reconnect-retry on detail 429 under backconnect proxy #1771

Merged

lekss361 merged 1 commit from fix/avito-detail-429-reconnect into main

2026-06-19 04:42:12 +00:00

lekss361 commented

2026-06-18 20:30:28 +00:00

Owner

Problem (prod detail backfill run 217)

The detail backfill enriched only 19 then aborted at the task's max_consecutive_blocks=5 guard — all 5 blocks were HTTP 429 (backconnect conn-limit). #1768 added 403/firewall reconnect to the detail fetcher, but the 429 path only had a same-session short-retry (no reconnect). The 429s are persistent because the keep-alive session reuses one backconnect exit IP across sequential detail fetches → avito rate-limits that IP.

Fix — mirror #1769's SERP 429 reconnect into the detail fetcher

_AVITO_DETAIL_429_MAX_RETRIES 4 → 8 (same-session short-retry, first line — unchanged path).
New _AVITO_DETAIL_429_RECONNECT_RETRIES=3: on 429 short-retry exhaustion under backconnect, up to 3 ephemeral-session reconnects (fresh exit IP) before raising AvitoRateLimitedError.
Reuses the exact ephemeral retry_session/finally machinery the 403 path already has — the caller-provided shared session is never closed/rebuilt; the single ownership-guarded finally covers both 403 and 429 triggers. 403/firewall, browser mode, and no-proxy paths untouched.

Tests

New tests/scrapers/test_avito_detail_429_reconnect.py (4): short-retry-exhausted→reconnect→200 (asserts caller session not closed, ephemeral closed); fully exhausted→AvitoRateLimitedError (3 reconnects); 429→200 within budget→no reconnect (regression); no-proxy→raise immediately. -k "avito and detail": 14 passed. Ruff clean.

Last of the avito ban-vector fixes: 403 (#1764/#1765), SERP 429 (#1769), detail 403 (#1768), and now detail 429. Unblocks detail-coverage growth past the current ~11%.

## Problem (prod detail backfill run 217) The detail backfill enriched only **19** then aborted at the task's `max_consecutive_blocks=5` guard — all 5 blocks were **HTTP 429** (`backconnect conn-limit`). #1768 added 403/firewall reconnect to the detail fetcher, but the 429 path only had a same-session short-retry (no reconnect). The 429s are persistent because the keep-alive session reuses **one** backconnect exit IP across sequential detail fetches → avito rate-limits that IP. ## Fix — mirror #1769's SERP 429 reconnect into the detail fetcher - `_AVITO_DETAIL_429_MAX_RETRIES` 4 → **8** (same-session short-retry, first line — unchanged path). - New `_AVITO_DETAIL_429_RECONNECT_RETRIES=3`: on 429 short-retry exhaustion under backconnect, up to 3 **ephemeral-session** reconnects (fresh exit IP) before raising `AvitoRateLimitedError`. - Reuses the **exact** ephemeral `retry_session`/`finally` machinery the 403 path already has — the caller-provided shared session is never closed/rebuilt; the single ownership-guarded `finally` covers both 403 and 429 triggers. 403/firewall, browser mode, and no-proxy paths untouched. ## Tests New `tests/scrapers/test_avito_detail_429_reconnect.py` (4): short-retry-exhausted→reconnect→200 (asserts caller session not closed, ephemeral closed); fully exhausted→`AvitoRateLimitedError` (3 reconnects); 429→200 within budget→no reconnect (regression); no-proxy→raise immediately. `-k "avito and detail"`: **14 passed**. Ruff clean. Last of the avito ban-vector fixes: 403 (#1764/#1765), SERP 429 (#1769), detail 403 (#1768), and now detail 429. Unblocks detail-coverage growth past the current ~11%.

lekss361 added 1 commit 2026-06-18 20:30:29 +00:00

fix(avito): reconnect-retry on detail 429 under backconnect proxy

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

d65a88f704

Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched
because the detail fetcher only had a same-session 429 short-retry (no reconnect).
429s are persistent: keep-alive reuses one backconnect exit IP across sequential
detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769):
widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP)
on 429 exhaustion before raising. Caller session never touched (as with 403 path).