fix(avito): tolerate 429 per-bucket instead of banning whole sweep #1769

Merged

lekss361 merged 1 commit from fix/avito-429-bucket-skip into main

2026-06-18 20:25:26 +00:00

lekss361 commented

2026-06-18 20:05:15 +00:00

Owner

Problem (prod run 216)

The merged 403 reconnect-retry works — run 216 recovered two 403s and got 2.4× further than ever (29 buckets, active 4360→5857). But it then banned on a 429 at page 36 (deep pagination of a dense bucket). The 429 path had only _AVITO_429_MAX_RETRIES=4 same-session short-retries, then raised AvitoRateLimitedError → propagated → whole run mark_banned.

Fix

A. _fetch_serp_html 429 path:

_AVITO_429_MAX_RETRIES 4 → 8 (same-session short-retry stays the first line).
New _AVITO_429_RECONNECT_RETRIES=3: on short-retry exhaustion under backconnect, do up to 3 _reset_cffi() + backoff reconnect attempts (fresh connection releases the held conn-limit slot) before raising. Mirrors the 403 reconnect block. 403/firewall logic untouched.

B. fetch_all_secondary per-bucket skip (new _AVITO_SWEEP_MAX_CONSECUTIVE_BLOCKED=4):

Both exhaustive (_walk_price_range) and incremental (_paginate_incremental_bracket) bracket calls wrapped in try/except (AvitoRateLimitedError, AvitoBlockedError). On block: log, consecutive_blocked += 1, skipped_blocked += 1, continue. On any success: reset counter.
At consecutive_blocked >= 4 → re-raise (preserves real hard-ban detection → run_avito_full_load still marks banned). So an isolated deep-bucket 429 skips that bucket and the sweep continues; only a sustained block run aborts.
DONE log now reports skipped_blocked=N.

Tests

New tests/scrapers/test_avito_429_bucket_skip.py (429 widened-retry → 200, 429-exhausted → reconnect → 200, fully exhausted → raise; per-bucket skip continues sweep; 4-consecutive → propagates; success resets counter). Adjusted one pre-existing test that hard-coded the old 429 count (now constant-derived). tests/scrapers/ -k avito: 78 passed; full tests/scrapers/: 143 passed. Ruff clean.

Complements the schedule split (incremental avoids deep pages; this hardens the weekly exhaustive run).

## Problem (prod run 216) The merged 403 reconnect-retry works — run 216 recovered two 403s and got 2.4× further than ever (29 buckets, active 4360→5857). But it then **banned on a 429 at page 36** (deep pagination of a dense bucket). The 429 path had only `_AVITO_429_MAX_RETRIES=4` same-session short-retries, then raised `AvitoRateLimitedError` → propagated → whole run `mark_banned`. ## Fix **A. `_fetch_serp_html` 429 path:** - `_AVITO_429_MAX_RETRIES` 4 → **8** (same-session short-retry stays the first line). - New `_AVITO_429_RECONNECT_RETRIES=3`: on short-retry exhaustion under backconnect, do up to 3 `_reset_cffi()` + backoff reconnect attempts (fresh connection releases the held conn-limit slot) before raising. Mirrors the 403 reconnect block. 403/firewall logic untouched. **B. `fetch_all_secondary` per-bucket skip** (new `_AVITO_SWEEP_MAX_CONSECUTIVE_BLOCKED=4`): - Both exhaustive (`_walk_price_range`) and incremental (`_paginate_incremental_bracket`) bracket calls wrapped in `try/except (AvitoRateLimitedError, AvitoBlockedError)`. On block: log, `consecutive_blocked += 1`, `skipped_blocked += 1`, `continue`. On any success: reset counter. - At `consecutive_blocked >= 4` → re-raise (preserves real hard-ban detection → `run_avito_full_load` still marks banned). So an isolated deep-bucket 429 skips that bucket and the sweep continues; only a sustained block run aborts. - DONE log now reports `skipped_blocked=N`. ## Tests New `tests/scrapers/test_avito_429_bucket_skip.py` (429 widened-retry → 200, 429-exhausted → reconnect → 200, fully exhausted → raise; per-bucket skip continues sweep; 4-consecutive → propagates; success resets counter). Adjusted one pre-existing test that hard-coded the old 429 count (now constant-derived). `tests/scrapers/ -k avito`: **78 passed**; full `tests/scrapers/`: **143 passed**. Ruff clean. Complements the schedule split (incremental avoids deep pages; this hardens the weekly exhaustive run).

lekss361 added 1 commit 2026-06-18 20:05:15 +00:00

fix(avito): tolerate 429 per-bucket instead of banning whole sweep

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

bee44ffd3d

Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36
(deep pagination, backconnect conn-limit). Widen the 429 retry budget + one
reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip
a blocked bucket and continue rather than aborting the run; only N consecutive
blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.