fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy #1768

Merged

lekss361 merged 1 commit from fix/avito-detail-403-reconnect-retry into main

2026-06-18 19:14:39 +00:00

lekss361 commented

2026-06-18 19:13:33 +00:00

Owner

Problem — avito detail coverage stuck at 14% of active

fetch_detail raised on the first 403/429 with no retry (avito_detail.py), so the nightly avito_detail_backfill aborts on a few blocks — run 191 enriched only 48 listings then zombied. Result: only 669/4776 active avito listings (14%) have detail; 86% never enriched.

The detail fetcher egresses through settings.scraper_proxy_url = the live backconnect proxy (same as SERP — confirmed via _avito_proxies()), which rotates exit IP per new connection. So a 403 just means the current exit IP is flagged — exactly the situation the merged SERP fix (#1764/#1765) handles by rebuilding the session. The detail path simply never got that treatment.

(Diagnostic note: cadastral_number=0 and Domoteka fields at 25–33% are expected, not bugs — avito doesn't publish cadastral, and the Rosreestr widget is optional. This PR is purely about throughput/coverage, not those fields.)

Fix — mirror #1765 in the detail fetcher

Constants _AVITO_DETAIL_403_MAX_RETRIES=5, _AVITO_DETAIL_403_BACKOFF_SEC=2.0, _AVITO_DETAIL_429_MAX_RETRIES=4, _AVITO_DETAIL_429_BACKOFF_SEC=1.5.
_build_detail_session() helper (extracted from the own-session branch).
Curl GET → bounded retry loop: 429 → short-retry on the same session; 403/firewall under backconnect → rebuild an ephemeral session (fresh exit IP) and retry up to 5×, else raise AvitoBlockedError.
New: HTTP-200 firewall-interstitial check on the curl detail path (it previously only checked firewall in browser mode).

Session ownership (the careful bit)

The caller-provided cffi_session (shared from the backfill loop / sweep) is never closed or rebuilt — retries always use an ephemeral retry_session. finally closes session only when own_session, and always closes any retry_session created (guarded against double-close).

Tests

New tests/scrapers/test_avito_detail_403_reconnect.py (6 tests): caller-session 403→403→200 (asserts caller session NOT closed), firewall(200)→200, 403 exhausted→raise, 429 short-retry→200 (same session reused), no-proxy single-403→raise, own-session 403→200 (both sessions closed). tests/scrapers/ -k "avito and detail": 10 passed; full tests/scrapers/: 136 passed. Ruff clean.

Next: after merge+deploy, re-trigger the detail backfill and measure enriched/attempted vs run 124's 88% baseline; active coverage should climb past 14%.

## Problem — avito detail coverage stuck at 14% of active `fetch_detail` raised on the **first** 403/429 with no retry (`avito_detail.py`), so the nightly `avito_detail_backfill` aborts on a few blocks — run 191 enriched only 48 listings then zombied. Result: only 669/4776 active avito listings (14%) have detail; 86% never enriched. The detail fetcher egresses through `settings.scraper_proxy_url` = the **live backconnect** proxy (same as SERP — confirmed via `_avito_proxies()`), which rotates exit IP per new connection. So a 403 just means the current exit IP is flagged — exactly the situation the merged SERP fix (#1764/#1765) handles by rebuilding the session. The detail path simply never got that treatment. (Diagnostic note: `cadastral_number=0` and Domoteka fields at 25–33% are **expected**, not bugs — avito doesn't publish cadastral, and the Rosreestr widget is optional. This PR is purely about throughput/coverage, not those fields.) ## Fix — mirror #1765 in the detail fetcher - Constants `_AVITO_DETAIL_403_MAX_RETRIES=5`, `_AVITO_DETAIL_403_BACKOFF_SEC=2.0`, `_AVITO_DETAIL_429_MAX_RETRIES=4`, `_AVITO_DETAIL_429_BACKOFF_SEC=1.5`. - `_build_detail_session()` helper (extracted from the own-session branch). - Curl GET → bounded retry loop: 429 → short-retry on the **same** session; 403/firewall under backconnect → rebuild an **ephemeral** session (fresh exit IP) and retry up to 5×, else raise `AvitoBlockedError`. - **New:** HTTP-200 firewall-interstitial check on the curl detail path (it previously only checked firewall in browser mode). ### Session ownership (the careful bit) The caller-provided `cffi_session` (shared from the backfill loop / sweep) is **never** closed or rebuilt — retries always use an ephemeral `retry_session`. `finally` closes `session` only when `own_session`, and always closes any `retry_session` created (guarded against double-close). ## Tests New `tests/scrapers/test_avito_detail_403_reconnect.py` (6 tests): caller-session 403→403→200 (asserts caller session NOT closed), firewall(200)→200, 403 exhausted→raise, 429 short-retry→200 (same session reused), no-proxy single-403→raise, own-session 403→200 (both sessions closed). `tests/scrapers/ -k "avito and detail"`: **10 passed**; full `tests/scrapers/`: **136 passed**. Ruff clean. Next: after merge+deploy, re-trigger the detail backfill and measure enriched/attempted vs run 124's 88% baseline; active coverage should climb past 14%.

lekss361 added 1 commit 2026-06-18 19:13:34 +00:00

fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

c993243bc7

fetch_detail raised on the first 403/429, aborting the detail backfill (run 191
enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active.
Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy,
rebuild an ephemeral session (fresh exit IP) and retry up to N before raising;
short-retry 429. Never close/rebuild the caller-provided shared session. Also
add the missing HTTP-200 firewall-interstitial check on the curl detail path.