fix(tradein/avito): route sidecar timeout/transport errors as transient, not crash #2523

Merged

lekss361 merged 1 commit from fix/avito-serp-timeout-graceful into main

2026-07-13 20:59:39 +00:00

bot-backend commented

2026-07-13 20:53:36 +00:00

Collaborator

Summary

avito_full_load has failed since 2026-07-02 (8 runs status=failed, some with error='').
Root cause: _fetch_serp_html_browser (serp.py) only caught httpx.HTTPStatusError from BrowserFetcher.fetch. When the sidecar fetch fails with httpx.TimeoutException/httpx.TransportError (ReadTimeout/ConnectError/RemoteProtocolError) after BrowserFetcher's own internal retry, the exception was unhandled and propagated through _paginate_incremental_bracket into run_avito_full_load's generic except Exception → mark_failed(str(exc)). str(ReadTimeout())=='' produces an empty error, and mark_failed does NOT persist done_buckets (unlike mark_banned), losing the resume checkpoint.

Root cause / fix

Widened the except clause in _fetch_serp_html_browser to also catch httpx.TimeoutException/httpx.TransportError, mirroring the existing pattern already used in _probe_total in the same file.
Non-HTTPStatusError exceptions have no HTTP status → classified status=0 with a descriptive error_text (f"{type(exc).__name__}: {exc}") and routed through the existing transient-backoff branch — confirmed status=0 / the exception text never match the soft-ban keyword checks (too_many_requests, status==503, browser unavailable, proxy may be down), so no false soft-ban/rotation is triggered.
Once the bounded transient-retry budget (_AVITO_SIDECAR_TRANSIENT_RETRIES=2) is exhausted, it raises AvitoRateLimitedError — same as a sidecar HTTPStatusError — which run_avito_full_load's dedicated except (AvitoBlockedError, AvitoRateLimitedError) routes to mark_banned, which does persist done_buckets (verified by reading pipeline.py:3289-3298 vs the generic-exception branch at 3329-3332).
Does NOT fix the underlying cause (mobile proxy instability) — that's infra/devops, out of scope here. This PR only makes the failure mode graceful (partial-save + resumable) instead of a hard crash with an empty error message.

Test plan

Added test_fetch_serp_html_browser_timeout_routes_transient_not_bare_exception in tradein-mvp/backend/tests/test_pipeline_browser_routing.py: mocks BrowserFetcher.fetch to raise httpx.ReadTimeout(""), asserts _fetch_serp_html does NOT propagate an unhandled exception, retries 3× total (1 initial + 2 transient retries), and raises AvitoRateLimitedError (the graceful banned-path) rather than a bare/generic exception.
cd tradein-mvp/backend && uv run python -m pytest tests/ -k "avito or scraper_kit_pipeline or serp" -q → 165 passed.
uv run --no-project --with ruff ruff check on both changed files → All checks passed.
No new deps → no uv lock needed.

Refs #2464 (avito prod-incident cluster)

## Summary - `avito_full_load` has failed since 2026-07-02 (8 runs `status=failed`, some with `error=''`). - Root cause: `_fetch_serp_html_browser` (`serp.py`) only caught `httpx.HTTPStatusError` from `BrowserFetcher.fetch`. When the sidecar fetch fails with `httpx.TimeoutException`/`httpx.TransportError` (ReadTimeout/ConnectError/RemoteProtocolError) after `BrowserFetcher`'s own internal retry, the exception was unhandled and propagated through `_paginate_incremental_bracket` into `run_avito_full_load`'s generic `except Exception` → `mark_failed(str(exc))`. `str(ReadTimeout())==''` produces an empty error, and `mark_failed` does NOT persist `done_buckets` (unlike `mark_banned`), losing the resume checkpoint. ## Root cause / fix - Widened the `except` clause in `_fetch_serp_html_browser` to also catch `httpx.TimeoutException`/`httpx.TransportError`, mirroring the existing pattern already used in `_probe_total` in the same file. - Non-`HTTPStatusError` exceptions have no HTTP status → classified `status=0` with a descriptive `error_text` (`f"{type(exc).__name__}: {exc}"`) and routed through the existing transient-backoff branch — confirmed `status=0` / the exception text never match the soft-ban keyword checks (`too_many_requests`, `status==503`, `browser unavailable`, `proxy may be down`), so no false soft-ban/rotation is triggered. - Once the bounded transient-retry budget (`_AVITO_SIDECAR_TRANSIENT_RETRIES=2`) is exhausted, it raises `AvitoRateLimitedError` — same as a sidecar `HTTPStatusError` — which `run_avito_full_load`'s dedicated `except (AvitoBlockedError, AvitoRateLimitedError)` routes to `mark_banned`, which **does** persist `done_buckets` (verified by reading `pipeline.py:3289-3298` vs the generic-exception branch at `3329-3332`). - **Does NOT fix the underlying cause (mobile proxy instability)** — that's infra/devops, out of scope here. This PR only makes the failure mode graceful (partial-save + resumable) instead of a hard crash with an empty error message. ## Test plan - Added `test_fetch_serp_html_browser_timeout_routes_transient_not_bare_exception` in `tradein-mvp/backend/tests/test_pipeline_browser_routing.py`: mocks `BrowserFetcher.fetch` to raise `httpx.ReadTimeout("")`, asserts `_fetch_serp_html` does NOT propagate an unhandled exception, retries 3× total (1 initial + 2 transient retries), and raises `AvitoRateLimitedError` (the graceful banned-path) rather than a bare/generic exception. - `cd tradein-mvp/backend && uv run python -m pytest tests/ -k "avito or scraper_kit_pipeline or serp" -q` → **165 passed**. - `uv run --no-project --with ruff ruff check` on both changed files → **All checks passed**. - No new deps → no `uv lock` needed. Refs #2464 (avito prod-incident cluster)

bot-backend added 1 commit 2026-07-13 20:53:36 +00:00

fix(tradein/avito): route sidecar timeout/transport errors as transient, not crash

CI Trade-In / changes (pull_request) Successful in 7s

Details

CI / changes (pull_request) Successful in 8s

Details

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI Trade-In / backend-tests (pull_request) Successful in 4m54s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

6573edba51

avito_full_load has been failing since 2026-07-02 (8 runs status=failed, some
with error=''). Root cause: _fetch_serp_html_browser only caught
httpx.HTTPStatusError from BrowserFetcher.fetch. When the sidecar fetch fails
with httpx.TimeoutException/httpx.TransportError (ReadTimeout/ConnectError/
RemoteProtocolError) after BrowserFetcher's own internal retry, the exception
was unhandled and propagated through _paginate_incremental_bracket into
run_avito_full_load's generic `except Exception` -> mark_failed(str(exc)).
str(ReadTimeout())=="" produced an empty error, and mark_failed does not
persist done_buckets (unlike mark_banned), losing the resume checkpoint.

Widen the except clause to also catch httpx.TimeoutException/TransportError,
mirroring the existing pattern in _probe_total. Non-HTTPStatusError exceptions
have no HTTP status, so they're classified status=0 with a descriptive
error_text and routed through the existing transient-backoff branch (not
soft-ban — status=0/exception text never matches the soft-ban keyword checks).
Once the bounded retry budget is exhausted, they raise AvitoRateLimitedError,
same as a sidecar HTTPStatusError -> run_avito_full_load's dedicated
except (AvitoBlockedError, AvitoRateLimitedError) -> mark_banned, which DOES
persist done_buckets, giving graceful partial-save instead of a crash.

Does NOT fix the underlying cause (mobile proxy instability) -- that's
infra/devops, out of scope. This only makes the failure mode graceful.