bot-backend
a50af40ad6
avito_detail_backfill (mode=browser) aborted prod run 458 with
attempted=5 enriched=0 blocked=5: the lat-null pending queue is dominated
by DEAD listings (404 — that's why they are stale coord-holes). In browser
mode a dead listing's 404 page ("Ошибка 404. Страница не найдена") has no
item-view, so _is_detail_soft_block returned True → AvitoBlockedError →
backfill counted it as `blocked` and the consecutive-block breaker aborted
the run before reaching live listings.
- New AvitoListingGoneError (subclass of common AvitoError, NOT of
AvitoBlockedError/AvitoRateLimitedError) so the backfill block-trap
does not catch it.
- New _is_detail_not_found() detector + browser-mode branch in fetch_detail,
checked BEFORE firewall/soft-block (keyed on the 404 title so it does not
swallow the generic soft-block deflect). Curl path unchanged (404 → 302/
non-200 → ValueError → failed, as before).
- Backfill: new `gone` counter; dedicated except AvitoListingGoneError branch
that is neutral to the consecutive-block breaker and marks the listing
is_active=FALSE (SAVEPOINT) so it leaves the snapshot scope.
Tests: unit for _is_detail_not_found (404 True; soft-block/item-view/empty
False), browser-mode fetch_detail raising AvitoListingGoneError (not Blocked),
and backfill gone-row marks inactive without aborting the breaker.
36 lines
1.6 KiB
Python
36 lines
1.6 KiB
Python
"""Avito-specific exceptions для anti-bot detection."""
|
||
|
||
|
||
class AvitoError(Exception):
|
||
"""Base для всех Avito scraper exceptions."""
|
||
|
||
|
||
class AvitoBlockedError(AvitoError):
|
||
"""HTTP 403 от Avito — IP-level block detected."""
|
||
|
||
|
||
class AvitoRateLimitedError(AvitoError):
|
||
"""HTTP 429 от Avito — rate limit triggered."""
|
||
|
||
|
||
class AvitoListingGoneError(AvitoError):
|
||
"""Объявление удалено / вернуло 404 (страница «Ошибка 404» вместо item-view).
|
||
|
||
Отдельный сигнал от AvitoBlockedError/AvitoRateLimitedError: 404 — НЕ анти-бот
|
||
deflect, а мёртвый листинг (его незачем ротировать/ретраить). Наследует общий
|
||
AvitoError, но НЕ AvitoBlockedError — чтобы backfill-ловушка
|
||
`except (AvitoBlockedError, AvitoRateLimitedError)` его НЕ перехватывала и не крутила
|
||
consecutive-block breaker на мёртвых координатных дырах (run 458: 5 dead → abort).
|
||
"""
|
||
|
||
|
||
class AvitoContentBlockedError(AvitoBlockedError):
|
||
"""HTTP 200 от Avito, но 0 карточек на первой странице — content-block / captcha.
|
||
|
||
Наследует AvitoBlockedError, чтобы существующие except-блоки на AvitoBlockedError
|
||
продолжали ловить этот сигнал (mark_banned, pipeline abort и т.д.).
|
||
"""
|
||
|
||
|
||
class AvitoParseError(AvitoError):
|
||
"""Cannot parse Avito HTML structure (selector changes)."""
|