ee87575052
Production был заблокирован 23.05 17:48-17:55 MSK из-за tight loop без delay в detail enrichment (~20 req/sec к Avito). Все 3 подряд runs получили 403. Bundled fix: 1. AvitoBlockedError / AvitoRateLimitedError (новый avito_exceptions.py) 2. fetch_around, fetch_detail, fetch_house_catalog — raise на HTTP 403/429 3. scrape_pipeline: shared AsyncSession через весь sweep (один TLS handshake) 4. asyncio.sleep с +-20% random jitter между detail requests (default 7s) 5. Early abort: после 3 consecutive blocks — abort sweep + propagate 6. scrape_runs.mark_rate_limited (status=rate_limited vs failed/done) 7. detail_top_n default 20->10 в CitySweepStartRequest 8. logger.error вместо warning для 403/429 -> GlitchTip alert Confirmed root cause: prod logs показали 403 в 40-60ms intervals (blast pattern без delay между fetch_detail calls).
17 lines
445 B
Python
17 lines
445 B
Python
"""Avito-specific exceptions для anti-bot detection."""
|
|
|
|
|
|
class AvitoError(Exception):
|
|
"""Base для всех Avito scraper exceptions."""
|
|
|
|
|
|
class AvitoBlockedError(AvitoError):
|
|
"""HTTP 403 от Avito — IP-level block detected."""
|
|
|
|
|
|
class AvitoRateLimitedError(AvitoError):
|
|
"""HTTP 429 от Avito — rate limit triggered."""
|
|
|
|
|
|
class AvitoParseError(AvitoError):
|
|
"""Cannot parse Avito HTML structure (selector changes)."""
|