bot-backend
52fc920507
Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path: - GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy -> bypasses QRATOR (prod-verified live: 20 items returned, no block). - Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets. - Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat gap); per-card EKB geo-guard. - Honest status (#1968): QRATOR block OR fetch errors with 0 lots -> mark_failed, not mark_done. Parse failures are per-page resilient (break bucket, keep partial lots) so a transient bad page never aborts the whole sweep. - Schedule shipped DORMANT (enabled=false, pages=100) via 138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke. 42 scraper tests (price-split boundaries, parse-error resilience, honest status); full backend suite green; ruff clean. Layer B (detail renovation/wallType/ priceHistory) deferred to #1846 follow-up.
13 lines
614 B
Python
13 lines
614 B
Python
"""DomClick-specific exceptions для anti-bot detection."""
|
||
|
||
|
||
class DomClickBlockedError(Exception):
|
||
"""DomClick BFF вернул QRATOR block-страницу (HTTP 200 + block HTML).
|
||
|
||
QRATOR (qrator.net) — WAF/DDoS-защита domclick.ru. Блокирует datacenter-IP
|
||
и возвращает HTML с маркерами: "qrator", "bot_mitigation", "система защиты",
|
||
"403 | домклик", "captcha", "access denied".
|
||
|
||
Обходится через shared mobile proxy (BrowserFetcher(source="domclick") →
|
||
generic provider → мобильный egress).
|
||
"""
|