|
|
52fc920507
|
fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status (#1846, #1968)
Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP
QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path:
- GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via
BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy
-> bypasses QRATOR (prod-verified live: 20 items returned, no block).
- Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for
oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets.
- Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat
gap); per-card EKB geo-guard.
- Honest status (#1968): QRATOR block OR fetch errors with 0 lots -> mark_failed,
not mark_done. Parse failures are per-page resilient (break bucket, keep partial
lots) so a transient bad page never aborts the whole sweep.
- Schedule shipped DORMANT (enabled=false, pages=100) via
138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke.
42 scraper tests (price-split boundaries, parse-error resilience, honest status);
full backend suite green; ruff clean. Layer B (detail renovation/wallType/
priceHistory) deferred to #1846 follow-up.
|
2026-06-27 13:57:07 +03:00 |
|
