54a5d13c48
feat(tradein/domclick): enrich Layer A SERP parser — description, repair-inference, minhash, seller
...
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Capture extra bff-search fields in _map_item (verified live):
- description → ScrapedLot.description + infer_repair_state_from_text → repair_state
(DomClick search has no renovation-enum; text inference like cian #622 )
- sha1 description_minhash (cross-source dedup)
- agency_name from seller (defensive str|dict)
- raw_payload: is_sber_collateral, has_discount, discount_value, duplicates_offer_count, seller
Additive only; existing Layer A behavior unchanged. Deep card detail (renovation-enum,
wallType, full price-history, owners_count) is card-only = QRATOR-blocked on prod proxy
→ separate effort (needs dedicated residential IP).
2026-06-27 16:35:25 +03:00
52fc920507
fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status ( #1846 , #1968 )
...
Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP
QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path:
- GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via
BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy
-> bypasses QRATOR (prod-verified live: 20 items returned, no block).
- Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for
oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets.
- Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat
gap); per-card EKB geo-guard.
- Honest status (#1968 ): QRATOR block OR fetch errors with 0 lots -> mark_failed,
not mark_done. Parse failures are per-page resilient (break bucket, keep partial
lots) so a transient bad page never aborts the whole sweep.
- Schedule shipped DORMANT (enabled=false, pages=100) via
138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke.
42 scraper tests (price-split boundaries, parse-error resilience, honest status);
full backend suite green; ruff clean. Layer B (detail renovation/wallType/
priceHistory) deferred to #1846 follow-up.
2026-06-27 13:57:07 +03:00
b9eb478c8f
feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)
...
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.
Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.
BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
2026-06-18 09:01:50 +03:00
4c0338ce1d
fix(tradein): domklik source name, scraper delay, DataDome block detect ( #796 fixup)
2026-05-31 17:21:30 +03:00
3c5e9ad368
feat(tradein): DomClick вторичка scraper via BrowserFetcher ( #796 )
2026-05-31 16:58:12 +03:00
