Commit graph

4 commits

Author SHA1 Message Date
dbef242716 feat(tradein/scrapers): domclick вторичка через bff-search-web JSON API (Layer A)
- DomClickScraper переписан на GET bff-search-web.domclick.ru/api/offers/v1
  с рекурсивным бинарным price-band bucketing (OFFSET_CAP=2000, open-band split
  через _HIGH_ANCHOR=30M); geo-guard по offerRegionName==ЕКБ / bbox
- scrape_pipeline.py: rooms=[0..5] (добавлен 5+), pages default=100,
  _DOMCLICK_PER_FETCH_S=4.0, честный статус mark_failed при 0 лотов + errors
- config.py: bff-search-web.domclick.ru добавлен в scrape_allowed_hosts
- data/sql/132_update_domclick_sweep_params.sql: UPDATE default_params (DORMANT)
2026-06-27 13:51:33 +03:00
b9eb478c8f feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.

Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.

BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
2026-06-18 09:01:50 +03:00
4c0338ce1d fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
3c5e9ad368 feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796) 2026-05-31 16:58:12 +03:00