Commit graph

5 commits

Author SHA1 Message Date
b9eb478c8f feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Route each scraper source (avito/cian/yandex/domclick) to its own camoufox
browser+proxy so they no longer wedge each other through a single global egress.

Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the
/fetch and /login paths are byte-for-byte the existing single-browser behavior.
When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP
(BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its
own lazy-launched lock. /login stays single-browser in Phase 1.

BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body;
all scraper callsites pass their source. No docker-compose/.env.runtime changes
(Phase 2, owner-gated).
2026-06-18 09:01:50 +03:00
278dafba76 fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела (#974)
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count /
text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех
ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится
как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а
RE_RATING требовал \s+ перед «из» → match=None.

- RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален).
  \s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий
  разделитель перед «из».
- test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы.

Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+).

Refs #974
2026-06-15 21:29:43 +03:00
feab585ad5 feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974)
All checks were successful
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 7s
- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
  houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
  (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
  fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)
2026-06-15 18:28:58 +03:00
b21d7c7c85 feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 32s
2026-05-23 15:28:34 +00:00
51650276b2 feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 37s
Deploy Trade-In / deploy (push) Successful in 21s
2026-05-23 13:46:45 +00:00