Commit graph

3 commits

Author SHA1 Message Date
cc3ff7c3d2 feat(tradein/scrapers): detail_backfill через curl_cffi+backconnect вместо браузера/auv (развязка прокси)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
SERP (full_load/city_sweep) и detail_backfill делили один прокси-аккаунт auv (~5 коннектов).
Browser-фетч в backfill открывал десятки CONNECT-туннелей → cap превышался → HTTP 500/краши.

- config.py: avito_detail_backfill_use_curl=True (ENV AVITO_DETAIL_BACKFILL_USE_CURL)
- avito_detail_backfill.py: browser_mode гейтится флагом; use_curl=True → session=None,
  browser_fetcher=None → fetch_detail строит _build_detail_session() через scraper_proxy_url
  (backconnect mproxy, авто-ротация, 1 коннект/запрос)
- tests: 2 новых теста (use_curl=True → BrowserFetcher не создаётся; False → создаётся)
2026-06-20 15:55:24 +03:00
37fc50ad08 fix(scrapers): avito_detail_backfill — снапшот ЕКБ+active, detail browser-only без curl-fallback на бан (#1814)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-19 21:58:35 +03:00
227e82dc01 feat(tradein): avito detail-enrichment nightly backfill (#1551)
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
  * Single snapshot SELECT at start (guarantees termination)
  * Same proxy/AsyncSession path as scrape_pipeline.py step 5
  * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
  * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368)
  * start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
  budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
  full CI suite 1809 passed
2026-06-16 13:25:29 +03:00