Commit graph

6 commits

Author SHA1 Message Date
lekss361
5bf650c843 fix(tradein): route sweep shared-session (SERP+detail) через mobile proxy (#623)
Корень банов city-sweep: pipeline создаёт собственную shared AsyncSession и
присваивает её scraper._cffi, ПЕРЕЗАТИРАЯ проксированную сессию из
AvitoScraper.__aenter__ (#788). Через эту же сессию идёт и SERP, и detail-энричмент
(fetch_detail), и houses. Сессия создавалась без proxies → detail-страницы летели
с datacenter-IP → HTTP 429 → весь run mark_banned (scrape_runs id=9: «Avito detail
HTTP 429 …»). Единичный /admin/scrape работал, т.к. зовёт fetch_around напрямую
(проксированный _cffi из __aenter__), а не pipeline.

- scrape_pipeline.py: helper _avito_proxies() (из settings.avito_proxy_url) →
  proxies= в обе shared AsyncSession (run_avito_pipeline + run_avito_city_sweep).
  Пусто/нет env → прямое подключение (dev no-op). n1/yandex не трогаем (им прокси
  не нужен, не жжём мобильный трафик).
- test: _avito_proxies None без env / {http,https} при заданном URL.

Проверено: 18 passed (anti-bot + serp_date). ruff clean. Требует deploy (cron
city-sweep каждые 2ч на проде продолжит банить detail-429 до выката).

Refs #623, #788, #562.
2026-05-30 21:02:26 +03:00
6762a532eb feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) (#745)
Some checks failed
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 15:43:38 +00:00
7ec19b3536 feat(scheduler): dormant Yandex Realty city-sweep (#561) (#642)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 43s
Deploy Trade-In / deploy (push) Successful in 33s
2026-05-29 08:50:10 +00:00
222389cba9 fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-23 20:08:40 +00:00
2914ff2ef2 feat(tradein): city sweep — auto ЕКБ pipeline (#477)
All checks were successful
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).

- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests

Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
eb4764f75a feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 28s
2026-05-23 12:55:13 +00:00