|
|
e5f4e37415
|
feat(scrapers): route cian sessions through mobile proxy (Refs #806)
Снимает egress-часть #639: cian-скрейперы ходили с datacenter-IP (без proxies)
→ даже валидные куки не фетчили с прод-бокса. Прокидываем тот же mobile-proxy,
что у avito (#623/#805).
- config: generic scraper_proxy_url (env SCRAPER_PROXY_URL → fallback
AVITO_PROXY_URL → None/direct). Прод .env.runtime с AVITO_PROXY_URL работает
без изменений env.
- avito.py + scrape_pipeline._avito_proxies(): на settings.scraper_proxy_url
(поведение идентично).
- cian.py / cian_detail.fetch_detail / cian_session.verify_session: proxies=
{http,https} из settings.scraper_proxy_url; пусто → direct (dev no-op).
Cookie-refresh DMIR_AUTH (#639) — НЕ тронут (needs-human). 7 тестов
(precedence + dict-shape + fallback), ruff clean.
|
2026-05-30 22:01:21 +03:00 |
|
|
|
950d0f60de
|
fix(tradein): sweep shared-session (SERP+detail) через mobile proxy (#623) (#805)
Deploy Trade-In / test (push) Successful in 26s
Deploy Trade-In / build-backend (push) Successful in 42s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 34s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
|
2026-05-30 18:07:37 +00:00 |
|
|
|
6762a532eb
|
feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) (#745)
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
|
2026-05-30 15:43:38 +00:00 |
|
|
|
7ec19b3536
|
feat(scheduler): dormant Yandex Realty city-sweep (#561) (#642)
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 43s
Deploy Trade-In / deploy (push) Successful in 33s
|
2026-05-29 08:50:10 +00:00 |
|
|
|
222389cba9
|
fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
|
2026-05-23 20:08:40 +00:00 |
|
|
|
2914ff2ef2
|
feat(tradein): city sweep — auto ЕКБ pipeline (#477)
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).
- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests
Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
|
2026-05-23 14:38:07 +00:00 |
|
|
|
eb4764f75a
|
feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451)
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 28s
|
2026-05-23 12:55:13 +00:00 |
|
