gendesign/tradein-mvp/backend/app/services/scrapers
lekss361 2914ff2ef2
All checks were successful
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
feat(tradein): city sweep — auto ЕКБ pipeline (#477)
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).

- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests

Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py feat(tradein): city sweep — auto ЕКБ pipeline (#477) 2026-05-23 14:38:07 +00:00
avito_detail.py feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443) 2026-05-23 12:17:15 +00:00
avito_houses.py feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449) 2026-05-23 12:37:24 +00:00
avito_imv.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
base.py feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455) 2026-05-23 13:05:20 +00:00
cian.py feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450) 2026-05-23 12:52:04 +00:00
cian_detail.py feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457) 2026-05-23 13:22:00 +00:00
cian_newbuilding.py feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) 2026-05-23 13:23:11 +00:00
cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
cian_valuation.py feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465) 2026-05-23 13:43:34 +00:00
n1.py feat(tradein): пагинация Циан/N1 3→8 страниц — плотность данных 2026-05-21 14:34:29 +03:00
yandex_detail.py feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) 2026-05-23 13:45:09 +00:00
yandex_helpers.py feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456) 2026-05-23 13:14:11 +00:00
yandex_newbuilding.py feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467) 2026-05-23 13:46:45 +00:00
yandex_realty.py feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) 2026-05-23 16:21:47 +03:00
yandex_valuation.py feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468) 2026-05-23 13:46:20 +00:00