lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/tests
History
lekss361 9c75ec0f78 feat(tradein): city sweep — auto ЕКБ pipeline (anchors × pages + enrichment + cancel) 
- avito.py: fetch_around(pages=N, delay_override_sec) — pagination 1..N (backward compat default=1)
- scrape_pipeline.py:
  * EKB_ANCHORS const (5 точек ЕКБ)
  * run_avito_pipeline: pages + request_delay_sec params (backward compat defaults)
  * CitySweepCounters dataclass с anchors_done/total + aggregate counters
  * run_avito_city_sweep(db, run_id, *, pages_per_anchor, detail_top_n, request_delay_sec)
  * Cooperative cancel: checks scrape_runs.status каждый anchor
  * Per-anchor heartbeat + counters refresh
- scrape_runs.py NEW: utility module (create_run/update_heartbeat/mark_done/failed/cancelled/list_recent/is_cancelled)
- admin.py: 3 NEW endpoints:
  * POST /admin/scrape/avito-city-sweep — start (returns run_id, FastAPI BackgroundTasks)
  * GET  /admin/scrape/avito-city-sweep/runs?limit=N — recent runs list
  * POST /admin/scrape/avito-city-sweep/{run_id}/cancel — cooperative cancel
- 051_scrape_runs_extend.sql: ADD COLUMN params/counters/error/finished_at + cancelled в status enum
- avito-city-sweep.sh: cron wrapper с random 0-3h delay (anti-pattern-detection)
- tests/test_city_sweep.py: 16 offline smoke tests (counters, scrape_runs utils, endpoints)

Crontab entry: 0 2 * * * /opt/.../avito-city-sweep.sh (runs 02:00-05:00 UTC daily)
2026-05-23 17:33:34 +03:00
..
matching feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) 2026-05-23 14:03:35 +00:00
__init__.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
test_api_avito_stage4a.py feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460) 2026-05-23 13:34:38 +00:00
test_avito_detail_parse.py feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443) 2026-05-23 12:17:15 +00:00
test_avito_houses_parse.py feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449) 2026-05-23 12:37:24 +00:00
test_avito_imv_parse.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
test_base_save_listings.py feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455) 2026-05-23 13:05:20 +00:00
test_cian_detail.py feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457) 2026-05-23 13:22:00 +00:00
test_cian_newbuilding.py feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) 2026-05-23 13:23:11 +00:00
test_cian_serp_scraper.py feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450) 2026-05-23 12:52:04 +00:00
test_cian_session.py feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457) 2026-05-23 13:22:00 +00:00
test_cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
test_cian_valuation.py feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465) 2026-05-23 13:43:34 +00:00
test_city_sweep.py feat(tradein): city sweep — auto ЕКБ pipeline (anchors × pages + enrichment + cancel) 2026-05-23 17:33:34 +03:00
test_estimator_imv_integration.py feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) (#452) 2026-05-23 13:08:22 +00:00
test_estimator_yandex_integration.py feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473) 2026-05-23 14:14:49 +00:00
test_matching.py feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470) 2026-05-23 14:12:16 +00:00
test_scrape_pipeline.py feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451) 2026-05-23 12:55:13 +00:00
test_yandex_detail.py feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) 2026-05-23 13:45:09 +00:00
test_yandex_helpers.py feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456) 2026-05-23 13:14:11 +00:00
test_yandex_newbuilding.py feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467) 2026-05-23 13:46:45 +00:00
test_yandex_realty_serp.py feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) 2026-05-23 16:21:47 +03:00
test_yandex_valuation.py feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468) 2026-05-23 13:46:20 +00:00