lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/data/sql
History
lekss361 9c75ec0f78 feat(tradein): city sweep — auto ЕКБ pipeline (anchors × pages + enrichment + cancel) 
- avito.py: fetch_around(pages=N, delay_override_sec) — pagination 1..N (backward compat default=1)
- scrape_pipeline.py:
  * EKB_ANCHORS const (5 точек ЕКБ)
  * run_avito_pipeline: pages + request_delay_sec params (backward compat defaults)
  * CitySweepCounters dataclass с anchors_done/total + aggregate counters
  * run_avito_city_sweep(db, run_id, *, pages_per_anchor, detail_top_n, request_delay_sec)
  * Cooperative cancel: checks scrape_runs.status каждый anchor
  * Per-anchor heartbeat + counters refresh
- scrape_runs.py NEW: utility module (create_run/update_heartbeat/mark_done/failed/cancelled/list_recent/is_cancelled)
- admin.py: 3 NEW endpoints:
  * POST /admin/scrape/avito-city-sweep — start (returns run_id, FastAPI BackgroundTasks)
  * GET  /admin/scrape/avito-city-sweep/runs?limit=N — recent runs list
  * POST /admin/scrape/avito-city-sweep/{run_id}/cancel — cooperative cancel
- 051_scrape_runs_extend.sql: ADD COLUMN params/counters/error/finished_at + cancelled в status enum
- avito-city-sweep.sh: cron wrapper с random 0-3h delay (anti-pattern-detection)
- tests/test_city_sweep.py: 16 offline smoke tests (counters, scrape_runs utils, endpoints)

Crontab entry: 0 2 * * * /opt/.../avito-city-sweep.sh (runs 02:00-05:00 UTC daily)
2026-05-23 17:33:34 +03:00
..
001_trade_in_estimates.sql feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
002_core_tables.sql feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
003_seed_deals.sql fix(tradein): убрать TRUNCATE из seed — деплой больше не wipe-ит данные 2026-05-21 13:19:24 +03:00
004_extend_trade_in_estimates.sql feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
005_geocode_tracking.sql feat(tradein): geocode-missing для deals (#4 завершение) 2026-05-21 20:04:19 +03:00
006_cad_buildings.sql feat(tradein): кадастр зданий как источник house_metadata (#393) 2026-05-22 10:54:42 +05:00
007_estimate_photos.sql feat(tradein): backend загрузки фото квартиры (#394) 2026-05-22 11:05:35 +05:00
008_crm_fields.sql feat(tradein): CRM-поля в форме оценки (#395) 2026-05-22 11:15:15 +05:00
009_houses.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
010_houses_alter.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
011_listings_alter.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
012_sellers.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
013_listings_alter_seller.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
014_house_reviews.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
015_scrape_runs.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
016_listings_snapshots.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
017_house_placement_history.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
018_avito_imv_evaluations.sql feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440) 2026-05-23 11:41:44 +00:00
019_listings_alter_cian.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
020_houses_alter_cian.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
021_management_companies.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
022_agents_table.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
023_offer_price_history.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
024_houses_price_dynamics.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
025_house_reliability_checks.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
026_external_valuations.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
027_cian_session_cookies.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
028_matching_tables.sql feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies 2026-05-23 15:14:49 +03:00
029_extend_matching_valuation_dynamics.sql feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448) 2026-05-23 12:35:03 +00:00
030_avito_imv_cache_key_unique.sql fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) (#454) 2026-05-23 13:04:21 +00:00
030_listings_alter_yandex.sql feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453) 2026-05-23 13:00:54 +00:00
031_houses_alter_yandex.sql feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453) 2026-05-23 13:00:54 +00:00
032_yandex_history.sql feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453) 2026-05-23 13:00:54 +00:00
040_houses_extend.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
041_house_sources_noop.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
042_listing_sources_price_divergence_idx.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
043_house_reviews_extend.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
044_external_valuations_link.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
045_house_placement_history_extend.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
046_views.sql feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464) 2026-05-23 13:30:22 +00:00
050_search_optimization.sql feat(tradein): search matview + indexes (Phase 3.1) (#469) 2026-05-23 14:01:18 +00:00
051_scrape_runs_extend.sql feat(tradein): city sweep — auto ЕКБ pipeline (anchors × pages + enrichment + cancel) 2026-05-23 17:33:34 +03:00