lekss361/gendesign
1
0
Fork 0
Code Issues 57 Pull requests Projects Releases Packages Wiki Activity Actions
gendesign/tradein-mvp/backend/app/services
History
bot-backend 8204802d36 feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) 
n1.py scraper existed and was wired into the generic point-/scrape endpoint, but
had no city-sweep + schedule entry. Mirror the yandex sweep (N1 is a simple
regional SERP — search → save, no houses/detail enrich):

- scrape_pipeline.run_n1_city_sweep + N1CitySweepCounters (anchors × pages →
  save_listings; cooperative cancel; abort after N consecutive anchor failures).
- scheduler.trigger_n1_city_sweep_run + dispatch (source='n1_city_sweep').
- data/sql/084: scrape_schedules seed, enabled=true (vs dormant avito/yandex) —
  #575 activates N1 as a source; one-line UPDATE to disable if it blocks.
- POST /api/v1/admin/scrape/n1 (+ /scrape/n1/runs) for on-demand city-sweep.

estimator.sources_used is derived from {lot.source} — 'n1' flows in automatically
once n1 listings exist (no estimator change).

Tests: test_n1_city_sweep.py (5) — aggregate, run_id threading, empty-page stop,
mid-sweep cancel, consecutive-failure abort. ruff clean; 32 scheduler/sweep
regression tests pass.

Refs #575
2026-05-30 18:39:36 +03:00
..
exporters chore(tradein): close pre-commit ruff gap + add CI pytest gate (#666) (#675) 2026-05-29 19:18:48 +00:00
matching feat(tradein): same-building матчинг аналогов по house_id (#6) 2026-05-28 13:55:51 +05:00
scrapers fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) (#729) 2026-05-30 14:13:38 +00:00
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
account_quota.py feat(tradein): configurable estimate quota + new exhausted-quota copy (#658) (#671) 2026-05-29 16:07:02 +00:00
brand.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
cache.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
cian_session.py fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) 2026-05-23 14:52:24 +00:00
dadata.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
estimator.py fix(tradein): sync n_analogs to anchor count when anchor drives headline (#695) (#739) 2026-05-30 15:25:35 +00:00
geocoder.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
house_metadata.py feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) 2026-05-24 08:57:30 +00:00
image_sanitizer.py fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511) 2026-05-24 11:32:04 +00:00
scheduler.py feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) 2026-05-30 18:39:36 +03:00
scrape_pipeline.py feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) 2026-05-30 18:39:36 +03:00
scrape_runs.py fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487) 2026-05-23 20:08:40 +00:00
scraper_settings.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
search_query.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00