gendesign/tradein-mvp/backend/app/services/scrapers
lekss361 e5fb209ced feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers
Backend foundation for the Yandex admin scraper page (UI follows in next PR).

scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)

scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
  resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat

4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)

admin.py -- 5 new endpoints:
- GET    /scraper-settings           list[ScraperSetting]
- PUT    /scraper-settings/{source}  upsert + invalidate_cache
- POST   /scrape/yandex-detail       offer_url= -> detail snapshot
- POST   /scrape/yandex-newbuilding  slug=, id=, city=ekaterinburg -> JK snapshot
- POST   /scrape/yandex-valuation    address=, offer_category=, offer_type=, page= -> house meta

Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
2026-05-23 18:22:27 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py feat(tradein): city sweep — auto ЕКБ pipeline (#477) 2026-05-23 14:38:07 +00:00
avito_detail.py feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443) 2026-05-23 12:17:15 +00:00
avito_houses.py feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449) 2026-05-23 12:37:24 +00:00
avito_imv.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
base.py feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455) 2026-05-23 13:05:20 +00:00
cian.py feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450) 2026-05-23 12:52:04 +00:00
cian_detail.py feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457) 2026-05-23 13:22:00 +00:00
cian_newbuilding.py feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) 2026-05-23 13:23:11 +00:00
cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
cian_valuation.py feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465) 2026-05-23 13:43:34 +00:00
n1.py feat(tradein): пагинация Циан/N1 3→8 страниц — плотность данных 2026-05-21 14:34:29 +03:00
yandex_detail.py feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00
yandex_helpers.py feat(tradein): yandex_helpers.py — JSON-LD + regex + date + NLP + money helpers (#456) 2026-05-23 13:14:11 +00:00
yandex_newbuilding.py feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00
yandex_realty.py feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00
yandex_valuation.py feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00