gendesign/tradein-mvp/backend/app/services
lekss361 e5fb209ced feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers
Backend foundation for the Yandex admin scraper page (UI follows in next PR).

scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)

scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
  resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat

4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)

admin.py -- 5 new endpoints:
- GET    /scraper-settings           list[ScraperSetting]
- PUT    /scraper-settings/{source}  upsert + invalidate_cache
- POST   /scrape/yandex-detail       offer_url= -> detail snapshot
- POST   /scrape/yandex-newbuilding  slug=, id=, city=ekaterinburg -> JK snapshot
- POST   /scrape/yandex-valuation    address=, offer_category=, offer_type=, page= -> house meta

Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
2026-05-23 18:22:27 +03:00
..
exporters fix(tradein): security hotfix — C-3 config default + C-4 empty estimate persist + C-6 PDF URL allowlist (#435) 2026-05-23 09:38:23 +00:00
matching feat(tradein): cross-source matching service (3-tier: cadastr / fingerprint / geo / composite) (#470) 2026-05-23 14:12:16 +00:00
scrapers feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
brand.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
cache.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
cian_session.py fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) 2026-05-23 14:52:24 +00:00
estimator.py feat(tradein): estimator — Cian Valuation as on-demand 7th source (Stage 9) (#476) 2026-05-23 14:29:52 +00:00
geocoder.py fix(tradein): map-picker подставляет уличный адрес, а не название POI 2026-05-22 13:32:39 +05:00
house_metadata.py feat(tradein): кадастр зданий как источник house_metadata (#393) 2026-05-22 10:54:42 +05:00
scheduler.py feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482) 2026-05-23 14:54:25 +00:00
scrape_pipeline.py feat(tradein): city sweep — auto ЕКБ pipeline (#477) 2026-05-23 14:38:07 +00:00
scrape_runs.py feat(tradein): city sweep — auto ЕКБ pipeline (#477) 2026-05-23 14:38:07 +00:00
scraper_settings.py feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers 2026-05-23 18:22:27 +03:00
search_query.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00