lekss361
|
e5fb209ced
|
feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers
Backend foundation for the Yandex admin scraper page (UI follows in next PR).
scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)
scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat
4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)
admin.py -- 5 new endpoints:
- GET /scraper-settings list[ScraperSetting]
- PUT /scraper-settings/{source} upsert + invalidate_cache
- POST /scrape/yandex-detail offer_url= -> detail snapshot
- POST /scrape/yandex-newbuilding slug=, id=, city=ekaterinburg -> JK snapshot
- POST /scrape/yandex-valuation address=, offer_category=, offer_type=, page= -> house meta
Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
|
2026-05-23 18:22:27 +03:00 |
|