e5fb209ced
Backend foundation for the Yandex admin scraper page (UI follows in next PR).
scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)
scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat
4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)
admin.py -- 5 new endpoints:
- GET /scraper-settings list[ScraperSetting]
- PUT /scraper-settings/{source} upsert + invalidate_cache
- POST /scrape/yandex-detail offer_url= -> detail snapshot
- POST /scrape/yandex-newbuilding slug=, id=, city=ekaterinburg -> JK snapshot
- POST /scrape/yandex-valuation address=, offer_category=, offer_type=, page= -> house meta
Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
35 lines
1.3 KiB
Python
35 lines
1.3 KiB
Python
"""Smoke: each Yandex scraper instance reads request_delay_sec from scraper_settings."""
|
|
from unittest.mock import patch
|
|
|
|
from app.services.scrapers.yandex_detail import YandexDetailScraper
|
|
from app.services.scrapers.yandex_newbuilding import YandexNewbuildingScraper
|
|
from app.services.scrapers.yandex_realty import YandexRealtyScraper
|
|
from app.services.scrapers.yandex_valuation import YandexValuationScraper
|
|
|
|
|
|
def test_yandex_realty_picks_up_delay():
|
|
with patch("app.services.scrapers.yandex_realty.get_scraper_delay", return_value=9.5):
|
|
s = YandexRealtyScraper()
|
|
assert s.request_delay_sec == 9.5
|
|
|
|
|
|
def test_yandex_detail_picks_up_delay():
|
|
with patch("app.services.scrapers.yandex_detail.get_scraper_delay", return_value=9.5):
|
|
s = YandexDetailScraper()
|
|
assert s.request_delay_sec == 9.5
|
|
|
|
|
|
def test_yandex_newbuilding_picks_up_delay():
|
|
with patch(
|
|
"app.services.scrapers.yandex_newbuilding.get_scraper_delay", return_value=9.5
|
|
):
|
|
s = YandexNewbuildingScraper()
|
|
assert s.request_delay_sec == 9.5
|
|
|
|
|
|
def test_yandex_valuation_picks_up_delay():
|
|
with patch(
|
|
"app.services.scrapers.yandex_valuation.get_scraper_delay", return_value=9.5
|
|
):
|
|
s = YandexValuationScraper()
|
|
assert s.request_delay_sec == 9.5
|