e5fb209ced
Backend foundation for the Yandex admin scraper page (UI follows in next PR).
scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)
scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat
4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)
admin.py -- 5 new endpoints:
- GET /scraper-settings list[ScraperSetting]
- PUT /scraper-settings/{source} upsert + invalidate_cache
- POST /scrape/yandex-detail offer_url= -> detail snapshot
- POST /scrape/yandex-newbuilding slug=, id=, city=ekaterinburg -> JK snapshot
- POST /scrape/yandex-valuation address=, offer_category=, offer_type=, page= -> house meta
Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
28 lines
1.3 KiB
PL/PgSQL
28 lines
1.3 KiB
PL/PgSQL
-- 053_scraper_settings.sql
|
|
-- Purpose: Per-source live-config for scraper request delays (anti-ban knob).
|
|
-- Loaded by app at runtime; UI can update without restart.
|
|
-- Dependencies: none (standalone settings table)
|
|
-- Sources: User request 2026-05-23 -- Yandex admin page + global delay knob.
|
|
|
|
BEGIN;
|
|
|
|
CREATE TABLE IF NOT EXISTS scraper_settings (
|
|
source text PRIMARY KEY, -- 'yandex' / 'avito' / 'cian' / 'n1' / 'yandex_detail' / ...
|
|
request_delay_sec numeric(5,2) NOT NULL, -- seconds between requests (anti-ban). Range 1-60.
|
|
updated_at timestamptz NOT NULL DEFAULT NOW(),
|
|
-- Hint about who owns this knob (UI displays, no behavior)
|
|
description text
|
|
);
|
|
|
|
COMMENT ON TABLE scraper_settings IS
|
|
'Per-source runtime config for scrapers. Currently exposes request_delay_sec '
|
|
'(anti-ban delay between HTTP requests). Editable via admin UI; backend reads '
|
|
'with 60s in-process cache. Source key matches BaseScraper.name in code.';
|
|
|
|
-- Seed: Yandex umbrella key applies to all 4 Yandex scrapers
|
|
-- (yandex / yandex_detail / yandex_newbuilding / yandex_valuation read this key).
|
|
INSERT INTO scraper_settings (source, request_delay_sec, description)
|
|
VALUES ('yandex', 5.0, 'Global delay applied to all Yandex Realty scrapers (SERP, detail, newbuilding, valuation)')
|
|
ON CONFLICT (source) DO NOTHING;
|
|
|
|
COMMIT;
|