lekss361/gendesign

Fork 0

feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints #484

Merged

lekss361 merged 1 commit from feat/tradein-yandex-admin-backend into main

2026-05-23 15:28:35 +00:00

lekss361 commented

2026-05-23 15:25:54 +00:00

Owner

Motivation

Backend foundation for the new Yandex admin scraper page (UI mirrors Avito at tradein-mvp/frontend/src/app/scrapers/avito/page.tsx; frontend lands in a follow-up PR).

User asked: "сделай страничку аналогичную авито только нужно добавить глобальный делэй для запросов что бы настройка работала на все запросы" → analogous Yandex page + one global delay knob that all 4 Yandex scrapers obey at runtime.

Changes

`scraper_settings` table (mig 053)

Column	Type	Note
`source`	text PRIMARY KEY	matches `BaseScraper.name` (or umbrella alias)
`request_delay_sec`	numeric(5,2)	range 1-60 sec
`updated_at`	timestamptz	NOW() on insert/update
`description`	text	human hint

Seeded with single row yandex = 5.0s — all 4 Yandex scrapers resolve to this umbrella key via _KEY_ALIASES.

`app/services/scraper_settings.py` (new)

get_scraper_delay(source: str) → float — 60s in-process cache, DB read on miss, hardcoded fallback on error
invalidate_cache(source | None) — called by admin PUT to make change live without restart
_KEY_ALIASES maps yandex_detail / yandex_newbuilding / yandex_realty_nb / yandex_valuation → umbrella "yandex"

4 Yandex scrapers wired

Each now reads delay at __init__ time:

def __init__(self) -> None:
    super().__init__()
    self.request_delay_sec = get_scraper_delay(self.name)

Class-level request_delay_sec = 5.0 kept as last-resort fallback.

5 new admin endpoints (`/api/v1/admin/...`)

Method	Path	Purpose
GET	`/scraper-settings`	List all settings rows
PUT	`/scraper-settings/{source}`	UPSERT delay + invalidate cache
POST	`/scrape/yandex-detail`	`?offer_url=` → DetailEnrichment snapshot
POST	`/scrape/yandex-newbuilding`	`?slug=&id=&city=ekaterinburg` → ЖК snapshot
POST	`/scrape/yandex-valuation`	`?address=&offer_category=&offer_type=&page=` → house meta + history count

All POST triggers are read-only debug (no DB write — main production path goes through /estimate per PR #473).

Tests

test_scraper_settings.py: 10 passed (cache hit/miss/TTL/alias/error paths)
test_yandex_scrapers_delay_wiring.py: 4 passed (each Yandex scraper picks up patched delay)
tests/ -k yandex: 139 passed (no regression)
ruff: All checks passed

Files

File	Δ
`data/sql/053_scraper_settings.sql`	+28 (new)
`app/services/scraper_settings.py`	+101 (new)
`app/api/v1/admin.py`	+206 / -1
`app/services/scrapers/yandex_realty.py`	+3 / 0
`app/services/scrapers/yandex_detail.py`	+6 / -1
`app/services/scrapers/yandex_newbuilding.py`	+6 / -1
`app/services/scrapers/yandex_valuation.py`	+6 / -1
`tests/test_scraper_settings.py`	+118 (new)
`tests/test_yandex_scrapers_delay_wiring.py`	+35 (new)

Total: 9 files, +509 / -3.

Follow-up

Frontend page (new app/scrapers/yandex/page.tsx) — separate PR after this merges, so types/contracts are locked.

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE.

## Motivation Backend foundation for the new Yandex admin scraper page (UI mirrors Avito at `tradein-mvp/frontend/src/app/scrapers/avito/page.tsx`; frontend lands in a follow-up PR). User asked: "сделай страничку аналогичную авито только нужно добавить глобальный делэй для запросов что бы настройка работала на все запросы" → analogous Yandex page + **one global delay knob** that all 4 Yandex scrapers obey at runtime. ## Changes ### `scraper_settings` table (mig 053) | Column | Type | Note | |---|---|---| | `source` | text PRIMARY KEY | matches `BaseScraper.name` (or umbrella alias) | | `request_delay_sec` | numeric(5,2) | range 1-60 sec | | `updated_at` | timestamptz | NOW() on insert/update | | `description` | text | human hint | Seeded with single row `yandex` = 5.0s — all 4 Yandex scrapers resolve to this umbrella key via `_KEY_ALIASES`. ### `app/services/scraper_settings.py` (new) - `get_scraper_delay(source: str) → float` — 60s in-process cache, DB read on miss, hardcoded fallback on error - `invalidate_cache(source | None)` — called by admin PUT to make change live without restart - `_KEY_ALIASES` maps `yandex_detail` / `yandex_newbuilding` / `yandex_realty_nb` / `yandex_valuation` → umbrella `"yandex"` ### 4 Yandex scrapers wired Each now reads delay at `__init__` time: ```python def __init__(self) -> None: super().__init__() self.request_delay_sec = get_scraper_delay(self.name) ``` Class-level `request_delay_sec = 5.0` kept as last-resort fallback. ### 5 new admin endpoints (`/api/v1/admin/...`) | Method | Path | Purpose | |---|---|---| | GET | `/scraper-settings` | List all settings rows | | PUT | `/scraper-settings/{source}` | UPSERT delay + invalidate cache | | POST | `/scrape/yandex-detail` | `?offer_url=` → DetailEnrichment snapshot | | POST | `/scrape/yandex-newbuilding` | `?slug=&id=&city=ekaterinburg` → ЖК snapshot | | POST | `/scrape/yandex-valuation` | `?address=&offer_category=&offer_type=&page=` → house meta + history count | All POST triggers are read-only debug (no DB write — main production path goes through `/estimate` per PR #473). ## Tests ``` test_scraper_settings.py: 10 passed (cache hit/miss/TTL/alias/error paths) test_yandex_scrapers_delay_wiring.py: 4 passed (each Yandex scraper picks up patched delay) tests/ -k yandex: 139 passed (no regression) ruff: All checks passed ``` ## Files | File | Δ | |---|---| | `data/sql/053_scraper_settings.sql` | +28 (new) | | `app/services/scraper_settings.py` | +101 (new) | | `app/api/v1/admin.py` | +206 / -1 | | `app/services/scrapers/yandex_realty.py` | +3 / 0 | | `app/services/scrapers/yandex_detail.py` | +6 / -1 | | `app/services/scrapers/yandex_newbuilding.py` | +6 / -1 | | `app/services/scrapers/yandex_valuation.py` | +6 / -1 | | `tests/test_scraper_settings.py` | +118 (new) | | `tests/test_yandex_scrapers_delay_wiring.py` | +35 (new) | Total: 9 files, +509 / -3. ## Follow-up Frontend page (new `app/scrapers/yandex/page.tsx`) — separate PR after this merges, so types/contracts are locked. ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE.

lekss361 added 1 commit 2026-05-23 15:25:54 +00:00

feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers e5fb209ced

Backend foundation for the Yandex admin scraper page (UI follows in next PR).

scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)

scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
  resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat

4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)

admin.py -- 5 new endpoints:
- GET    /scraper-settings           list[ScraperSetting]
- PUT    /scraper-settings/{source}  upsert + invalidate_cache
- POST   /scrape/yandex-detail       offer_url= -> detail snapshot
- POST   /scrape/yandex-newbuilding  slug=, id=, city=ekaterinburg -> JK snapshot
- POST   /scrape/yandex-valuation    address=, offer_category=, offer_type=, page= -> house meta

Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.

lekss361 merged commit b21d7c7c85 into main

2026-05-23 15:28:35 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 15:28:36 +00:00

feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)

lekss361 commented

2026-05-23 15:28:57 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Verified:

Migration 053 idempotent (BEGIN/COMMIT, IF NOT EXISTS, ON CONFLICT DO NOTHING)
Cache pattern correct: 60s TTL + Lock, invalidate via alias resolves umbrella key, DB error fallback to hardcoded default
_KEY_ALIASES verified against actual scraper.name: yandex / yandex_detail / yandex_realty_nb / yandex_valuation all map to umbrella yandex
init timing OK: estimator.py:278 creates fresh YandexValuationScraper per call → picks up new delay within seconds of admin PUT
psycopg v3 compliant: :param binds, no CAST traps, no SQL injection vectors
10+4+139 tests pass, ruff clean
No regression for /estimate production path (class-level fallback preserved)

Minor (non-blocking, follow-up):

DB-level CHECK constraint on request_delay_sec (1-60) would harden against direct SQL writes bypassing Pydantic. Currently validation only at API layer.
Multi-replica: per-process cache means PUT on replica A keeps replica B stale up to 60s. Acceptable for pilot single-replica.

Merged via deep-code-reviewer — verdict APPROVE. Verified: - Migration 053 idempotent (BEGIN/COMMIT, IF NOT EXISTS, ON CONFLICT DO NOTHING) - Cache pattern correct: 60s TTL + Lock, invalidate via alias resolves umbrella key, DB error fallback to hardcoded default - _KEY_ALIASES verified against actual scraper.name: yandex / yandex_detail / yandex_realty_nb / yandex_valuation all map to umbrella `yandex` - __init__ timing OK: estimator.py:278 creates fresh YandexValuationScraper per call → picks up new delay within seconds of admin PUT - psycopg v3 compliant: :param binds, no CAST traps, no SQL injection vectors - 10+4+139 tests pass, ruff clean - No regression for /estimate production path (class-level fallback preserved) Minor (non-blocking, follow-up): - DB-level CHECK constraint on request_delay_sec (1-60) would harden against direct SQL writes bypassing Pydantic. Currently validation only at API layer. - Multi-replica: per-process cache means PUT on replica A keeps replica B stale up to 60s. Acceptable for pilot single-replica.

lekss361 referenced this pull request

2026-05-23 15:31:47 +00:00

feat(tradein): global scraper delay setting (applies across all scrapers) #485

lekss361 referenced this pull request

2026-05-23 15:36:42 +00:00

feat(tradein-ui): Cian scraper admin page + global request delay control #486

lekss361 referenced this pull request

2026-05-23 15:39:53 +00:00

feat(tradein): global scraper delay setting (applies across all scrapers) #485

lekss361 referenced this pull request from a commit

2026-05-23 15:39:56 +00:00

feat(tradein-ui): Yandex admin scraper page — mirror Avito + global delay control