lekss361
|
e5fb209ced
|
feat(tradein): scraper_settings -- live-config request delay for Yandex scrapers + admin triggers
Backend foundation for the Yandex admin scraper page (UI follows in next PR).
scraper_settings (mig 053):
- New table source PK + request_delay_sec numeric(5,2) + updated_at + description
- Seeded with row 'yandex' = 5.0s (umbrella key for all 4 Yandex scrapers)
scraper_settings.py (new):
- get_scraper_delay(source) -> float, with 60s in-process cache + DB fallback
- Yandex sub-scrapers (yandex_detail / yandex_newbuilding / yandex_valuation)
resolve to umbrella 'yandex' key via _KEY_ALIASES
- invalidate_cache(source | None) for admin PUT side-effect
- Deferred SessionLocal import (_open_session helper) for unit-test compat
4 Yandex scrapers wired to load delay at __init__:
- yandex_realty.py / yandex_detail.py / yandex_newbuilding.py / yandex_valuation.py
- Class default bumped to 5.0; instance value overridden via get_scraper_delay(self.name)
admin.py -- 5 new endpoints:
- GET /scraper-settings list[ScraperSetting]
- PUT /scraper-settings/{source} upsert + invalidate_cache
- POST /scrape/yandex-detail offer_url= -> detail snapshot
- POST /scrape/yandex-newbuilding slug=, id=, city=ekaterinburg -> JK snapshot
- POST /scrape/yandex-valuation address=, offer_category=, offer_type=, page= -> house meta
Tests: 10 + 4 = 14 new unit tests (loader cache/alias/error paths + scraper wiring).
Ruff clean.
|
2026-05-23 18:22:27 +03:00 |
|
lekss361
|
ead2ac16c5
|
feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)
Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state
post-hydration) with selectolax DOM parsing using helpers from PR #456.
Changes:
- New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo)
- Selector: [data-test="OffersSerpItem"] x ~23 per page
- Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex),
price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date),
street address, photos (avatars.mds.yandex domain filter + size upgrade),
newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id)
- ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing
fills later), listing_segment="vtorichka"
- Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by
source_id. **kwargs swallow keeps old rooms/sort/pages callers compat.
- Preserved: BaseScraper interface, name="yandex" (matches existing rows +
avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed),
request_delay_sec=5.0, sleep_between_requests, tenacity retry.
Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio /
no-price / empty page / multi-card / URL builder / city override) — all pass.
Ruff clean.
|
2026-05-23 16:21:47 +03:00 |
|
lekss361
|
02267d0306
|
feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest
Первый запуск на сервере (вручную, один раз):
- создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
- docker network create gendesign_shared (если нет)
- docker compose -p gendesign-tradein up -d
- docker compose -p gendesign exec caddy caddy reload
|
2026-05-21 00:25:39 +03:00 |
|