Commit graph

2 commits

Author SHA1 Message Date
bot-backend
ea24289f47 feat(tradein/scraper-kit): migrate admin.py debug routes + domclick ingest to kit (#2305)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m36s
Group A of the scraper_kit migration epic (#2277). Switches admin.py's manual
"run parser" debug endpoints and scripts/ingest_domclick_jsonl.py from direct
app.services.scrapers.* imports to their scraper_kit.providers.* equivalents,
using the DI adapters (RealScraperConfig/RealMatcherAdapter/RealProxyProvider,
app.services.scraper_settings.get_scraper_delay) already established by
app.scheduler_main._run_kit_scheduler.

Migrated: /scrape (AvitoScraper/CianScraper/YandexRealtyScraper + save_listings),
scrape_avito_house, scrape_avito_detail, scrape_avito_imv, scrape_yandex_detail,
scrape_yandex_valuation, scrape_cian_detail, cian_auto_login's BrowserFetcher.
scripts/ingest_domclick_jsonl.py: ScrapedLot/save_listings + (now that #2307/
Group D ported a kit equivalent while this was in flight) DomClickDetailEnrichment/
save_detail_enrichment.

Deliberately NOT migrated (documented in admin.py): scrape_yandex_newbuilding /
scrape_cian_newbuilding — their kit equivalents
(providers/{yandex,cian}/newbuilding.py) construct an internal BrowserFetcher(...)
without the mandatory `endpoint` kwarg, so any call crashes/silently-fails
regardless of caller-side DI. Bug lives in scraper_kit provider code, out of
scope here (only consuming, not touching provider logic) — flagged as follow-up.

Parity proven via tests/support/parity.py (assert_parity) against the exact
functions each debug route now calls, on offline fixtures — no live network/DB.

Refs #2305
2026-07-04 00:27:01 +03:00
43b392b809 feat(tradein/domclick): local-runner + ingest tooling → main (split from parked Layer-B)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Выделяет МЕРДЖАБЕЛЬНУЮ часть DomClick-тулинга из запаркованного PR #2005
(BLOCKED #2000 — прод Layer-B скрейпер через прокси режет QRATOR-403):

- scripts/domclick_local_runner.py — ручной offline-сборщик с домашнего IP:
  SERP-enumerate (фронт, BFF-JSON режет QRATOR) + card __SSR_STATE__ +
  offer-card v3 (AVM price_prediction + sold_similar). --enum-cache (enumerate
  1 раз → файл с sentinel → следующий запуск грузит кэш и пропускает enumerate),
  wait_for_selector (SERP-карточки догидрируются), resume по id. Self-contained.
- scripts/ingest_domclick_jsonl.py — JSONL → trade-in БД (save_listings +
  house-match в SAVEPOINT + save_detail_enrichment, идемпотентно, --limit/--dry-run).
- app/services/scrapers/domclick_detail.py + domclick_exceptions.py — card-SSR
  parse + save_detail_enrichment (нужны ингесту). Импортятся ТОЛЬКО ингестом
  (operator-run); прод-runtime их не дёргает (dormant в образе).
- tests/scrapers/test_domclick_detail.py + обновлённый scripts/README.md.

НЕ включает прод Layer-B (tasks/domclick_detail_backfill.py, scheduler-расписание,
data/sql/139_*) — остаётся запаркован на PR #2005 до чистого прокси (#2000).
БД-схема не меняется (detail-колонки уже в проде, миграции нет).

Live-validated: раннер качает ЕКБ-вторичку прямо сейчас (600+ карт, 0 блоков),
ингест залил их в прод-БД (house_matched, идемпотентно).
2026-06-28 10:26:52 +03:00