ea24289f47
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m36s
Group A of the scraper_kit migration epic (#2277). Switches admin.py's manual "run parser" debug endpoints and scripts/ingest_domclick_jsonl.py from direct app.services.scrapers.* imports to their scraper_kit.providers.* equivalents, using the DI adapters (RealScraperConfig/RealMatcherAdapter/RealProxyProvider, app.services.scraper_settings.get_scraper_delay) already established by app.scheduler_main._run_kit_scheduler. Migrated: /scrape (AvitoScraper/CianScraper/YandexRealtyScraper + save_listings), scrape_avito_house, scrape_avito_detail, scrape_avito_imv, scrape_yandex_detail, scrape_yandex_valuation, scrape_cian_detail, cian_auto_login's BrowserFetcher. scripts/ingest_domclick_jsonl.py: ScrapedLot/save_listings + (now that #2307/ Group D ported a kit equivalent while this was in flight) DomClickDetailEnrichment/ save_detail_enrichment. Deliberately NOT migrated (documented in admin.py): scrape_yandex_newbuilding / scrape_cian_newbuilding — their kit equivalents (providers/{yandex,cian}/newbuilding.py) construct an internal BrowserFetcher(...) without the mandatory `endpoint` kwarg, so any call crashes/silently-fails regardless of caller-side DI. Bug lives in scraper_kit provider code, out of scope here (only consuming, not touching provider logic) — flagged as follow-up. Parity proven via tests/support/parity.py (assert_parity) against the exact functions each debug route now calls, on offline fixtures — no live network/DB. Refs #2305
51 lines
2 KiB
Python
51 lines
2 KiB
Python
"""Parity-proof: admin.py `scrape_cian_detail` (#2305, Group A) — legacy vs scraper_kit.
|
||
|
||
`app/api/v1/admin.py::scrape_cian_detail` теперь зовёт
|
||
`scraper_kit.providers.cian.detail.fetch_detail/save_detail_enrichment` вместо
|
||
`app.services.scrapers.cian_detail`. `_extract_price_changes` — чистая,
|
||
offline-тестируемая normalize-функция, которую `fetch_detail` вызывает при
|
||
сборке `DetailEnrichment.price_changes` (см. `tests/support/README.md`:
|
||
"Live network/DB в parity-тестах избегайте").
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import os
|
||
|
||
# cian_detail.py импортирует app.core.config при module-level import — фиктивный
|
||
# DSN мостит офлайн-запуск (mirror test_avito_detail_kit_parity.py).
|
||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
|
||
|
||
from scraper_kit.providers.cian.detail import _extract_price_changes as kit_extract_price_changes
|
||
|
||
from app.services.scrapers.cian_detail import (
|
||
_extract_price_changes as legacy_extract_price_changes,
|
||
)
|
||
from tests.support.parity import assert_parity
|
||
|
||
# Три реальных формы priceChanges: curl_cffi flat, browser-rendered nested, оба-пусто.
|
||
_OFFER_FLAT = {
|
||
"priceChanges": [
|
||
{"changeTime": "2026-01-15T00:00:00", "price": 9000000, "diffPercent": -2.5},
|
||
{"changeTime": "2026-02-01T00:00:00", "priceRub": 8800000, "diff_percent": -2.2},
|
||
]
|
||
}
|
||
_OFFER_DATA_NESTED = {
|
||
"priceChanges": [
|
||
{"changeTime": "2026-01-15T00:00:00", "priceData": {"price": 9000000}},
|
||
]
|
||
}
|
||
_STATE_EMPTY: dict[str, object] = {}
|
||
|
||
|
||
def test_parity_cian_detail_price_changes() -> None:
|
||
"""`scrape_cian_detail` (admin.py) → fetch_detail → _extract_price_changes."""
|
||
assert_parity(
|
||
legacy_fn=legacy_extract_price_changes,
|
||
kit_fn=kit_extract_price_changes,
|
||
fixtures=[
|
||
(_OFFER_FLAT, {}, _STATE_EMPTY),
|
||
({}, _OFFER_DATA_NESTED, _STATE_EMPTY),
|
||
({}, {}, _STATE_EMPTY),
|
||
],
|
||
)
|