575c233e51
All checks were successful
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m2s
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 8s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
app.services.scrapers.{cian,cian_detail,yandex_realty,domclick,domclick_exceptions,
avito_houses} имели 0 runtime-импортёров после E1 (scrape_pipeline.py удалён,
#2405) — admin.py/cian_history_backfill.py уже на scraper_kit.providers.*.
Удаление:
- 6 модулей целиком (avito_houses.py, cian.py, cian_detail.py, domclick.py,
domclick_exceptions.py, yandex_realty.py)
- 3 осиротевших dev-sweep скрипта (local-sweep-ekb{,-cian,-yandex}.py) —
эксклюзивно драйвили удалённые легаси-скраперы, kit-путь их заменяет
Тесты (18 файлов DELETE — эксклюзивно тестировали удалённые модули или были
чистым legacy-vs-kit parity, чей легаси-эталон исчез):
test_cian_detail.py, test_cian_serp_scraper.py, tests/scrapers/test_cian_exhaustive.py,
test_yandex_realty_serp.py, tests/scrapers/test_domclick.py,
test_avito_houses_dedup.py, test_avito_houses_parse.py,
test_houses_rating_filter_entries_1789.py, test_scraper_kit_domclick_golden_parity.py,
tests/scrapers/test_admin_{avito,cian,yandex}_kit_parity.py.
CONVERT (файл покрывает СМЕШАННЫЙ live+legacy набор — вырезаны только
секции удалённых модулей, live-покрытие сохранено):
- test_scraper_kit_cian_golden_parity.py — SERP/`_format_address`/detail
(cian.py/cian_detail.py) убраны; extract_state (cian_state_parser, live)/
valuation (cian_valuation, live)/newbuilding (cian_newbuilding, live)/
no-app-imports guard — остались
- test_scraper_kit_yandex_golden_parity.py — SERP `_entity_to_lot`/
`_parse_gate_json`/`_build_url` (yandex_realty) убраны; normalize_house_type
(live)/newbuilding parse (yandex_newbuilding, live)/no-app-imports guard —
остались
- test_scraper_kit_bisection_parity.py — cian/domclick/yandex-ветки убраны;
avito-ветка (live, единственная оставшаяся легаси-бисекция) — осталась
полностью, включая full-seed-grid
- test_scraper_kit_group_c_backfill_kit_parity.py — cian fetch_detail
browser_fetcher parity-тест убран; avito/house_imv_backfill/
cian_history_backfill/ekb_geoportal тесты — остались
- test_scraper_kit_pricehistory_session_parity.py — fetch_detail
config-injection/proxy-wiring тесты (cian_detail) убраны; extract_state
(cian_state_parser, live)/ScrapedLot (base, live) parity — остались
- test_yandex_scrapers_delay_wiring.py — yandex_realty delay-тест убран
(kit-эквивалент уже покрыт test_kit_serp_proxy_pool.py); yandex_newbuilding
(live)/yandex_detail/yandex_valuation (kit) тесты — остались
- test_scraper_proxy.py — yandex_realty BrowserFetcher-тест убран (redundant
с test_kit_serp_proxy_pool.py::test_yandex_serp_no_pool_by_default)
RETARGET на kit (символ живёт, только источник поменялся):
- test_domclick_sweep.py — `_map_item` → scraper_kit.providers.domclick.serp
- test_yandex_city_sweep.py — DEFAULT_PRICE_RANGES/ROOM_PATH →
scraper_kit.providers.yandex.serp
- test_pipeline_browser_routing.py — fetch_house_catalog →
scraper_kit.providers.avito.houses (+ KitAvitoBlockedError для firewall-теста)
- test_snapshot_writer.py — DetailEnrichment/save_detail_enrichment →
scraper_kit.providers.cian.detail (byte-identical strangler-копия)
- test_backfill_wave2.py — DetailEnrichment → scraper_kit.providers.cian.detail
- tests/scrapers/test_avito_unix_date_tz_consistency.py — avito_houses →
scraper_kit.providers.avito.houses (houses-ветка _unix_to_date/RUS_MONTHS
consolidation-проверки)
Stale-комментарии поправлены в cian_price_history.py, tests/scrapers/
test_domclick_detail.py (описывали ещё-живой domclick.py/domclick_exceptions.py).
grep -rn scrapers\.\(cian\|cian_detail\|yandex_realty\|domclick\|domclick_exceptions\|
avito_houses\) app/ scripts/ packages/ — только historical footnotes
("strangler-копия app.services.scrapers.X", past-tense "удалён/deleted").
Backend suite: 2811 passed, 6 skipped, 1 failed (known flake test_search_cache_hit
#2208, не регрессия). ruff check чист на всех изменённых файлах.
274 lines
10 KiB
Python
274 lines
10 KiB
Python
"""Stage 1 #915 — route avito pipeline SERP+detail+houses through BrowserFetcher.
|
||
|
||
Тесты без сети и без БД. BrowserFetcher мокируется объектом с async fetch().
|
||
|
||
Покрытие:
|
||
1. fetch_detail(browser_fetcher=mock) → DetailEnrichment (browser branch taken)
|
||
2. fetch_detail(browser_fetcher=mock) → AvitoBlockedError при firewall HTML
|
||
3. fetch_house_catalog(browser_fetcher=mock) → browser branch taken
|
||
4. Curl mode (default) — browser_fetcher=None, curl path используется
|
||
|
||
Legacy `run_avito_pipeline` browser/curl-mode integration tests (было ранее секцией 4)
|
||
удалены вместе с `app.services.scrape_pipeline` (#2397 Part E1) — browser-routing
|
||
покрытие на уровне `fetch_detail`/`fetch_house_catalog`/`AvitoScraper._fetch_serp_html`
|
||
ниже остаётся нетронутым.
|
||
|
||
`fetch_house_catalog` ретаргетирован на `scraper_kit.providers.avito.houses` (#2397
|
||
Part E2) — legacy `app.services.scrapers.avito_houses` удалён (0 runtime-импортёров,
|
||
kit — единственный живой путь).
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import os
|
||
import urllib.parse
|
||
from pathlib import Path
|
||
from unittest.mock import AsyncMock, MagicMock
|
||
|
||
import pytest
|
||
|
||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
|
||
|
||
from scraper_kit.avito_exceptions import AvitoBlockedError as KitAvitoBlockedError
|
||
from scraper_kit.providers.avito.houses import fetch_house_catalog
|
||
|
||
from app.services.scrapers.avito_detail import fetch_detail
|
||
from app.services.scrapers.avito_exceptions import AvitoBlockedError
|
||
|
||
FIXTURES = Path(__file__).parent / "fixtures"
|
||
|
||
# Минимальный HTML для fetch_detail — достаточен для parse_detail_html
|
||
DETAIL_HTML = """
|
||
<html><body>
|
||
<div data-marker="item-view/item-id">№ 99887766 · 18 мая в 10:00 · 100 просмотров</div>
|
||
<span itemprop="price" content="5000000">5 000 000 ₽</span>
|
||
<div data-marker="item-map-wrapper"
|
||
data-map-lat="56.8" data-map-lon="60.6" data-location-id="1"></div>
|
||
<div data-marker="item-view/item-params">
|
||
<ul>
|
||
<li>Количество комнат: 2</li>
|
||
<li>Общая площадь: 50 м²</li>
|
||
<li>Этаж: 5 из 10</li>
|
||
</ul>
|
||
</div>
|
||
</body></html>
|
||
"""
|
||
|
||
# Firewall HTML — содержит маркер который _is_firewall_page ловит
|
||
FIREWALL_HTML = """
|
||
<html><head><title>Доступ ограничен</title></head>
|
||
<body><div class="firewall-container">Доступ ограничен</div></body></html>
|
||
"""
|
||
|
||
_MINIMAL_STATE = {
|
||
"data": {
|
||
"data": {
|
||
"page": {
|
||
"placeholders": [
|
||
{
|
||
"type": "housePage",
|
||
"props": {
|
||
"developmentData": {
|
||
"avitoId": 12345,
|
||
"id": "abc123",
|
||
"title": "ЖК Тест",
|
||
"address": "ул. Тестовая, 1",
|
||
"fullAddress": "Екатеринбург, ул. Тестовая, 1",
|
||
"coords": {"lat": 56.8, "lng": 60.6},
|
||
"aboutDevelopment": {"expandParams": {"items": []}},
|
||
"developer": {},
|
||
"mapPreview": {},
|
||
}
|
||
},
|
||
}
|
||
]
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
_encoded_state = urllib.parse.quote(json.dumps(_MINIMAL_STATE, ensure_ascii=False))
|
||
HOUSES_HTML = f"""
|
||
<html><body>
|
||
<script>window.__preloadedState__ = "{_encoded_state}";</script>
|
||
</body></html>
|
||
"""
|
||
|
||
|
||
# ── Helpers ──────────────────────────────────────────────────────────────────
|
||
|
||
|
||
def make_mock_fetcher(html: str) -> MagicMock:
|
||
"""Создаёт mock BrowserFetcher с async fetch(), возвращающим заданный HTML."""
|
||
mock = MagicMock()
|
||
mock.fetch = AsyncMock(return_value=html)
|
||
return mock
|
||
|
||
|
||
# ── 1. fetch_detail browser branch → DetailEnrichment ────────────────────────
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_detail_browser_returns_enrichment() -> None:
|
||
"""browser_fetcher.fetch() вызывается, curl AsyncSession не трогается."""
|
||
mock_fetcher = make_mock_fetcher(DETAIL_HTML)
|
||
item_url = "/ekaterinburg/kvartiry/test_99887766-1234"
|
||
|
||
result = await fetch_detail(item_url, browser_fetcher=mock_fetcher)
|
||
|
||
# browser path взят
|
||
mock_fetcher.fetch.assert_awaited_once()
|
||
called_url = mock_fetcher.fetch.call_args[0][0]
|
||
assert called_url.startswith("https://www.avito.ru"), f"unexpected url: {called_url}"
|
||
assert called_url.endswith(item_url)
|
||
|
||
assert result.item_id == "99887766"
|
||
assert result.price_rub == 5_000_000
|
||
assert result.rooms == 2
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_detail_browser_full_url_passthrough() -> None:
|
||
"""Если item_url уже абсолютный — передаётся без urljoin."""
|
||
mock_fetcher = make_mock_fetcher(DETAIL_HTML)
|
||
full_url = "https://www.avito.ru/ekaterinburg/kvartiry/test_99887766-1234"
|
||
|
||
result = await fetch_detail(full_url, browser_fetcher=mock_fetcher)
|
||
|
||
called_url = mock_fetcher.fetch.call_args[0][0]
|
||
assert called_url == full_url
|
||
assert result.item_id == "99887766"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_detail_browser_no_curl_session_used() -> None:
|
||
"""cffi_session не используется когда browser_fetcher передан."""
|
||
mock_fetcher = make_mock_fetcher(DETAIL_HTML)
|
||
mock_session = MagicMock()
|
||
mock_session.get = AsyncMock() # НЕ должен быть вызван
|
||
|
||
result = await fetch_detail(
|
||
"/ekaterinburg/kvartiry/test_99887766-1234",
|
||
cffi_session=mock_session,
|
||
browser_fetcher=mock_fetcher,
|
||
)
|
||
|
||
mock_session.get.assert_not_awaited()
|
||
assert result.item_id == "99887766"
|
||
|
||
|
||
# ── 2. fetch_detail browser firewall → AvitoBlockedError ─────────────────────
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_detail_browser_firewall_raises() -> None:
|
||
"""Firewall HTML через browser path → AvitoBlockedError."""
|
||
mock_fetcher = make_mock_fetcher(FIREWALL_HTML)
|
||
|
||
with pytest.raises(AvitoBlockedError, match="browser-mode"):
|
||
await fetch_detail(
|
||
"/ekaterinburg/kvartiry/blocked_item-9999",
|
||
browser_fetcher=mock_fetcher,
|
||
)
|
||
|
||
|
||
# ── 3. fetch_house_catalog browser branch ────────────────────────────────────
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_house_catalog_browser_branch_taken() -> None:
|
||
"""browser_fetcher.fetch() вызывается, curl path не трогается."""
|
||
mock_fetcher = make_mock_fetcher(HOUSES_HTML)
|
||
house_path = "/catalog/houses/ekaterinburg/ul_test/3171365"
|
||
|
||
result = await fetch_house_catalog(house_path, browser_fetcher=mock_fetcher)
|
||
|
||
mock_fetcher.fetch.assert_awaited_once()
|
||
called_url = mock_fetcher.fetch.call_args[0][0]
|
||
assert "avito.ru" in called_url
|
||
assert house_path in called_url
|
||
|
||
assert result.house.ext_id == 12345
|
||
assert result.house.title == "ЖК Тест"
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_house_catalog_browser_no_curl_used() -> None:
|
||
"""cffi_session.get не вызывается в browser mode."""
|
||
mock_fetcher = make_mock_fetcher(HOUSES_HTML)
|
||
mock_session = MagicMock()
|
||
mock_session.get = AsyncMock()
|
||
|
||
await fetch_house_catalog(
|
||
"/catalog/houses/ekaterinburg/ul_test/3171365",
|
||
cffi_session=mock_session,
|
||
browser_fetcher=mock_fetcher,
|
||
)
|
||
|
||
mock_session.get.assert_not_awaited()
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_house_catalog_browser_firewall_raises() -> None:
|
||
"""Firewall HTML через browser path → AvitoBlockedError (kit)."""
|
||
mock_fetcher = make_mock_fetcher(FIREWALL_HTML)
|
||
|
||
with pytest.raises(KitAvitoBlockedError, match="browser-mode"):
|
||
await fetch_house_catalog(
|
||
"/catalog/houses/ekaterinburg/ul_test/9999",
|
||
browser_fetcher=mock_fetcher,
|
||
)
|
||
|
||
|
||
# ── 4. Curl mode (default) — browser_fetcher stays None ──────────────────────
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_detail_curl_mode_no_browser_fetcher() -> None:
|
||
"""Если browser_fetcher=None, curl path используется (собственная сессия)."""
|
||
from unittest.mock import patch as _patch
|
||
|
||
# Мокаем создание собственной сессии чтобы не нужен реальный curl_cffi
|
||
mock_resp = MagicMock()
|
||
mock_resp.status_code = 200
|
||
mock_resp.text = DETAIL_HTML
|
||
|
||
mock_session = AsyncMock()
|
||
mock_session.get = AsyncMock(return_value=mock_resp)
|
||
mock_session.close = AsyncMock()
|
||
|
||
with _patch(
|
||
"app.services.scrapers.avito_detail.AsyncSession",
|
||
return_value=mock_session,
|
||
):
|
||
result = await fetch_detail("/ekaterinburg/kvartiry/test_99887766-1234")
|
||
|
||
# curl path взят — session.get был вызван
|
||
mock_session.get.assert_awaited_once()
|
||
assert result.item_id == "99887766"
|
||
|
||
|
||
# ── 5. SERP routing: real _fetch_serp_html routes through browser ────────────
|
||
|
||
|
||
@pytest.mark.asyncio
|
||
async def test_fetch_serp_html_routes_through_browser() -> None:
|
||
"""With _browser set and _cffi=None, the REAL _fetch_serp_html returns the
|
||
browser HTML and never dereferences _cffi (no AssertionError). Proves #915
|
||
SERP routing on top of #901's browser branch."""
|
||
from unittest.mock import AsyncMock
|
||
|
||
from app.services.scrapers.avito import AvitoScraper
|
||
|
||
scraper = AvitoScraper()
|
||
assert scraper._cffi is None
|
||
scraper._browser = AsyncMock()
|
||
scraper._browser.fetch = AsyncMock(return_value="<html><body>real serp listing</body></html>")
|
||
|
||
html = await scraper._fetch_serp_html(
|
||
"https://www.avito.ru/ekaterinburg/kvartiry/prodam", page=1
|
||
)
|
||
|
||
assert html == "<html><body>real serp listing</body></html>"
|
||
scraper._browser.fetch.assert_awaited_once()
|