bot-backend
d6dca24ea2
Investigated with a real curl_cffi capture (HTTP 200, 46 cards): the Avito SERP lazy-renders — data-marker="item-date" appears only on the top above-the-fold cards (~10/46), so ~79% of cards have no date in the raw HTML curl_cffi receives. The marker wasn't renamed; the field is just absent. But every listing carries sortTimeStamp (epoch-ms) in the embedded per-item JSON (56 occurrences/page) — the line-266 'state empty' note was stale. Map item_id (= DOM data-item-id) -> sortTimeStamp -> date as the primary source, keep the DOM item-date + _parse_relative_date as fallback for the top cards. Verified on the real capture: 42/46 = 91% coverage (was ~21% DOM-only), clearing the >80% DoD bar. Adds a distilled real-structure fixture (avito_serp_sample.html) + unit test covering both the JSON (lazy card) and DOM (top card) paths. Refs #726
58 lines
2.5 KiB
Python
58 lines
2.5 KiB
Python
"""#726 — avito SERP listing_date extraction (sortTimeStamp JSON + DOM fallback).
|
||
|
||
Offline: парсит фикстуру через AvitoScraper._parse_html, без сети/curl_cffi/БД.
|
||
Регрессия на дрейф разметки: дата у lazy-карточек (без DOM item-date) берётся из
|
||
per-item JSON sortTimeStamp; у верхних карточек — из DOM data-marker="item-date".
|
||
"""
|
||
|
||
import os
|
||
from datetime import UTC, date, datetime
|
||
from pathlib import Path
|
||
|
||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
|
||
|
||
from app.services.scrapers.avito import (
|
||
AvitoScraper,
|
||
_build_sort_timestamp_map,
|
||
)
|
||
|
||
FIXTURE = Path(__file__).parent / "fixtures" / "avito_serp_sample.html"
|
||
|
||
# Из фикстуры (sortTimeStamp, epoch-ms).
|
||
_TS_LAZY = 1700000000000
|
||
_TS_TOP = 1700086400000
|
||
|
||
|
||
def _expected(ms: int) -> date:
|
||
return datetime.fromtimestamp(ms / 1000, tz=UTC).date()
|
||
|
||
|
||
def test_sort_timestamp_map_built_from_json() -> None:
|
||
html = FIXTURE.read_text(encoding="utf-8")
|
||
ts_map = _build_sort_timestamp_map(html)
|
||
assert ts_map["8043936560"] == _expected(_TS_LAZY)
|
||
assert ts_map["8163084615"] == _expected(_TS_TOP)
|
||
|
||
|
||
def test_parse_html_sets_listing_date_for_all_cards() -> None:
|
||
"""Обе карточки получают listing_date != None (DoD #726)."""
|
||
html = FIXTURE.read_text(encoding="utf-8")
|
||
lots = AvitoScraper()._parse_html(html, "https://www.avito.ru/ekaterinburg/kvartiry")
|
||
assert len(lots) == 2
|
||
by_url = {lot.source_url: lot for lot in lots}
|
||
assert all(lot.listing_date is not None for lot in lots), "every card must carry a date"
|
||
|
||
# Lazy card (нет DOM item-date) → дата из JSON sortTimeStamp.
|
||
lazy = next(lot for lot in lots if "8043936560" in (lot.source_url or ""))
|
||
assert lazy.listing_date == _expected(_TS_LAZY)
|
||
assert by_url # touch to keep linter calm if structure changes
|
||
|
||
|
||
def test_lazy_card_date_comes_from_json_not_dom() -> None:
|
||
"""Карточка без data-marker='item-date' всё равно датируется (JSON primary)."""
|
||
html = FIXTURE.read_text(encoding="utf-8")
|
||
assert 'data-item-id="8043936560"' in html
|
||
# у этой карточки в фикстуре НЕТ DOM item-date — проверяем, что дата не потерялась
|
||
lots = AvitoScraper()._parse_html(html, "https://www.avito.ru/x")
|
||
lazy = next(lot for lot in lots if "8043936560" in (lot.source_url or ""))
|
||
assert lazy.listing_date == _expected(_TS_LAZY)
|