bf4ce1ba36
All checks were successful
CI Trade-In / changes (pull_request) Successful in 12s
CI / changes (pull_request) Successful in 13s
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m3s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
#2487 avito oblast per-city sweep был dead-on-arrival: _parse_html дропал все карточки без /ekaterinburg/ в source_url, поэтому oblast-sweep по городу вне ЕКБ терял 100% выдачи. Kept-slug теперь параметризуется через AvitoScraper.target_city_slug (дефолт "ekaterinburg" — ЕКБ-поведение неизменно), прокинут scheduler → run_avito_city_sweep → AvitoScraper. avito_serp_ekb_only НЕ отключается. Bug #2 (oblast bare-street mis-bucket): голые адреса 'ул. Ленина 100' в Н.Тагиле мис-баккетились в одноимённые ЕКБ-дома через Tier-2b (match по глобально-уникальному normalized_address без geo/city guard). Добавлен coarse-guard: coords present → ST_DWithin ≤3км до geom дома; coords absent → требуется city-token (self- disambiguating); иначе skip → консервативно New house вместо мис-матча. _insert_alias при коллизии normalized_address больше не перебивает fingerprint чужого дома (иначе guard деградировал бы в Tier-2a-дыру). Оба дефекта латентные (oblast per-city schedules отключены) — правка делает capability безопасной к включению, без влияния на прод сегодня.
84 lines
3.6 KiB
Python
84 lines
3.6 KiB
Python
"""Avito SERP city-slug filter (#2487 oblast rollout).
|
||
|
||
`_parse_html` drops padding cards ("по всей России") when `avito_serp_ekb_only`
|
||
is on. The kept city-slug used to be hardcoded `/ekaterinburg/`, so an oblast
|
||
per-city sweep (`_job_avito_city_sweep` with a `city` param) discarded 100% of the
|
||
target city's cards. The slug is now parameterized via `AvitoScraper.target_city_slug`
|
||
(default → "ekaterinburg", i.e. ЕКБ behavior unchanged when no target is set).
|
||
|
||
No network / no DB — `_parse_html` runs on inline HTML with a stub config.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import os
|
||
from types import SimpleNamespace
|
||
|
||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")
|
||
|
||
from scraper_kit.providers.avito.serp import AvitoScraper
|
||
|
||
_BASE_URL = "https://www.avito.ru/ekaterinburg/kvartiry/prodam-ASgBAgICAUSSA8YQ?p=1"
|
||
|
||
|
||
def _card(slug: str, item_id: str, price: int = 5_000_000) -> str:
|
||
"""Minimal Avito SERP card whose per-card href carries `slug` as the city segment."""
|
||
href = f"/{slug}/kvartiry/2k_kvartira_50_m_5_5et_{item_id}"
|
||
return (
|
||
f'<div data-marker="item" data-item-id="{item_id}">'
|
||
f'<a data-marker="item-title" href="{href}">2-к. квартира, 50 м², 5/9 эт.</a>'
|
||
f'<meta itemprop="price" content="{price}">'
|
||
f"</div>"
|
||
)
|
||
|
||
|
||
def _html(*cards: str) -> str:
|
||
return "<html><body>" + "".join(cards) + "</body></html>"
|
||
|
||
|
||
def _ekb_only_cfg() -> SimpleNamespace:
|
||
# _parse_html only reads config.avito_serp_ekb_only; a stub keeps the test
|
||
# deterministic regardless of AVITO_SERP_EKB_ONLY env in the runner.
|
||
return SimpleNamespace(avito_serp_ekb_only=True)
|
||
|
||
|
||
def test_parse_keeps_target_oblast_city_drops_off_target_padding() -> None:
|
||
"""Sweep targeting Н.Тагил keeps its cards; ЕКБ/Москва padding is dropped."""
|
||
s = AvitoScraper(_ekb_only_cfg(), target_city_slug="nizhniy_tagil") # type: ignore[arg-type]
|
||
html = _html(
|
||
_card("nizhniy_tagil", "nt1"), # target city → KEPT
|
||
_card("ekaterinburg", "ekb1"), # off-target padding → DROPPED
|
||
_card("moskva", "msk1"), # off-target padding → DROPPED
|
||
)
|
||
lots = s._parse_html(html, source_url_base=_BASE_URL)
|
||
urls = [lot.source_url or "" for lot in lots]
|
||
|
||
assert len(lots) == 1, urls
|
||
assert "/nizhniy_tagil/" in urls[0]
|
||
assert all("/ekaterinburg/" not in u for u in urls)
|
||
assert all("/moskva/" not in u for u in urls)
|
||
|
||
|
||
def test_parse_default_target_is_ekb_and_drops_oblast_padding() -> None:
|
||
"""No target (ЕКБ sweep/full-load): kept slug defaults to 'ekaterinburg' — behavior
|
||
identical to the previous hardcode. ЕКБ card kept, oblast padding dropped."""
|
||
s = AvitoScraper(_ekb_only_cfg(), target_city_slug=None) # type: ignore[arg-type]
|
||
html = _html(
|
||
_card("ekaterinburg", "e1"), # ЕКБ → KEPT
|
||
_card("nizhniy_tagil", "nt1"), # off-target padding → DROPPED
|
||
)
|
||
lots = s._parse_html(html, source_url_base=_BASE_URL)
|
||
urls = [lot.source_url or "" for lot in lots]
|
||
|
||
assert len(lots) == 1, urls
|
||
assert "/ekaterinburg/" in urls[0]
|
||
assert all("/nizhniy_tagil/" not in u for u in urls)
|
||
|
||
|
||
def test_parse_ekb_only_disabled_keeps_all_cities() -> None:
|
||
"""avito_serp_ekb_only=False → no city filter at all, every card is kept."""
|
||
cfg = SimpleNamespace(avito_serp_ekb_only=False)
|
||
s = AvitoScraper(cfg, target_city_slug="nizhniy_tagil") # type: ignore[arg-type]
|
||
html = _html(_card("nizhniy_tagil", "nt1"), _card("ekaterinburg", "e1"))
|
||
lots = s._parse_html(html, source_url_base=_BASE_URL)
|
||
assert len(lots) == 2
|