fix(tradein/oblast): city-slug в avito SERP + geo/city guard в Tier-2b матчинга
All checks were successful
CI Trade-In / changes (pull_request) Successful in 12s
CI / changes (pull_request) Successful in 13s
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m3s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped

#2487 avito oblast per-city sweep был dead-on-arrival: _parse_html дропал все
карточки без /ekaterinburg/ в source_url, поэтому oblast-sweep по городу вне ЕКБ
терял 100% выдачи. Kept-slug теперь параметризуется через AvitoScraper.target_city_slug
(дефолт "ekaterinburg" — ЕКБ-поведение неизменно), прокинут scheduler →
run_avito_city_sweep → AvitoScraper. avito_serp_ekb_only НЕ отключается.

Bug #2 (oblast bare-street mis-bucket): голые адреса 'ул. Ленина 100' в Н.Тагиле
мис-баккетились в одноимённые ЕКБ-дома через Tier-2b (match по глобально-уникальному
normalized_address без geo/city guard). Добавлен coarse-guard: coords present →
ST_DWithin ≤3км до geom дома; coords absent → требуется city-token (self-
disambiguating); иначе skip → консервативно New house вместо мис-матча. _insert_alias
при коллизии normalized_address больше не перебивает fingerprint чужого дома (иначе
guard деградировал бы в Tier-2a-дыру).

Оба дефекта латентные (oblast per-city schedules отключены) — правка делает
capability безопасной к включению, без влияния на прод сегодня.
This commit is contained in:
bot-backend 2026-07-12 22:38:11 +03:00
parent 0f4ed2f6e9
commit bf4ce1ba36
7 changed files with 366 additions and 20 deletions

View file

@ -17,6 +17,7 @@ from sqlalchemy.orm import Session
from app.services.matching.normalize import ( from app.services.matching.normalize import (
address_fingerprint, address_fingerprint,
has_city_token,
has_house_number, has_house_number,
house_number_token, house_number_token,
normalize_address, normalize_address,
@ -24,6 +25,14 @@ from app.services.matching.normalize import (
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Tier-2b oblast guard radius (metres). A bare street+number normalized_address carries
# NO city token yet is globally unique in house_address_aliases, so 'улица ленина 100'
# physically exists in ЕКБ, Н.Тагил, Каменск… When the incoming card HAS coords we accept
# the alias only if its house geom is within this radius. Cities in обл.66 are ≥ ~9-15 km
# apart, so 3 km cleanly separates them while absorbing the intra-city geocoder drift that
# Tier-2b exists to bridge (same building, different provider coords).
_TIER2B_GUARD_M = 3000
def match_or_create_house( def match_or_create_house(
db: Session, db: Session,
@ -193,17 +202,52 @@ def match_or_create_house(
# Gated on has_num (P1): never match a bare-street normalized_address — any # Gated on has_num (P1): never match a bare-street normalized_address — any
# numberless listing would otherwise collapse into whichever house first # numberless listing would otherwise collapse into whichever house first
# registered that street. # registered that street.
row = ( #
db.execute( # Oblast guard (bug #2, обл.66 per-city rollout): normalized_address has NO city
text( # token, so a bare street+number is globally unique in the alias table yet exists
"SELECT house_id FROM house_address_aliases " # physically in several cities. Accept the match only when the incoming listing is
"WHERE normalized_address = :na LIMIT 1" # confirmably the same place:
), # • coords present → aliased house geom within _TIER2B_GUARD_M metres;
{"na": norm_addr}, # • coords absent → norm_addr carries a city token (self-disambiguating —
# 'екатеринбург улица ленина 5' can't collide with a Н.Тагил address);
# • neither → skip. A bare common-street with no geo signal is too
# ambiguous — matching would mis-bucket an oblast card into a same-named ЕКБ
# house. Conservative: fall through to a New house rather than mis-match.
if lat is not None and lon is not None:
row = (
db.execute(
text(
"SELECT a.house_id FROM house_address_aliases a "
"JOIN houses h ON h.id = a.house_id "
"WHERE a.normalized_address = :na "
" AND h.geom IS NOT NULL "
" AND ST_DWithin(h.geom::geography, "
" ST_MakePoint(:lon, :lat)::geography, :thr) "
"LIMIT 1"
),
{"na": norm_addr, "lon": lon, "lat": lat, "thr": _TIER2B_GUARD_M},
)
.mappings()
.first()
)
elif has_city_token(norm_addr):
row = (
db.execute(
text(
"SELECT house_id FROM house_address_aliases "
"WHERE normalized_address = :na LIMIT 1"
),
{"na": norm_addr},
)
.mappings()
.first()
)
else:
logger.info(
"house tier2b skip: bare common-street, no coords/city na=%r src=%s",
norm_addr,
ext_source,
) )
.mappings()
.first()
)
if row: if row:
house_id = int(row["house_id"]) house_id = int(row["house_id"])
_upsert_house_source( _upsert_house_source(
@ -520,6 +564,12 @@ def _insert_alias(
row with the latest fingerprint, which is then found by Tier 2a on the next scrape. row with the latest fingerprint, which is then found by Tier 2a on the next scrape.
house_id is not updated on conflict: the first writer wins canonical ownership. house_id is not updated on conflict: the first writer wins canonical ownership.
Oblast guard (bug #2): the fingerprint/source are refreshed ONLY when the conflicting
row belongs to the SAME house. A DIFFERENT house colliding on a bare common-street
normalized_address (an oblast card that fell through Tier-2b's guard to a New INSERT)
must NOT rewrite the owner's fingerprint — doing so would downgrade a safe coord-bearing
alias to a coord-less one and let the next no-coord card mis-bucket via Tier 2a.
P1: a bare-street normalized_address (no house number) is NOT registered as an P1: a bare-street normalized_address (no house number) is NOT registered as an
alias it is too ambiguous to serve as a building key. Any later numberless alias it is too ambiguous to serve as a building key. Any later numberless
listing would otherwise hit that alias via Tier 2b and be mass-dumped into the listing would otherwise hit that alias via Tier 2b and be mass-dumped into the
@ -533,8 +583,12 @@ def _insert_alias(
INSERT INTO house_address_aliases (house_id, normalized_address, fingerprint, source) INSERT INTO house_address_aliases (house_id, normalized_address, fingerprint, source)
VALUES (CAST(:hid AS bigint), :na, :fp, :src) VALUES (CAST(:hid AS bigint), :na, :fp, :src)
ON CONFLICT (normalized_address) DO UPDATE SET ON CONFLICT (normalized_address) DO UPDATE SET
fingerprint = EXCLUDED.fingerprint, fingerprint = CASE
source = EXCLUDED.source WHEN house_address_aliases.house_id = EXCLUDED.house_id
THEN EXCLUDED.fingerprint ELSE house_address_aliases.fingerprint END,
source = CASE
WHEN house_address_aliases.house_id = EXCLUDED.house_id
THEN EXCLUDED.source ELSE house_address_aliases.source END
"""), """),
{ {
"hid": house_id, "hid": house_id,

View file

@ -112,6 +112,40 @@ def house_number_token(normalized: str | None) -> str | None:
return m.group(1) if m else None return m.group(1) if m else None
# Cities covered by the обл.66 sweep rollout (ЕКБ + oblast per-city schedules).
# Normalized form: lowercase, hyphens collapsed to spaces (mirrors normalize_address,
# e.g. 'каменск-уральский' -> 'каменск уральский'). Mirrors CITY_ANCHORS in
# scraper_kit.orchestration.pipeline — extend together as the sweep adds cities.
_CITY_TOKENS: tuple[str, ...] = (
"екатеринбург",
"нижний тагил",
"каменск уральский",
"первоуральск",
"верхняя пышма",
"серов",
)
# Token-bounded (not substring) so a street named after a city does NOT false-positive:
# 'улица серова 5' has token 'серова' (trailing 'а'), never bare 'серов'.
_CITY_TOKEN_RE = re.compile(
r"(?:^|\s)(?:" + "|".join(re.escape(c) for c in _CITY_TOKENS) + r")(?:\s|$)"
)
def has_city_token(normalized: str | None) -> bool:
"""True if the normalized address carries a known обл.66 city token.
A city token makes the normalized_address city-scoped two addresses in different
cities can then never collapse to the same key, so the alias is a safe cross-source
building key. Used by the Tier-2b oblast guard in match_or_create_house: a bare
common-street address with NO city token AND no coords is too ambiguous to match a
globally-unique normalized_address alias (it would mis-bucket an oblast card into a
same-named ЕКБ house bug #2 oblast rollout).
"""
if not normalized:
return False
return _CITY_TOKEN_RE.search(normalized) is not None
def address_fingerprint(address: str | None, lat: float | None, lon: float | None) -> str: def address_fingerprint(address: str | None, lat: float | None, lon: float | None) -> str:
"""SHA-256 fingerprint of normalized address + rounded coordinates (4 dp ≈ 11 m). """SHA-256 fingerprint of normalized address + rounded coordinates (4 dp ≈ 11 m).

View file

@ -0,0 +1,84 @@
"""Avito SERP city-slug filter (#2487 oblast rollout).
`_parse_html` drops padding cards ("по всей России") when `avito_serp_ekb_only`
is on. The kept city-slug used to be hardcoded `/ekaterinburg/`, so an oblast
per-city sweep (`_job_avito_city_sweep` with a `city` param) discarded 100% of the
target city's cards. The slug is now parameterized via `AvitoScraper.target_city_slug`
(default "ekaterinburg", i.e. ЕКБ behavior unchanged when no target is set).
No network / no DB `_parse_html` runs on inline HTML with a stub config.
"""
from __future__ import annotations
import os
from types import SimpleNamespace
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")
from scraper_kit.providers.avito.serp import AvitoScraper
_BASE_URL = "https://www.avito.ru/ekaterinburg/kvartiry/prodam-ASgBAgICAUSSA8YQ?p=1"
def _card(slug: str, item_id: str, price: int = 5_000_000) -> str:
"""Minimal Avito SERP card whose per-card href carries `slug` as the city segment."""
href = f"/{slug}/kvartiry/2k_kvartira_50_m_5_5et_{item_id}"
return (
f'<div data-marker="item" data-item-id="{item_id}">'
f'<a data-marker="item-title" href="{href}">2-к. квартира, 50 м², 5/9 эт.</a>'
f'<meta itemprop="price" content="{price}">'
f"</div>"
)
def _html(*cards: str) -> str:
return "<html><body>" + "".join(cards) + "</body></html>"
def _ekb_only_cfg() -> SimpleNamespace:
# _parse_html only reads config.avito_serp_ekb_only; a stub keeps the test
# deterministic regardless of AVITO_SERP_EKB_ONLY env in the runner.
return SimpleNamespace(avito_serp_ekb_only=True)
def test_parse_keeps_target_oblast_city_drops_off_target_padding() -> None:
"""Sweep targeting Н.Тагил keeps its cards; ЕКБ/Москва padding is dropped."""
s = AvitoScraper(_ekb_only_cfg(), target_city_slug="nizhniy_tagil") # type: ignore[arg-type]
html = _html(
_card("nizhniy_tagil", "nt1"), # target city → KEPT
_card("ekaterinburg", "ekb1"), # off-target padding → DROPPED
_card("moskva", "msk1"), # off-target padding → DROPPED
)
lots = s._parse_html(html, source_url_base=_BASE_URL)
urls = [lot.source_url or "" for lot in lots]
assert len(lots) == 1, urls
assert "/nizhniy_tagil/" in urls[0]
assert all("/ekaterinburg/" not in u for u in urls)
assert all("/moskva/" not in u for u in urls)
def test_parse_default_target_is_ekb_and_drops_oblast_padding() -> None:
"""No target (ЕКБ sweep/full-load): kept slug defaults to 'ekaterinburg' — behavior
identical to the previous hardcode. ЕКБ card kept, oblast padding dropped."""
s = AvitoScraper(_ekb_only_cfg(), target_city_slug=None) # type: ignore[arg-type]
html = _html(
_card("ekaterinburg", "e1"), # ЕКБ → KEPT
_card("nizhniy_tagil", "nt1"), # off-target padding → DROPPED
)
lots = s._parse_html(html, source_url_base=_BASE_URL)
urls = [lot.source_url or "" for lot in lots]
assert len(lots) == 1, urls
assert "/ekaterinburg/" in urls[0]
assert all("/nizhniy_tagil/" not in u for u in urls)
def test_parse_ekb_only_disabled_keeps_all_cities() -> None:
"""avito_serp_ekb_only=False → no city filter at all, every card is kept."""
cfg = SimpleNamespace(avito_serp_ekb_only=False)
s = AvitoScraper(cfg, target_city_slug="nizhniy_tagil") # type: ignore[arg-type]
html = _html(_card("nizhniy_tagil", "nt1"), _card("ekaterinburg", "e1"))
lots = s._parse_html(html, source_url_base=_BASE_URL)
assert len(lots) == 2

View file

@ -17,6 +17,7 @@ from app.services.matching.conflict_resolution import (
) )
from app.services.matching.normalize import ( from app.services.matching.normalize import (
address_fingerprint, address_fingerprint,
has_city_token,
has_house_number, has_house_number,
house_number_token, house_number_token,
normalize_address, normalize_address,
@ -168,6 +169,37 @@ def test_fingerprint_all_none():
assert len(fp) == 32 assert len(fp) == 32
# ---------------------------------------------------------------------------
# has_city_token — обл.66 city-token detection (Tier-2b oblast guard)
# ---------------------------------------------------------------------------
def test_has_city_token_detects_ekb():
assert has_city_token("екатеринбург улица ленина 5") is True
def test_has_city_token_detects_multiword_oblast_city():
assert has_city_token("нижний тагил улица ленина 100") is True
assert has_city_token("каменск уральский улица мира 7") is True
def test_has_city_token_bare_street_is_false():
# The collision-prone case: a bare street+number carries no city token.
assert has_city_token("улица ленина 100") is False
assert has_city_token("уральская улица 62к1") is False
def test_has_city_token_no_false_positive_on_street_named_after_city():
# 'улица серова' must NOT match the city token 'серов' (word-bounded, not substring).
assert has_city_token("улица серова 5") is False
assert has_city_token("первоуральская улица 3") is False
def test_has_city_token_handles_empty():
assert has_city_token("") is False
assert has_city_token(None) is False
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# match_or_create_house — mock DB tier routing # match_or_create_house — mock DB tier routing
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -622,6 +654,125 @@ def test_insert_alias_executes_when_house_number_present():
db.execute.assert_called_once() db.execute.assert_called_once()
def test_insert_alias_conflict_only_rebinds_same_house():
"""Oblast guard (#2): ON CONFLICT refreshes fingerprint/source ONLY for the same
house. A different house colliding on a bare common-street normalized_address must
not rewrite the owner's fingerprint (that would downgrade a coord-bearing alias to a
coord-less one and let the next no-coord card mis-bucket via Tier 2a)."""
from app.services.matching.houses import _insert_alias
db = MagicMock()
_insert_alias(db, house_id=1, address="улица Ленина 100", fp="c" * 32, source="avito")
sql = " ".join(str(db.execute.call_args.args[0]).split())
assert "ON CONFLICT (normalized_address) DO UPDATE" in sql
# fingerprint/source updated only when the existing owner == the incoming house.
assert "house_address_aliases.house_id = EXCLUDED.house_id" in sql
# ---------------------------------------------------------------------------
# match_or_create_house — Tier-2b oblast geo/city guard (bug #2)
# ---------------------------------------------------------------------------
def test_tier2b_coord_present_uses_geo_guard_and_matches_within_radius():
"""Coords present → Tier-2b runs the ST_DWithin JOIN guard; a house within the
radius is accepted as a fingerprint match (0.9)."""
from app.services.matching.houses import match_or_create_house
db = _make_db(
[
None, # pg_advisory_xact_lock
None, # house_sources miss (Tier 1)
None, # fingerprint miss (Tier 2a)
{"house_id": 77}, # Tier 2b JOIN+ST_DWithin hit (same place, within radius)
None, # _upsert_house_source
None, # _insert_alias
]
)
house_id, conf, method = match_or_create_house(
db, "avito", "ext-2b-geo", address="улица Ленина 100", lat=57.910, lon=59.980
)
assert (house_id, conf, method) == (77, 0.9, "fingerprint")
# The Tier-2b lookup must be the geo-guarded JOIN, not a bare normalized_address SELECT.
tier2b_sql = _executed_sqls(db)[3]
assert "ST_DWithin" in tier2b_sql
assert "JOIN houses" in tier2b_sql
def test_tier2b_coord_present_cross_city_falls_through_to_new():
"""A bare street+number oblast card WITH coords whose only same-street alias is a
far-away (cross-city) house: the ST_DWithin guard filters it out (JOIN miss) the
card falls through to a New house instead of mis-bucketing into the ЕКБ house."""
from app.services.matching.houses import match_or_create_house
db = _make_db(
[
None, # pg_advisory_xact_lock
None, # house_sources miss
None, # fingerprint miss (Tier 2a)
None, # Tier 2b JOIN+ST_DWithin miss (candidate house is >3km away)
None, # geo miss (Tier 3 — coords present)
{"id": 909}, # INSERT RETURNING id (New house)
None, # _upsert_house_source
None, # _insert_alias
]
)
house_id, conf, method = match_or_create_house(
db, "avito", "ext-2b-xcity", address="улица Ленина 100", lat=57.910, lon=59.980
)
assert (house_id, conf, method) == (909, 1.0, "new")
def test_tier2b_no_coords_no_city_token_skips_and_creates_new():
"""No coords AND a bare common-street (no city token) → Tier-2b is SKIPPED entirely
(no normalized_address lookup). The card falls through to a New house conservative:
do not mis-bucket a bare 'улица ленина 100' into a same-named ЕКБ alias."""
from app.services.matching.houses import match_or_create_house
db = _make_db(
[
None, # pg_advisory_xact_lock
None, # house_sources miss
None, # fingerprint miss (Tier 2a); Tier 2b SKIPPED (no coords, no city token)
{"id": 910}, # INSERT RETURNING id (New house) — no Tier 3 (no coords)
None, # _upsert_house_source
None, # _insert_alias
]
)
house_id, conf, method = match_or_create_house(
db, "avito", "ext-2b-bare", address="улица Ленина 100"
)
assert (house_id, conf, method) == (910, 1.0, "new")
assert not any(
"normalized_address = :na" in s for s in _executed_sqls(db)
), "bare common-street with no coords/city must not run a Tier-2b lookup"
def test_tier2b_no_coords_with_city_token_matches():
"""No coords but the normalized_address carries a city token → self-disambiguating,
so Tier-2b matches on normalized_address alone (0.9). ЕКБ cross-source dedup for
city-qualified addresses is preserved."""
from app.services.matching.houses import match_or_create_house
db = _make_db(
[
None, # pg_advisory_xact_lock
None, # house_sources miss
None, # fingerprint miss (Tier 2a)
{"house_id": 33}, # Tier 2b normalized_address hit (city token present)
None, # _upsert_house_source
None, # _insert_alias
]
)
house_id, conf, method = match_or_create_house(
db, "avito", "ext-2b-city", address="Екатеринбург, улица Ленина, 100"
)
assert (house_id, conf, method) == (33, 0.9, "fingerprint")
tier2b_sql = _executed_sqls(db)[3]
assert "normalized_address = :na" in tier2b_sql
assert "ST_DWithin" not in tier2b_sql
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# match_or_create_listing — mock DB tier routing # match_or_create_listing — mock DB tier routing
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------

View file

@ -829,6 +829,7 @@ async def run_avito_city_sweep(
shutdown_requested: Callable[[], bool] = lambda: False, shutdown_requested: Callable[[], bool] = lambda: False,
proxy_provider: ProxyProvider | None = None, proxy_provider: ProxyProvider | None = None,
anchors: list[tuple[float, float, str]] | None = None, anchors: list[tuple[float, float, str]] | None = None,
city_slug: str | None = None,
radius_m: int = 1500, radius_m: int = 1500,
pages_per_anchor: int = 3, pages_per_anchor: int = 3,
enrich_houses: bool = True, enrich_houses: bool = True,
@ -853,6 +854,9 @@ async def run_avito_city_sweep(
вместо прямых импортов app.* (см. scraper_kit.contracts). process_houses_imv_batch вместо прямых импортов app.* (см. scraper_kit.contracts). process_houses_imv_batch
зовётся через enrichment (EnrichmentJobs Protocol). зовётся через enrichment (EnrichmentJobs Protocol).
""" """
# city_slug (#2487): слаг города-цели (напр. "nizhniy_tagil") прокидывается в
# AvitoScraper.target_city_slug → SERP-фильтр оставляет карточки этого города.
# None → ЕКБ-дефолт (совпадает с anchors=EKB_ANCHORS fallback ниже).
_anchors = anchors if anchors is not None else EKB_ANCHORS _anchors = anchors if anchors is not None else EKB_ANCHORS
counters = CitySweepCounters(anchors_total=len(_anchors)) counters = CitySweepCounters(anchors_total=len(_anchors))
all_touched_house_ids: set[int] = set() all_touched_house_ids: set[int] = set()
@ -959,7 +963,9 @@ async def run_avito_city_sweep(
nonlocal all_touched_house_ids, sweep_rotations_done nonlocal all_touched_house_ids, sweep_rotations_done
# ── Phase 1+2: SERP + save ───────────────────────────────── # ── Phase 1+2: SERP + save ─────────────────────────────────
scraper = AvitoScraper(config) # #2487: target_city_slug → _parse_html оставляет карточки
# города-цели (oblast slug), а не хардкод /ekaterinburg/. None → ЕКБ.
scraper = AvitoScraper(config, target_city_slug=city_slug)
if browser_mode: if browser_mode:
scraper._browser = shared_bf scraper._browser = shared_bf
# Shared-browser режим: _cffi=None → curl_cffi-fallback на # Shared-browser режим: _cffi=None → curl_cffi-fallback на

View file

@ -455,6 +455,9 @@ async def _job_avito_city_sweep(
shutdown_requested=ctx.shutdown_requested, shutdown_requested=ctx.shutdown_requested,
proxy_provider=ctx.proxy_provider, proxy_provider=ctx.proxy_provider,
anchors=anchors, anchors=anchors,
# #2487: слаг города-цели → SERP-фильтр _parse_html оставляет карточки
# этого города (иначе oblast-sweep дропал 100% из-за хардкода /ekaterinburg/).
city_slug=city,
pages_per_anchor=int(params.get("pages_per_anchor", 3)), pages_per_anchor=int(params.get("pages_per_anchor", 3)),
detail_top_n=int(params.get("detail_top_n", 20)), detail_top_n=int(params.get("detail_top_n", 20)),
request_delay_sec=float(params.get("request_delay_sec", 7.0)), request_delay_sec=float(params.get("request_delay_sec", 7.0)),

View file

@ -350,6 +350,7 @@ class AvitoScraper(BaseScraper):
config: ScraperConfig, config: ScraperConfig,
*, *,
delay_provider: Callable[[str], float] | None = None, delay_provider: Callable[[str], float] | None = None,
target_city_slug: str | None = None,
) -> None: ) -> None:
super().__init__() super().__init__()
# Strangler-инжекция (#2133): конфиг и провайдер задержки приходят снаружи # Strangler-инжекция (#2133): конфиг и провайдер задержки приходят снаружи
@ -363,6 +364,10 @@ class AvitoScraper(BaseScraper):
self._browser: BrowserFetcher | None = None self._browser: BrowserFetcher | None = None
# #823: счётчик карточек, которые не удалось распарсить из-за неожиданной структуры DOM. # #823: счётчик карточек, которые не удалось распарсить из-за неожиданной структуры DOM.
self.parse_failures: int = 0 self.parse_failures: int = 0
# #2487 oblast rollout: slug города-цели sweep'а (напр. "nizhniy_tagil").
# None → ЕКБ (дефолт). Используется _parse_html-фильтром avito_serp_ekb_only,
# чтобы оставлять карточки TARGET-города, а не хардкодить /ekaterinburg/.
self._target_city_slug = target_city_slug
async def __aenter__(self) -> AvitoScraper: async def __aenter__(self) -> AvitoScraper:
# Проактивная ротация IP в начале sweep (#1731): apw-IP «протухает» по # Проактивная ротация IP в начале sweep (#1731): apw-IP «протухает» по
@ -1714,8 +1719,11 @@ class AvitoScraper(BaseScraper):
лениво только у верхних карточек (#726); DOM-marker остаётся fallback. лениво только у верхних карточек (#726); DOM-marker остаётся fallback.
Если self._config.avito_serp_ekb_only=True (дефолт), отбрасываем карточки, Если self._config.avito_serp_ekb_only=True (дефолт), отбрасываем карточки,
чей per-card source_url не содержит /ekaterinburg/ Avito добивает выдачу чей per-card source_url не содержит слаг ГОРОДА-ЦЕЛИ sweep'а — Avito добивает
«по всей России» при малом числе ЕКБ-результатов (4+ комн., дорогие сегменты). выдачу «по всей России» при малом числе результатов (4+ комн., дорогие сегменты).
Город-цель = self._target_city_slug (напр. "nizhniy_tagil"), по умолчанию
"ekaterinburg". Без параметризации (#2487) oblast-sweep по городу вне ЕКБ терял
100% карточек: их per-card slug (/nizhniy_tagil/) не содержит /ekaterinburg/.
Карточки с нераспознанным city-slug (href без ведущего /city/) пропускаются Карточки с нераспознанным city-slug (href без ведущего /city/) пропускаются
консервативно (не дропаются по ошибке). консервативно (не дропаются по ошибке).
""" """
@ -1724,6 +1732,10 @@ class AvitoScraper(BaseScraper):
cards = tree.css('[data-marker="item"]') cards = tree.css('[data-marker="item"]')
lots: list[ScrapedLot] = [] lots: list[ScrapedLot] = []
ekb_only = self._config.avito_serp_ekb_only ekb_only = self._config.avito_serp_ekb_only
# #2487: слаг города, карточки которого оставляем. Дефолт "ekaterinburg" —
# ЕКБ-поведение неизменно, когда target не задан (обычный ЕКБ-sweep/full-load).
kept_slug = self._target_city_slug or "ekaterinburg"
kept_seg = f"/{kept_slug}/"
dropped = 0 dropped = 0
no_city = 0 no_city = 0
for card in cards: for card in cards:
@ -1736,11 +1748,11 @@ class AvitoScraper(BaseScraper):
# urljoin всегда возвращает абсолютный URL (https://www.avito.ru/...). # urljoin всегда возвращает абсолютный URL (https://www.avito.ru/...).
# Avito city-slug — первый path-сегмент после netloc. # Avito city-slug — первый path-сегмент после netloc.
# Пример: https://www.avito.ru/ekaterinburg/... → /ekaterinburg/ присутствует. # Пример: https://www.avito.ru/ekaterinburg/... → /ekaterinburg/ присутствует.
# https://www.avito.ru/moskva/... → нет /ekaterinburg/. # https://www.avito.ru/moskva/... → нет слага города-цели.
# Для href без явного city (/kvartiry/... — теоретически не должно быть, # Для href без явного city (/kvartiry/... — теоретически не должно быть,
# но консервативно пропускаем). # но консервативно пропускаем).
if "/ekaterinburg/" in card_url: if kept_seg in card_url:
# ЕКБ-карточка — берём # Карточка города-цели — берём
pass pass
else: else:
# Проверяем, есть ли вообще city-slug (хотя бы один path-сегмент # Проверяем, есть ли вообще city-slug (хотя бы один path-сегмент
@ -1758,13 +1770,15 @@ class AvitoScraper(BaseScraper):
# Нет city-slug → консервативно берём карточку # Нет city-slug → консервативно берём карточку
lots.append(lot) lots.append(lot)
continue continue
# Есть city-slug, но не /ekaterinburg/ → padding по России → дропаем # Есть city-slug, но не города-цели → padding по России → дропаем
dropped += 1 dropped += 1
continue continue
lots.append(lot) lots.append(lot)
if ekb_only and (dropped or no_city): if ekb_only and (dropped or no_city):
logger.info( logger.info(
"avito SERP: dropped %d non-EKB cards (padding), %d cards without city-slug kept", "avito SERP (kept=%s): dropped %d off-target cards (padding), "
"%d cards without city-slug kept",
kept_slug,
dropped, dropped,
no_city, no_city,
) )