gendesign/tradein-mvp/backend/tests
Light1YT 77c4af9751 fix(tradein): yandex_realty captures house number + parse_rub strips NBSP
Two critical bugs в yandex_realty scraper, обнаружены при investigating
issue #582 (217+ houses без номера дома в address):

1. **Address скрейпился только до street name** (yandex_realty.py:162-163).
   Прежний код брал text() только anchor tag `a[href*="/kupit/kvartira/st-..."]`
   что = только «улица Малышева». House number жил sibling text node parent'а
   `.AddressWithGeoLinks__addressContainer--<hash>`.

   Fix: читаем text() parent контейнера + normalise whitespace → получаем
   «улица Малышева, 125». Fallback на legacy street-only link если container
   отсутствует — не регрессим в NULL.

   Tests cover: letter suffix («2В»), slash («14/3»), district prefix
   («Берёзовский, …»), fallback.

2. **parse_rub() не стрипает NBSP** (yandex_helpers.py:289-294).
   Yandex SERP форматирует «9\xa0800\xa0000» через non-breaking space
   (U+00A0). Прежний `.replace(" ", "")` стрипал только regular space →
   `int("9\xa0800\xa0000")` raise ValueError → возвращали None → scraper
   skipping все listings c ценой.

   На production main парсилось **0/23 cards** из live HTML by-this-reason.
   Это объясняет почему `yandex_realty_nb` накопил мало houses (305) и
   почему `_card_to_lot` silently failed.

   Fix: `re.sub(r"\s", "", ...)` covers `\xa0`, тонкие пробелы, табы и т.д.
   То же для RE_RUB_MLN branch для consistency.

   Tests cover NBSP в obычной цене и в «4,4\xa0млн\xa0₽» форме.

После merge → deploy → следующий yandex sweep должен parse ~23/23 cards
с full addresses включая house number. Existing 217 houses с street-only
remain — пусть починятся через `backfill_house_coords.py` (precision
filter теперь сможет matchнуть exact).

Tests: 52/52 pass (yandex_realty_serp + yandex_helpers).
Ruff clean.
2026-05-27 12:23:11 +05:00
..
fixtures feat(tradein): Phase 1 of #582 — address mismatch audit infrastructure 2026-05-25 12:50:30 +05:00
matching feat(tradein): conflict_resolution — register Yandex sources in priority dicts (#471) 2026-05-23 14:03:35 +00:00
scrapers fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487) 2026-05-23 20:08:40 +00:00
services feat(tradein): persistent infra для backfill scripts (PR F) (#592) 2026-05-27 06:59:54 +00:00
tasks fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490) 2026-05-23 20:03:48 +00:00
__init__.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
test_api_avito_stage4a.py feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460) 2026-05-23 13:34:38 +00:00
test_audit_address_mismatch.py feat(tradein): Phase 1 of #582 — address mismatch audit infrastructure 2026-05-25 12:50:30 +05:00
test_avito_address_clean.py fix(tradein-avito): strip Emotion CSS from listings.address (#502) 2026-05-24 10:52:50 +00:00
test_avito_detail_parse.py feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443) 2026-05-23 12:17:15 +00:00
test_avito_houses_parse.py feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) (#449) 2026-05-23 12:37:24 +00:00
test_avito_imv_parse.py feat(tradein): avito_imv.py — Avito IMV evaluation API client (2 requests) (#444) 2026-05-23 12:16:47 +00:00
test_avito_relative_date.py feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508) 2026-05-24 11:17:13 +00:00
test_backfill_house_coords.py feat(tradein): Phase 2-3 of #582 — Yandex backfill + canonical audit (#591) 2026-05-27 06:30:54 +00:00
test_base_save_listings.py feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455) 2026-05-23 13:05:20 +00:00
test_cian_detail.py feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457) 2026-05-23 13:22:00 +00:00
test_cian_newbuilding.py feat(tradein): cian_newbuilding.py — ЖК catalog scraper (chart + reliability + offers, Wave 4 Worker C) (#459) 2026-05-23 13:23:11 +00:00
test_cian_serp_scraper.py feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450) 2026-05-23 12:52:04 +00:00
test_cian_session.py fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) 2026-05-23 14:52:24 +00:00
test_cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
test_cian_valuation.py feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465) 2026-05-23 13:43:34 +00:00
test_city_sweep.py feat(tradein): city sweep — auto ЕКБ pipeline (#477) 2026-05-23 14:38:07 +00:00
test_estimator_cian_integration.py fix(tradein): filter ДКП-only in rosreestr importer + re-enable deals (#549) 2026-05-24 19:42:28 +00:00
test_estimator_cohort.py fix(tradein-estimator): extend cohort filter to Tier S + Tier H, fix late_soviet overlap (#522) 2026-05-24 13:14:59 +00:00
test_estimator_floor_optional.py feat(tradein): этаж/этажность optional + best test presets по deal count (#558) 2026-05-24 21:44:44 +00:00
test_estimator_imv_integration.py feat(tradein): integrate Avito IMV as 5th evaluation source (on-demand cached) (#452) 2026-05-23 13:08:22 +00:00
test_estimator_source_quota.py feat(tradein): source-quota in estimator analogs (fix Cian/Yandex starvation) (#491) 2026-05-23 20:59:58 +00:00
test_estimator_yandex_integration.py feat(yandex-valuation): link history rows to houses via match_or_create_house (#531) 2026-05-24 14:28:30 +00:00
test_extract_short_addr.py feat(tradein-estimator): tiered house-match S→H→W (#507) 2026-05-24 12:41:12 +00:00
test_matching.py fix(tradein-matching): pg_advisory_xact_lock to prevent duplicate house INSERTs (#501) 2026-05-24 10:52:10 +00:00
test_rbac.py feat(rbac): pilot scope → только /trade-in/** (#587) 2026-05-26 07:40:44 +00:00
test_scheduler.py feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482) 2026-05-23 14:54:25 +00:00
test_scrape_pipeline.py feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) (#451) 2026-05-23 12:55:13 +00:00
test_scraper_settings.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
test_search_api.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
test_street_deals_endpoint.py fix(tradein): extract_street_name handles Nominatim reverse format (#557) 2026-05-24 21:03:42 +00:00
test_yandex_detail.py feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466) 2026-05-23 13:45:09 +00:00
test_yandex_helpers.py fix(tradein): yandex_realty captures house number + parse_rub strips NBSP 2026-05-27 12:23:11 +05:00
test_yandex_newbuilding.py feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) (#467) 2026-05-23 13:46:45 +00:00
test_yandex_realty_serp.py fix(tradein): yandex_realty captures house number + parse_rub strips NBSP 2026-05-27 12:23:11 +05:00
test_yandex_scrapers_delay_wiring.py feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484) 2026-05-23 15:28:34 +00:00
test_yandex_valuation.py fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552) 2026-05-24 19:57:17 +00:00
test_yandex_valuation_save.py feat(yandex-valuation): link history rows to houses via match_or_create_house (#531) 2026-05-24 14:28:30 +00:00