Commit graph

5 commits

Author SHA1 Message Date
77c4af9751 fix(tradein): yandex_realty captures house number + parse_rub strips NBSP
Two critical bugs в yandex_realty scraper, обнаружены при investigating
issue #582 (217+ houses без номера дома в address):

1. **Address скрейпился только до street name** (yandex_realty.py:162-163).
   Прежний код брал text() только anchor tag `a[href*="/kupit/kvartira/st-..."]`
   что = только «улица Малышева». House number жил sibling text node parent'а
   `.AddressWithGeoLinks__addressContainer--<hash>`.

   Fix: читаем text() parent контейнера + normalise whitespace → получаем
   «улица Малышева, 125». Fallback на legacy street-only link если container
   отсутствует — не регрессим в NULL.

   Tests cover: letter suffix («2В»), slash («14/3»), district prefix
   («Берёзовский, …»), fallback.

2. **parse_rub() не стрипает NBSP** (yandex_helpers.py:289-294).
   Yandex SERP форматирует «9\xa0800\xa0000» через non-breaking space
   (U+00A0). Прежний `.replace(" ", "")` стрипал только regular space →
   `int("9\xa0800\xa0000")` raise ValueError → возвращали None → scraper
   skipping все listings c ценой.

   На production main парсилось **0/23 cards** из live HTML by-this-reason.
   Это объясняет почему `yandex_realty_nb` накопил мало houses (305) и
   почему `_card_to_lot` silently failed.

   Fix: `re.sub(r"\s", "", ...)` covers `\xa0`, тонкие пробелы, табы и т.д.
   То же для RE_RUB_MLN branch для consistency.

   Tests cover NBSP в obычной цене и в «4,4\xa0млн\xa0₽» форме.

После merge → deploy → следующий yandex sweep должен parse ~23/23 cards
с full addresses включая house number. Existing 217 houses с street-only
remain — пусть починятся через `backfill_house_coords.py` (precision
filter теперь сможет matchнуть exact).

Tests: 52/52 pass (yandex_realty_serp + yandex_helpers).
Ruff clean.
2026-05-27 12:23:11 +05:00
d4cbd4bb9f feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)
Some checks failed
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Has been cancelled
2026-05-24 11:17:13 +00:00
b21d7c7c85 feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 32s
2026-05-23 15:28:34 +00:00
lekss361
ead2ac16c5 feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)
Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state
post-hydration) with selectolax DOM parsing using helpers from PR #456.

Changes:
- New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo)
- Selector: [data-test="OffersSerpItem"] x ~23 per page
- Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex),
  price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date),
  street address, photos (avatars.mds.yandex domain filter + size upgrade),
  newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id)
- ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing
  fills later), listing_segment="vtorichka"
- Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by
  source_id. **kwargs swallow keeps old rooms/sort/pages callers compat.
- Preserved: BaseScraper interface, name="yandex" (matches existing rows +
  avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed),
  request_delay_sec=5.0, sleep_between_requests, tenacity retry.

Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio /
no-price / empty page / multi-card / URL builder / city override) — all pass.
Ruff clean.
2026-05-23 16:21:47 +03:00
lekss361
02267d0306 feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest

Первый запуск на сервере (вручную, один раз):
  - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
  - docker network create gendesign_shared (если нет)
  - docker compose -p gendesign-tradein up -d
  - docker compose -p gendesign exec caddy caddy reload
2026-05-21 00:25:39 +03:00