lekss361/gendesign

Fork 0

feat(tradein): yandex_valuation.py — anonymous house-history scraper #468

Merged

lekss361 merged 1 commit from feat/tradein-yandex-valuation-api into main

2026-05-23 13:46:20 +00:00

lekss361 commented

2026-05-23 13:41:46 +00:00

Owner

Motivation

Stage 6 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker C — parallel with yandex_detail.py and yandex_newbuilding.py).

Yandex Valuation tool is the only anonymous house-history source in our stack — vs Cian Calculator (cookies required) and Avito IMV (POST + JWT geoHash). Pagination via ?page=N yields 12+ historical offers per house — the widest history net we have.

Module

tradein-mvp/backend/app/services/scrapers/yandex_valuation.py — YandexValuationScraper(BaseScraper) with async fetch_house_history(address, offer_category="APARTMENT", offer_type="SELL", page=1).

URL: GET /otsenka-kvartiry-po-adresu-onlayn/?address=...&offerCategory=...&offerType=...&page=N — NO cookies, NO auth.

Models

ValuationHouseMeta: year_built, total_floors, house_type, ceiling_height, has_lift, total_objects, has_panorama

ValuationHistoryItem: area_m2, rooms (0 for studio), floor, start_price, start_price_per_m2, last_price, last_price_per_m2, publish_date (DMY), exposure_days, status (В продаже / Снято)

YandexValuationResult: house + history_items[] + pagination metadata + source_url + raw_payload

History extraction strategy

Two-tier fallback (Yandex history items have unstable selectors):

Primary: explicit [data-test*="HistoryItem"] / [data-test*="ValuationItem"] containers (if Yandex adds them later)
Fallback: split body text into ±200/+100 char windows around every DD.MM.YYYY date match; parse each chunk as candidate item; dedup by (publish_date, area_m2, floor). Capped at 30 items per page.

Worker notes

_RE_PRICE_TOKEN carries a negative lookahead (?!\s*за\s*м²) to prevent per-m2 price tokens being captured as total-price tokens. Without it, last_price would mis-resolve to e.g. 117000 (the m² rate) instead of 3_100_000 (the actual price).

Tests

tests/test_yandex_valuation.py: 16 passed, 0 failed (0.78s)
ruff check: All checks passed (line ≤ 100)

Files

tradein-mvp/backend/app/services/scrapers/yandex_valuation.py (new, 331 lines)
tradein-mvp/backend/tests/test_yandex_valuation.py (new, 242 lines)

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE. Wave 4 sibling PRs: yandex_detail.py + yandex_newbuilding.py.

## Motivation Stage 6 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker C — parallel with `yandex_detail.py` and `yandex_newbuilding.py`). Yandex Valuation tool is the **only anonymous** house-history source in our stack — vs Cian Calculator (cookies required) and Avito IMV (POST + JWT geoHash). Pagination via `?page=N` yields 12+ historical offers per house — the widest history net we have. ## Module `tradein-mvp/backend/app/services/scrapers/yandex_valuation.py` — `YandexValuationScraper(BaseScraper)` with async `fetch_house_history(address, offer_category="APARTMENT", offer_type="SELL", page=1)`. URL: `GET /otsenka-kvartiry-po-adresu-onlayn/?address=...&offerCategory=...&offerType=...&page=N` — **NO cookies, NO auth**. ## Models **ValuationHouseMeta**: `year_built`, `total_floors`, `house_type`, `ceiling_height`, `has_lift`, `total_objects`, `has_panorama` **ValuationHistoryItem**: `area_m2`, `rooms` (0 for studio), `floor`, `start_price`, `start_price_per_m2`, `last_price`, `last_price_per_m2`, `publish_date` (DMY), `exposure_days`, `status` (В продаже / Снято) **YandexValuationResult**: house + history_items[] + pagination metadata + source_url + raw_payload ## History extraction strategy Two-tier fallback (Yandex history items have unstable selectors): 1. **Primary**: explicit `[data-test*="HistoryItem"]` / `[data-test*="ValuationItem"]` containers (if Yandex adds them later) 2. **Fallback**: split body text into ±200/+100 char windows around every DD.MM.YYYY date match; parse each chunk as candidate item; dedup by `(publish_date, area_m2, floor)`. Capped at 30 items per page. ## Worker notes `_RE_PRICE_TOKEN` carries a negative lookahead `(?!\s*за\s*м²)` to prevent per-m2 price tokens being captured as total-price tokens. Without it, `last_price` would mis-resolve to e.g. 117000 (the m² rate) instead of 3_100_000 (the actual price). ## Tests ``` tests/test_yandex_valuation.py: 16 passed, 0 failed (0.78s) ruff check: All checks passed (line ≤ 100) ``` ## Files - `tradein-mvp/backend/app/services/scrapers/yandex_valuation.py` (new, 331 lines) - `tradein-mvp/backend/tests/test_yandex_valuation.py` (new, 242 lines) ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE. Wave 4 sibling PRs: yandex_detail.py + yandex_newbuilding.py.

lekss361 added 1 commit 2026-05-23 13:41:47 +00:00

feat(tradein): yandex_valuation.py — anonymous house-history scraper fda9ae4dc1

Stage 6 of YandexRealtyScraper v1 (Wave 4 / Worker C — parallel with
yandex_detail.py and yandex_newbuilding.py).

Module: app/services/scrapers/yandex_valuation.py
- YandexValuationScraper(BaseScraper) — fetch_house_history(address, ...)
  URL: GET /otsenka-kvartiry-po-adresu-onlayn/?address=...&offerCategory=...
       &offerType=...&page=N
  ANONYMOUS — no cookies, no auth (vs Cian Calculator).
- ValuationHouseMeta (year_built, total_floors, house_type, ceiling_height,
  has_lift, total_objects, has_panorama)
- ValuationHistoryItem (area_m2, rooms, floor, start/last price + ppm2,
  publish_date DMY, exposure_days, status)
- YandexValuationResult — house + history_items[] + pagination metadata
- Two-strategy history extraction: data-test container (if present), else
  text-chunk-around-DMY-date fallback with dedup by (date, area, floor).
- _RE_PRICE_TOKEN uses negative lookahead (?!\s*за\s*м²) to exclude ppm2
  tokens from total-price extraction.

Tests: 16 unit tests against fixture HTML. All pass. Ruff clean.

lekss361 referenced this pull request

2026-05-23 13:45:36 +00:00

feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser #466

lekss361 merged commit 7056ecb23b into main

2026-05-23 13:46:20 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:46:23 +00:00

feat(tradein): yandex_valuation.py — anonymous house-history scraper (#468)

lekss361 commented

2026-05-23 13:47:02 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE (with MINOR nits).

Two-tier fallback assessment: graceful. Primary CSS selector returns empty (Yandex doesn't expose data-test*="HistoryItem" currently), fallback iterates over DD.MM.YYYY matches with ±200/+100 char windows, dedups by (publish_date, area_m2, floor), caps at 30/page. Empty pages return YandexValuationResult with empty history_items — orchestrator can paginate until len()==0. No crashes on no-date body.

Price regex correctness (_RE_PRICE_TOKEN): confirmed.

"3 200 000 ₽" → matches, parse_rub → 3_200_000 ✓
"117 000 ₽ за м²" → excluded by negative lookahead ✓
"4,4 млн ₽" → matches via (?:[.,]\d+)?\s*(?:млн)? branch ✓
Order of first 2 tokens = (start_price, last_price) — correct per fixture

Anonymous fetch reliability: plain httpx via BaseScraper UA rotation + 4.0s request_delay_sec + tenacity retry on 403/429/5xx. HTTP errors return None (logged warning, no crash). Sufficient for anonymous endpoint; if Yandex adds rate-limit/captcha later, graceful degradation already in place.

Minor nits (non-blocking, can be follow-up PR):

_RE_PRICE_TOKEN lookahead (?!\s*за\s*м²) doesn't cover slash variant "₽/м²" — if Yandex ever switches notation, would mis-capture. Suggest: (?!\s*(?:за\s*м²|/\s*м²)).
status enum regex (В\s+продаже|Снят[оа]) misses "Продано" — add |Продан[оа]? if surfaces in real HTML.
RE_ITEM_FLOOR = (\d+)\s*этаж also matches "9 этажей" from house meta block — if chunk window pulls in meta, spurious floor. Tighten to (\d+)\s*этаж\b(?!ей|и) or scope to item containers.
test_parse_history_item_full assertion "В" in item.status is too loose (any Cyrillic capital В passes) — strict-match item.status.startswith("В продаже").
raw_payload keeps only body_len+items_count — re-parse from cached HTML not possible. Trade-off vs OOM is fine; document if needed.
No async test with mocked httpx for fetch_house_history — synchronous parse() is fully covered (16/16), but integration path untested. Snapshot test against real Yandex HTML recommended for Stage 7+.

Cross-check: no DB writes, no migration, no API contract change. Pure module addition. No conflict with PR #466/#467 (different files). Vault decision YandexRealtyScraper_v1_Implementation_Plan.md Stage 6 scope matches. Worker should add code/modules/tradein/Yandex_Valuation_Scraper.md entry.

Next: verify deploy-tradein.yml triggers + CI 16/16 green on main.

Merged via deep-code-reviewer — verdict APPROVE (with MINOR nits). **Two-tier fallback assessment**: graceful. Primary CSS selector returns empty (Yandex doesn't expose `data-test*="HistoryItem"` currently), fallback iterates over `DD.MM.YYYY` matches with ±200/+100 char windows, dedups by `(publish_date, area_m2, floor)`, caps at 30/page. Empty pages return `YandexValuationResult` with empty `history_items` — orchestrator can paginate until `len()==0`. No crashes on no-date body. **Price regex correctness (`_RE_PRICE_TOKEN`)**: confirmed. - `"3 200 000 ₽"` → matches, parse_rub → 3_200_000 ✓ - `"117 000 ₽ за м²"` → excluded by negative lookahead ✓ - `"4,4 млн ₽"` → matches via `(?:[.,]\d+)?\s*(?:млн)?` branch ✓ - Order of first 2 tokens = (start_price, last_price) — correct per fixture **Anonymous fetch reliability**: plain httpx via BaseScraper UA rotation + 4.0s `request_delay_sec` + tenacity retry on 403/429/5xx. HTTP errors return `None` (logged warning, no crash). Sufficient for anonymous endpoint; if Yandex adds rate-limit/captcha later, graceful degradation already in place. **Minor nits (non-blocking, can be follow-up PR)**: 1. `_RE_PRICE_TOKEN` lookahead `(?!\s*за\s*м²)` doesn't cover slash variant `"₽/м²"` — if Yandex ever switches notation, would mis-capture. Suggest: `(?!\s*(?:за\s*м²|/\s*м²))`. 2. `status` enum regex `(В\s+продаже|Снят[оа])` misses `"Продано"` — add `|Продан[оа]?` if surfaces in real HTML. 3. `RE_ITEM_FLOOR = (\d+)\s*этаж` also matches `"9 этажей"` from house meta block — if chunk window pulls in meta, spurious floor. Tighten to `(\d+)\s*этаж\b(?!ей|и)` or scope to item containers. 4. `test_parse_history_item_full` assertion `"В" in item.status` is too loose (any Cyrillic capital В passes) — strict-match `item.status.startswith("В продаже")`. 5. `raw_payload` keeps only `body_len`+`items_count` — re-parse from cached HTML not possible. Trade-off vs OOM is fine; document if needed. 6. No async test with mocked httpx for `fetch_house_history` — synchronous `parse()` is fully covered (16/16), but integration path untested. Snapshot test against real Yandex HTML recommended for Stage 7+. **Cross-check**: no DB writes, no migration, no API contract change. Pure module addition. No conflict with PR #466/#467 (different files). Vault decision `YandexRealtyScraper_v1_Implementation_Plan.md` Stage 6 scope matches. Worker should add `code/modules/tradein/Yandex_Valuation_Scraper.md` entry. Next: verify `deploy-tradein.yml` triggers + CI 16/16 green on main.

lekss361 referenced this pull request

2026-05-23 13:47:34 +00:00

feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) #467