feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser #466

Merged

lekss361 merged 1 commit from feat/tradein-yandex-detail-parser into main

2026-05-23 13:45:10 +00:00

lekss361 commented

2026-05-23 13:41:15 +00:00

Owner

Motivation

Stage 4 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker A — parallel with #PRs yandex_newbuilding.py and yandex_valuation.py).

Module

tradein-mvp/backend/app/services/scrapers/yandex_detail.py — YandexDetailScraper(BaseScraper) with async fetch_detail(offer_url). Authoritative price comes from Product JSON-LD offers.price (exact int); DOM sections supply description, agent block, summary stats, metro, photos, NLP fields.

DetailEnrichment fields

Group	Fields
Identification	`offer_id`, `source_url`
Pricing	`price_rub` (Product LD), `price_per_m2` (summary text fallback)
Basic params	`title`, `rooms`, `area_m2`, `floor`, `total_floors`
Address	`address` (full RU format)
Description	`description` (text of section after H2 "Описание")
Sale	`sale_type_text` (raw RU phrase)
Stats	`views_total`, `publish_date`, `publish_date_relative`
Agency block (OfferCardAuthorInfo)	`agency_name`, `agency_founded_year`, `agency_objects_count`, `seller_name`
Metro	`metro_stations: list[MetroStation]` from "Расположение"
Photos	`photo_urls` (8 sizes from Product.image[] array)
Newbuilding	`newbuilding_url`, `newbuilding_id`
NLP best-effort	`house_type_nlp`, `year_built_hint`, `metro_walk_min`

Tests

tests/test_yandex_detail.py: 35 passed, 0 failed
ruff check: All checks passed (line ≤ 100)

Worker notes (worth tracking)

selectolax.text(strip=True) concatenates text nodes without separators. When the agency block uses separate inline elements (e.g. "Год основания 2003" in one span + "247 объектов" in another), the digit strings merge into "2003247 объектов" and RE_AGENCY_OBJECTS misfires. Fixture HTML in tests uses a single <p> to avoid this. Real Yandex DOM may need additional whitespace normalization — followup for production smoke or limitations/ vault entry.

Files

tradein-mvp/backend/app/services/scrapers/yandex_detail.py (new, 369 lines)
tradein-mvp/backend/tests/test_yandex_detail.py (new, 399 lines)

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE. Wave 4 sibling PRs: yandex_newbuilding.py + yandex_valuation.py.

## Motivation Stage 4 of 9 for YandexRealtyScraper v1 (Wave 4 / Worker A — parallel with #PRs `yandex_newbuilding.py` and `yandex_valuation.py`). ## Module `tradein-mvp/backend/app/services/scrapers/yandex_detail.py` — `YandexDetailScraper(BaseScraper)` with async `fetch_detail(offer_url)`. Authoritative price comes from Product JSON-LD `offers.price` (exact int); DOM sections supply description, agent block, summary stats, metro, photos, NLP fields. ## DetailEnrichment fields | Group | Fields | |---|---| | Identification | `offer_id`, `source_url` | | Pricing | `price_rub` (Product LD), `price_per_m2` (summary text fallback) | | Basic params | `title`, `rooms`, `area_m2`, `floor`, `total_floors` | | Address | `address` (full RU format) | | Description | `description` (text of section after H2 "Описание") | | Sale | `sale_type_text` (raw RU phrase) | | Stats | `views_total`, `publish_date`, `publish_date_relative` | | Agency block (OfferCardAuthorInfo) | `agency_name`, `agency_founded_year`, `agency_objects_count`, `seller_name` | | Metro | `metro_stations: list[MetroStation]` from "Расположение" | | Photos | `photo_urls` (8 sizes from Product.image[] array) | | Newbuilding | `newbuilding_url`, `newbuilding_id` | | NLP best-effort | `house_type_nlp`, `year_built_hint`, `metro_walk_min` | ## Tests ``` tests/test_yandex_detail.py: 35 passed, 0 failed ruff check: All checks passed (line ≤ 100) ``` ## Worker notes (worth tracking) `selectolax.text(strip=True)` concatenates text nodes without separators. When the agency block uses separate inline elements (e.g. "Год основания 2003" in one span + "247 объектов" in another), the digit strings merge into `"2003247 объектов"` and `RE_AGENCY_OBJECTS` misfires. Fixture HTML in tests uses a single `<p>` to avoid this. Real Yandex DOM may need additional whitespace normalization — followup for production smoke or `limitations/` vault entry. ## Files - `tradein-mvp/backend/app/services/scrapers/yandex_detail.py` (new, 369 lines) - `tradein-mvp/backend/tests/test_yandex_detail.py` (new, 399 lines) ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE. Wave 4 sibling PRs: yandex_newbuilding.py + yandex_valuation.py.

lekss361 added 1 commit 2026-05-23 13:41:16 +00:00

feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser f9e0b6bcbe

Stage 4 of YandexRealtyScraper v1 (Wave 4 / Worker A — parallel with
yandex_newbuilding.py and yandex_valuation.py).

Module: app/services/scrapers/yandex_detail.py
- YandexDetailScraper(BaseScraper) — async fetch_detail(offer_url)
- DetailEnrichment Pydantic model (offer_id, price/title/rooms/area/floor,
  description, sale_type, views, publish_date, agency block, metro stations,
  photos 8 sizes, newbuilding linkage, NLP best-effort fields)
- Helpers: _find_section_text(h2/h3 -> next siblings), _parse_metro_stations,
  _extract_relative_date, _extract_seller_name, _parse_title

Price source: Product JSON-LD offers.price (exact int) — falls back to None
if missing. Photos: 8 sizes from Product.image[] array.
NLP: parse_house_type / RE_YEAR / RE_METRO_WALK from description text
(best-effort — None when not present).

Tests: 35 unit tests against hand-crafted fixture HTML. All pass. Ruff clean.

lekss361 merged commit 3d90221fa0 into main

2026-05-23 13:45:10 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:45:10 +00:00

feat(tradein): yandex_detail.py — Product JSON-LD + DOM detail parser (#466)

lekss361 commented

2026-05-23 13:45:36 +00:00

Author

Owner

Deep code review — verdict APPROVE, merged (squash, commit 3d90221).

Reviewed:

JSON-LD authoritative pricing — graceful: handles missing Product, offers as array vs dict, missing price field, non-int values. All edge paths covered.
Photo URLs — raw Product.image[] strings stored as-is. Safe downstream: _safe_url(cdn=True) in trade_in_pdf.py:63 allowlists avatars.mds.yandex.net. Frontend <img src> no JS exec. Acceptable for pilot.
Helper integrations (find_ld_by_type, parse_house_type, parse_ru_date, parse_rub, RE_AGENCY_*, RE_METRO_WALK, RE_VIEWS, RE_YEAR) — all present in yandex_helpers.py.
BaseScraper inheritance correct: fetch_around raises NotImplementedError (offer-id-based scraper), _http_get + sleep_between_requests used per pattern.
35 tests cover happy path + no-LD fallback + studio + no-agency + agency block + seller + newbuilding (present/absent) + metro (multi/empty/helper) + relative dates + publish date + photos (8-array/single-string) + invalid URL + title helpers.
Conventions clean: httpx via base, async, logger.*, Pydantic v2, no SQL, ruff line ≤100.
merge_base behind main by 1 commit (#460 Stage 4a API) — no file overlap in scrapers/, no rebase needed.

Worker's whitespace note acknowledged (selectolax .text(strip=True) concatenation):

Real Yandex DOM may emit agency_founded_year/objects_count from separate inline spans → digits merge → RE_AGENCY_OBJECTS misfires.
Followup tracked for production smoke: switch to .text(separator=' ', strip=True) or iterate child nodes with manual joining. Non-blocking — fixture uses <p> workaround.

Gap (non-blocking, recommended followup): real Yandex HTML snapshot test to catch DOM drift.

Wave 4 sibling PRs (#467 newbuilding, #468 valuation) target different files — no conflict expected.

Deep code review — verdict APPROVE, merged (squash, commit `3d90221`). Reviewed: - JSON-LD authoritative pricing — graceful: handles missing Product, offers as array vs dict, missing price field, non-int values. All edge paths covered. - Photo URLs — raw Product.image[] strings stored as-is. Safe downstream: `_safe_url(cdn=True)` in `trade_in_pdf.py:63` allowlists `avatars.mds.yandex.net`. Frontend `<img src>` no JS exec. Acceptable for pilot. - Helper integrations (`find_ld_by_type`, `parse_house_type`, `parse_ru_date`, `parse_rub`, `RE_AGENCY_*`, `RE_METRO_WALK`, `RE_VIEWS`, `RE_YEAR`) — all present in `yandex_helpers.py`. - BaseScraper inheritance correct: `fetch_around` raises `NotImplementedError` (offer-id-based scraper), `_http_get` + `sleep_between_requests` used per pattern. - 35 tests cover happy path + no-LD fallback + studio + no-agency + agency block + seller + newbuilding (present/absent) + metro (multi/empty/helper) + relative dates + publish date + photos (8-array/single-string) + invalid URL + title helpers. - Conventions clean: httpx via base, async, logger.*, Pydantic v2, no SQL, ruff line ≤100. - merge_base behind main by 1 commit (#460 Stage 4a API) — no file overlap in scrapers/, no rebase needed. Worker's whitespace note acknowledged (selectolax `.text(strip=True)` concatenation): - Real Yandex DOM may emit agency_founded_year/objects_count from separate inline spans → digits merge → `RE_AGENCY_OBJECTS` misfires. - Followup tracked for production smoke: switch to `.text(separator=' ', strip=True)` or iterate child nodes with manual joining. Non-blocking — fixture uses `<p>` workaround. Gap (non-blocking, recommended followup): real Yandex HTML snapshot test to catch DOM drift. Wave 4 sibling PRs (#467 newbuilding, #468 valuation) target different files — no conflict expected.

lekss361 referenced this pull request

2026-05-23 13:47:34 +00:00

feat(tradein): yandex_newbuilding.py — ЖК landing parser (149 selectors + NLP) #467