refactor(yandex-valuation): DOM-based parser via .OffersArchiveSearchOffers__row #544

Merged

lekss361 merged 1 commit from refactor/tradein-yandex-dom-based-parser into main

2026-05-24 16:25:57 +00:00

lekss361 commented

2026-05-24 16:20:54 +00:00

Owner

Summary

Replaces chunked-text fallback with structured DOM parsing via .OffersArchiveSearchOffers__row selectors discovered via live HTML recon.

Each history item is a row with 6 positional cells:

Cell	Content
0	area + rooms (e.g. `48,5 м², 1-комнатная`)
1	floor
2	start price + ppm2 (concatenated)
3	last price + ppm2
4	publish_date + exposure
5	status (`В продаже`) OR removed_date

Chunked-text fallback kept as last-resort path with logger.warning so we notice if Yandex changes class names.

Why

Live metrics after PR #542 (chunked-text regex tuned 3x):

area_filled: 12.7% (22/173 rows)
plausible area (10-500 m²): 8.7% (15/173)
max_area: 1657 m² (junk)

3 PRs of regex whack-a-mole (#538/#541/#542) each fixed one bug and broke another. DOM-based parsing eliminates all of them simultaneously:

area = year+area concat → impossible with cell isolation
area = sub-fragment 2,2 → impossible
date inversion → cell [4] always publish, cell [5] always removed
1-комн/2-комн/3-комн all parsed identically

Tests

40/40 pytest pass (36 prior + 4 new DOM-row tests for 2-row layout / studio / fallback / malformed row)
ruff clean

Followup

After merge: purge yandex_valuation history + cache, re-sweep Базовый 52, verify plausible area ≥80% (from 8.7%).

## Summary Replaces chunked-text fallback with structured DOM parsing via `.OffersArchiveSearchOffers__row` selectors discovered via live HTML recon. Each history item is a row with 6 positional cells: | Cell | Content | |---|---| | 0 | area + rooms (e.g. `48,5 м², 1-комнатная`) | | 1 | floor | | 2 | start price + ppm2 (concatenated) | | 3 | last price + ppm2 | | 4 | publish_date + exposure | | 5 | status (`В продаже`) OR removed_date | Chunked-text fallback kept as last-resort path with `logger.warning` so we notice if Yandex changes class names. ## Why Live metrics after PR #542 (chunked-text regex tuned 3x): - area_filled: **12.7%** (22/173 rows) - plausible area (10-500 m²): **8.7%** (15/173) - max_area: 1657 m² (junk) 3 PRs of regex whack-a-mole (#538/#541/#542) each fixed one bug and broke another. DOM-based parsing eliminates all of them simultaneously: - area = year+area concat → impossible with cell isolation - area = sub-fragment `2,2` → impossible - date inversion → cell [4] always publish, cell [5] always removed - 1-комн/2-комн/3-комн all parsed identically ## Tests - 40/40 pytest pass (36 prior + 4 new DOM-row tests for 2-row layout / studio / fallback / malformed row) - ruff clean ## Followup - After merge: purge `yandex_valuation` history + cache, re-sweep Базовый 52, verify `plausible area` ≥80% (from 8.7%).

lekss361 added 1 commit 2026-05-24 16:20:55 +00:00

refactor(yandex-valuation): DOM-based parser via .OffersArchiveSearchOffers__row e070f06c26

Live HTML recon discovered stable BEM-style classes. Each history item is a
.OffersArchiveSearchOffers__row with 6 positional cells: area+rooms, floor,
start_price+ppm2, last_price+ppm2, publish_date+exposure, status/removed_date.

Drops chunked-text fallback as primary strategy (kept as last-resort with
warning log). Eliminates 3 PRs of regex whack-a-mole (#538/#541/#542):
- area = year+area concat → impossible with cell isolation
- area = sub-fragment '2,2' → impossible
- date inversion → cell [4] is always publish, cell [5] is always removed
- 1-комн/2-комн/3-комн all parsed identically

Re-sweep Базовый 52 should now hit ~90%+ plausible area_filled vs 8.7%
from PR #542.

lekss361 commented

2026-05-24 16:25:50 +00:00

Author

Owner

Deep Code Review — verdict

Summary

Status: APPROVE
Files: 2 (P1: scraper service, P2: tests)
Lines: +254 / -7
PR: #544 @ e070f06

Architecture assessment

This is the correct response to 3 PRs of regex whack-a-mole (#538/#541/#542). DOM-based per-cell parsing eliminates the entire class of chunked-text ambiguity:

Year-concat (20244,6 m²) impossible — area cell holds only area+rooms
Sub-fragment (2,2 from inside 52,2) impossible — each cell parsed independently
Date inversion (publish/removed flip) impossible — cell[4] always publish, cell[5] always removed
N-room cells parsed uniformly through RE_ITEM_ROOMS / RE_ITEM_STUDIO

Fallback path with logger.warning is the right defensive pattern — if Yandex re-hashes BEM classes, we will see it in logs instead of silent failure.

Regression coverage cross-check

All 5 prior parser fixes preserved in the new DOM path:

PR	Fix	Preserved?
#508	`listing_date` (publish_date)	yes — cell[4] via `parse_dmy`
#526	`removed_date` / `total_floors` separation	yes — cell[5] for removed; `_parse_house_meta` unchanged
#538	area sanity cap (drop >10_000 or <=0)	yes — lines 304-305, belt-and-suspenders
#541	chronological date swap	yes — lines 364-365
#542	tight area regex (year-concat blocker)	yes — `RE_ITEM_AREA` constant reused

Legacy _parse_item_text / _parse_items_from_chunked_text kept intact — all 36 prior tests still exercise them through the fallback path.

Test coverage

40 tests total (36 prior + 4 new DOM): two-row layout, studio, fallback exercise, malformed row
test_dom_parser_extracts_two_rows covers the critical concatenation scenarios: "5,1 млн ₽105 155 ₽ за м²" price+ppm2 token split via _RE_PRICE_TOKEN / _RE_PPM2_TOKEN, "23.10.2023В экспозиции 945 дней" date+exposure split via parse_dmy + RE_ITEM_EXPOSURE, both rooms variants, and chronological date ordering (row 2: publish 2026-02-12, removed 2026-07-17)
test_dom_parser_skips_malformed_row exercises the len(cells) < 5 defensive guard
test_dom_parser_fallback_when_no_rows correctly under-asserts (only isinstance(.., list)) — fallback behavior is intentionally best-effort

Performance (bulk sweep #533)

No concern. tree.css(".OffersArchiveSearchOffers__row") then .css(".OffersArchiveSearchOffers__cell") per row = O(rows × cells) selectolax calls — selectolax is C-based (lexbor); ~30 rows × 6 selector calls is negligible vs httpx round-trip latency (request_delay_sec = 5.0). For a 1532-obj re-sweep: parser cost is far below the per-request delay floor, no regression.

Nits (non-blocking)

_parse_row_cells(cls, row) is decorated @classmethod but never references cls — would be cleaner as @staticmethod. Style only.
tree.css(".OffersArchiveSearchOffers__cell") is a descendant selector — if Yandex ever nests another .OffersArchiveSearchOffers__cell inside a cell, a child selector > would be more defensive. Fine in practice.
The fallback logger.warning fires once per parse; for a 1532-obj sweep where Yandex breaks the selector, log volume = 1532 lines. Consider a Sentry/GlitchTip counter metric on top of the warning. Followup, not blocker.
Defensive guard is len(cells) < 5 but per-cell code uses len(cell_texts) > N — minor asymmetry between minimum (5) and full layout (6). Both variants handled correctly; just an asymmetry to note.

Vault cross-check

This PR is the climax of the recurring "yandex parser whack-a-mole" pattern. The metrics-driven decision (12.7% area_filled at PR #542 → expected at least 80% post-DOM) is well-documented in the PR body. Recommend a vault decisions/ entry post-merge documenting "yandex_valuation: DOM-based parsing as architectural inflection point" with links to the 5 superseded chunked-text fix PRs.

Positive observations

Followup plan in PR body is concrete and measurable (purge history+cache, re-sweep Базовый 52, verify metric uplift)
Fixture HTML is realistic — includes the no-break-space replacement path and the no-separator price+ppm2 concatenation
Comment block on RE_ITEM_AREA (lines 123-127 of head) is exemplary — explains lookbehind, min-digit, and max-digit choices with reference to the 3 prior PRs

Complexity / blast radius

Risk: LOW — additive code path, fallback preserved, all prior tests pass
Reversibility: HIGH — revert restores chunked-text-only behavior; no schema/DB changes
Merge window: anytime

Verdict: APPROVE — merge unblocked.

## Deep Code Review — verdict ### Summary - Status: APPROVE - Files: 2 (P1: scraper service, P2: tests) - Lines: +254 / -7 - PR: #544 @ e070f06 ### Architecture assessment This is the correct response to 3 PRs of regex whack-a-mole (#538/#541/#542). DOM-based per-cell parsing eliminates the entire class of chunked-text ambiguity: - Year-concat (`20244,6 m²`) impossible — area cell holds only area+rooms - Sub-fragment (`2,2` from inside `52,2`) impossible — each cell parsed independently - Date inversion (publish/removed flip) impossible — cell[4] always publish, cell[5] always removed - N-room cells parsed uniformly through `RE_ITEM_ROOMS` / `RE_ITEM_STUDIO` Fallback path with `logger.warning` is the right defensive pattern — if Yandex re-hashes BEM classes, we will see it in logs instead of silent failure. ### Regression coverage cross-check All 5 prior parser fixes preserved in the new DOM path: | PR | Fix | Preserved? | |---|---|---| | #508 | `listing_date` (publish_date) | yes — cell[4] via `parse_dmy` | | #526 | `removed_date` / `total_floors` separation | yes — cell[5] for removed; `_parse_house_meta` unchanged | | #538 | area sanity cap (drop >10_000 or <=0) | yes — lines 304-305, belt-and-suspenders | | #541 | chronological date swap | yes — lines 364-365 | | #542 | tight area regex (year-concat blocker) | yes — `RE_ITEM_AREA` constant reused | Legacy `_parse_item_text` / `_parse_items_from_chunked_text` kept intact — all 36 prior tests still exercise them through the fallback path. ### Test coverage - 40 tests total (36 prior + 4 new DOM): two-row layout, studio, fallback exercise, malformed row - `test_dom_parser_extracts_two_rows` covers the critical concatenation scenarios: `"5,1 млн ₽105 155 ₽ за м²"` price+ppm2 token split via `_RE_PRICE_TOKEN` / `_RE_PPM2_TOKEN`, `"23.10.2023В экспозиции 945 дней"` date+exposure split via `parse_dmy` + `RE_ITEM_EXPOSURE`, both rooms variants, and chronological date ordering (row 2: publish 2026-02-12, removed 2026-07-17) - `test_dom_parser_skips_malformed_row` exercises the `len(cells) < 5` defensive guard - `test_dom_parser_fallback_when_no_rows` correctly under-asserts (only `isinstance(.., list)`) — fallback behavior is intentionally best-effort ### Performance (bulk sweep #533) No concern. `tree.css(".OffersArchiveSearchOffers__row")` then `.css(".OffersArchiveSearchOffers__cell")` per row = O(rows × cells) selectolax calls — selectolax is C-based (lexbor); ~30 rows × 6 selector calls is negligible vs httpx round-trip latency (`request_delay_sec = 5.0`). For a 1532-obj re-sweep: parser cost is far below the per-request delay floor, no regression. ### Nits (non-blocking) 1. `_parse_row_cells(cls, row)` is decorated `@classmethod` but never references `cls` — would be cleaner as `@staticmethod`. Style only. 2. `tree.css(".OffersArchiveSearchOffers__cell")` is a descendant selector — if Yandex ever nests another `.OffersArchiveSearchOffers__cell` inside a cell, a child selector `>` would be more defensive. Fine in practice. 3. The fallback `logger.warning` fires once per parse; for a 1532-obj sweep where Yandex breaks the selector, log volume = 1532 lines. Consider a Sentry/GlitchTip counter metric on top of the warning. Followup, not blocker. 4. Defensive guard is `len(cells) < 5` but per-cell code uses `len(cell_texts) > N` — minor asymmetry between minimum (5) and full layout (6). Both variants handled correctly; just an asymmetry to note. ### Vault cross-check This PR is the climax of the recurring "yandex parser whack-a-mole" pattern. The metrics-driven decision (12.7% area_filled at PR #542 → expected at least 80% post-DOM) is well-documented in the PR body. Recommend a vault `decisions/` entry post-merge documenting "yandex_valuation: DOM-based parsing as architectural inflection point" with links to the 5 superseded chunked-text fix PRs. ### Positive observations - Followup plan in PR body is concrete and measurable (purge history+cache, re-sweep Базовый 52, verify metric uplift) - Fixture HTML is realistic — includes the no-break-space replacement path and the no-separator price+ppm2 concatenation - Comment block on `RE_ITEM_AREA` (lines 123-127 of head) is exemplary — explains lookbehind, min-digit, and max-digit choices with reference to the 3 prior PRs ### Complexity / blast radius - Risk: LOW — additive code path, fallback preserved, all prior tests pass - Reversibility: HIGH — revert restores chunked-text-only behavior; no schema/DB changes - Merge window: anytime Verdict: APPROVE — merge unblocked.