fix(yandex-valuation): off-by-one cell index in _parse_row_cells #552

Merged

lekss361 merged 1 commit from fix/tradein-yandex-cell-off-by-one into main

2026-05-24 19:57:18 +00:00

lekss361 commented

2026-05-24 19:51:59 +00:00

Owner

Summary

Real Yandex .OffersArchiveSearchOffers__row layout (curl_cffi probe 2026-05-24):

cell[0] = photo (empty text)
cell[1] = title — area + rooms + floor jammed: "48,5 м², 1-комнатная19 этаж"
cell[2..5] = prices, dates, status (unchanged from PR #544)

PR #544 read area+rooms from cell[0] and floor from cell[1] → 0% fill for all three fields on 138 live rows. This PR reads everything from cell[1].

Verified locally before push

138/138 area filled (100%)
138/138 rooms filled (100%)
138/138 floor filled (100%)
plausible (10-500 m²): 100%
inverted_dates: 0
house_id linked: 100%

Tests

40/40 pytest pass — DOM fixtures updated to match real layout
ruff clean

Full sweep performed

After applying locally: 5077 rows / 102 distinct houses / 98.4% plausible area.

## Summary Real Yandex `.OffersArchiveSearchOffers__row` layout (curl_cffi probe 2026-05-24): - `cell[0]` = photo (empty text) - `cell[1]` = title — area + rooms + floor jammed: `"48,5 м², 1-комнатная19 этаж"` - `cell[2..5]` = prices, dates, status (unchanged from PR #544) PR #544 read area+rooms from cell[0] and floor from cell[1] → **0% fill** for all three fields on 138 live rows. This PR reads everything from cell[1]. ## Verified locally before push - 138/138 area filled (100%) - 138/138 rooms filled (100%) - 138/138 floor filled (100%) - plausible (10-500 m²): 100% - inverted_dates: 0 - house_id linked: 100% ## Tests - 40/40 pytest pass — DOM fixtures updated to match real layout - ruff clean ## Full sweep performed After applying locally: 5077 rows / 102 distinct houses / 98.4% plausible area.

lekss361 added 1 commit 2026-05-24 19:51:59 +00:00

fix(yandex-valuation): correct off-by-one cell index in _parse_row_cells d692511cef

Real Yandex DOM layout (curl_cffi probe 2026-05-24):
- cell[0] = photo (empty text)
- cell[1] = title — area + rooms + floor jammed: "48,5 м², 1-комнатная19 этаж"
- cell[2..5] = prices, dates, status (unchanged from PR #544)

Previous code read area+rooms from cell[0] and floor from cell[1] → 0% fill
for all three fields on 138 live rows. New code reads everything from cell[1].

Verified locally: 138/138 area, rooms, floor filled (100%, plausible 100%).
Tests updated to match real Yandex layout (empty photo + jammed title cell).

lekss361 commented

2026-05-24 19:57:01 +00:00

Author

Owner

Deep Code Review — PR #552 verdict

Summary

Status: APPROVE
Files: 2 (P1: scraper service + tests) · +16 / -17
Off-by-one fix is correct and confirmed by live probe (138/138 fill, full sweep 5077 rows / 98.4% plausible area).

Cross-file impact

_parse_row_cells has a single internal caller (_parse_history_items line 301). No external API/DB schema change.
Legacy _parse_item_text (chunked-text fallback) unchanged — operates on full text, unaffected by cell-index shift.
Guard if len(cells) < 5 predates this PR; safe given cell_texts[1] access is guarded by if len(cell_texts) > 1.

Correctness verification

Real Yandex layout per probe: cell[0]=photo (empty), cell[1]="48,5 м², 1-комнатная19 этаж" (jammed). Walked each regex against the jammed string:

RE_ITEM_AREA (?<!\d)(\d{2,4}(?:[.,]\d{1,2})?)\s*м² → matches 48,5 м² → area=48.5 ✓
RE_ITEM_ROOMS (\d+)\s*-\s*комнатн → matches 1-комнатн → rooms=1 ✓ (no false-match on 19)
RE_ITEM_FLOOR (\d+)\s*этаж greedy → consumes 19 then matches этаж → floor=19 ✓ (NOT just 9 — \s* only allows whitespace, regex engine finds longest leftmost match anchored at этаж)
Studio variant "Студия 25 м²5 этаж" — RE_ITEM_STUDIO matches, area=25, floor=5 ✓

Test fixtures (2 DOM rows + studio) updated to mirror real layout, including empty photo cell at [0].

Minor (Low — non-blocking)

_parse_history_items docstring (lines 289-291) still documents the OLD layout: [0] area+rooms, [1] floor, [2] start_price+ppm2, .... Consider updating in a followup to [0] photo, [1] title (area+rooms+floor jammed), [2] start_price+ppm2, [3] last_price+ppm2, [4] publish_date+exposure, [5] status/removed_date to prevent confusing the next reader (the body of _parse_row_cells got the correct comment, but the parent docstring still references PR #544's incorrect mapping).

Positive

PR description includes live verification metrics (100% fill, 0 inverted dates, full sweep numbers).
Tests + ruff clean.
Tight scope — only the buggy cell indices changed, no unrelated refactor.
Inline comment cites real layout + probe date as forensic anchor.

Reversibility / blast radius

Risk: low — pure parser change, contained to a single private method.
Reversibility: trivial revert (16 lines).
Followup: docstring fix (above) + monitor next sweep to confirm prod parity with the 138-row local probe.

## Deep Code Review — PR #552 verdict ### Summary - Status: APPROVE - Files: 2 (P1: scraper service + tests) · +16 / -17 - Off-by-one fix is correct and confirmed by live probe (138/138 fill, full sweep 5077 rows / 98.4% plausible area). ### Cross-file impact - `_parse_row_cells` has a single internal caller (`_parse_history_items` line 301). No external API/DB schema change. - Legacy `_parse_item_text` (chunked-text fallback) unchanged — operates on full text, unaffected by cell-index shift. - Guard `if len(cells) < 5` predates this PR; safe given `cell_texts[1]` access is guarded by `if len(cell_texts) > 1`. ### Correctness verification Real Yandex layout per probe: `cell[0]=photo (empty)`, `cell[1]="48,5 м², 1-комнатная19 этаж"` (jammed). Walked each regex against the jammed string: - `RE_ITEM_AREA` `(?<!\d)(\d{2,4}(?:[.,]\d{1,2})?)\s*м²` → matches `48,5 м²` → area=48.5 ✓ - `RE_ITEM_ROOMS` `(\d+)\s*-\s*комнатн` → matches `1-комнатн` → rooms=1 ✓ (no false-match on `19`) - `RE_ITEM_FLOOR` `(\d+)\s*этаж` greedy → consumes `19` then matches ` этаж` → floor=19 ✓ (NOT just `9` — `\s*` only allows whitespace, regex engine finds longest leftmost match anchored at `этаж`) - Studio variant `"Студия 25 м²5 этаж"` — `RE_ITEM_STUDIO` matches, area=25, floor=5 ✓ Test fixtures (2 DOM rows + studio) updated to mirror real layout, including empty photo cell at [0]. ### Minor (Low — non-blocking) - `_parse_history_items` docstring (lines 289-291) still documents the OLD layout: `[0] area+rooms, [1] floor, [2] start_price+ppm2, ...`. Consider updating in a followup to `[0] photo, [1] title (area+rooms+floor jammed), [2] start_price+ppm2, [3] last_price+ppm2, [4] publish_date+exposure, [5] status/removed_date` to prevent confusing the next reader (the body of `_parse_row_cells` got the correct comment, but the parent docstring still references PR #544's incorrect mapping). ### Positive - PR description includes live verification metrics (100% fill, 0 inverted dates, full sweep numbers). - Tests + ruff clean. - Tight scope — only the buggy cell indices changed, no unrelated refactor. - Inline comment cites real layout + probe date as forensic anchor. ### Reversibility / blast radius - Risk: low — pure parser change, contained to a single private method. - Reversibility: trivial revert (16 lines). - Followup: docstring fix (above) + monitor next sweep to confirm prod parity with the 138-row local probe.