fix(tradein/cian): extract_all_states — discover new .concat() format, not just legacy .push() #2438

Merged

lekss361 merged 1 commit from fix/tradein-cian-state-parser-concat-discovery into main

2026-07-04 21:03:33 +00:00

lekss361 commented

2026-07-04 20:50:33 +00:00

Owner

Summary

Live bug found 2026-07-04 while verifying #2435 (Cian bti_data persistence). A real Cian offer page (200 OK, 595KB) parsed fine via extract_state() — which already tries the new .concat([...]) format first, .push({...}) as fallback (per #639 2026-05 format drift) — but extract_all_states() returned {}. It only ever scanned the legacy .push() pattern (_RE_CIAN_PUSH), never updated for the concat format.

This silently starved both consumers of real data on every current-format page:

cian/detail.py bti sister-state — the exact gap #2435 built persistence for, which could never receive data
cian/newbuilding.py sister-state extraction

Fix

New _extract_all_from_concat(html) — mirrors the existing _extract_from_concat() array-boundary logic (_RE_CIAN_CONCAT + _balanced_array, both unmodified) but without the single-key filter, collecting every {key,value} pair from every .concat([...]) call.
extract_all_states() now merges concat results first, then push-format matches for any (mfe, key) not already found via concat — mirroring extract_state()'s concat-first / push-fallback precedence.

Strictly additive: previously-empty dict now populated; both callers already guard absent keys via .get(..., {}) + truthiness / isinstance, so no behavior regression — only the empty-fallback path gets real data.

Tests

New tests/test_cian_state_parser.py (module had zero coverage before): concat discovery, multiple mfe containers, legacy push regression, mixed concat+push precedence (concat wins), no-config edge case, and an extract_state()/extract_all_states() consistency invariant — the exact invariant the bug violated.

Test plan

pytest tests/test_cian_state_parser.py -v -> 6/6 pass
pytest tests/ -k cian -q -> 118 passed
ruff check on both changed files -> clean
Live-verified before/after on a real 595KB offer page (temporary in-container file swap, reverted; production left unmodified): before -> 0 top-level MFE keys; after -> 6 real MFE containers incl. frontend-offer-card with ~20 sub-keys.

No SQL / schema / auth / pipeline-rule changes. Pure string/regex parsing.

Refs #2435

## Summary Live bug found 2026-07-04 while verifying #2435 (Cian bti_data persistence). A real Cian offer page (200 OK, 595KB) parsed fine via `extract_state()` — which already tries the new `.concat([...])` format first, `.push({...})` as fallback (per #639 2026-05 format drift) — but `extract_all_states()` returned `{}`. It only ever scanned the legacy `.push()` pattern (`_RE_CIAN_PUSH`), never updated for the concat format. This silently starved both consumers of real data on every current-format page: - `cian/detail.py` bti sister-state — the exact gap #2435 built persistence for, which could never receive data - `cian/newbuilding.py` sister-state extraction ### Fix - New `_extract_all_from_concat(html)` — mirrors the existing `_extract_from_concat()` array-boundary logic (`_RE_CIAN_CONCAT` + `_balanced_array`, both unmodified) but without the single-key filter, collecting every `{key,value}` pair from every `.concat([...])` call. - `extract_all_states()` now merges concat results first, then push-format matches for any `(mfe, key)` not already found via concat — mirroring `extract_state()`'s concat-first / push-fallback precedence. Strictly additive: previously-empty dict now populated; both callers already guard absent keys via `.get(..., {})` + truthiness / `isinstance`, so no behavior regression — only the empty-fallback path gets real data. ### Tests New `tests/test_cian_state_parser.py` (module had zero coverage before): concat discovery, multiple mfe containers, legacy push regression, mixed concat+push precedence (concat wins), no-config edge case, and an `extract_state()`/`extract_all_states()` consistency invariant — the exact invariant the bug violated. ## Test plan - [x] `pytest tests/test_cian_state_parser.py -v` -> 6/6 pass - [x] `pytest tests/ -k cian -q` -> 118 passed - [x] `ruff check` on both changed files -> clean - [x] Live-verified before/after on a real 595KB offer page (temporary in-container file swap, reverted; production left unmodified): before -> 0 top-level MFE keys; after -> 6 real MFE containers incl. `frontend-offer-card` with ~20 sub-keys. No SQL / schema / auth / pipeline-rule changes. Pure string/regex parsing. Refs #2435

lekss361 added 1 commit 2026-07-04 20:50:34 +00:00

fix(tradein/cian): extract_all_states — discover new .concat() format, not just legacy .push()

CI Trade-In / changes (pull_request) Successful in 8s

Details

CI / changes (pull_request) Successful in 9s

Details

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI Trade-In / backend-tests (pull_request) Successful in 51s

Details

7833c354e9

Found live 2026-07-04 while verifying #2435 (bti_data persistence): a real Cian
offer page (200 OK, 595KB) parsed fine via extract_state() (which already tries
.concat() first, .push() as fallback per the #639 2026-05 format drift), but
extract_all_states() returned {} — it only ever scanned the legacy .push()
pattern via _RE_CIAN_PUSH, never updated when .concat() became the live format.

This silently starved both of its consumers of real data on every current-format
page: cian/detail.py's bti sister-state (the exact gap #2435 built persistence
for, which could never receive data) and cian/newbuilding.py's sister-state
extraction. Confirmed via live re-fetch after the fix: extract_all_states() now
returns 6 real MFE containers instead of an empty dict.

No test file previously existed for cian_state_parser.py (a stale .pyc from a
pre-scraper_kit-migration test was the only trace) — added tests/test_cian_state_parser.py
covering concat discovery, legacy push fallback, mixed-format precedence, and the
extract_state()/extract_all_states() consistency invariant that the bug violated.

Full backend suite: 2345 passed, 6 skipped, 1 pre-existing unrelated failure
(test_search_cache_hit, verified via git stash against clean main).

Refs #2435