feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields #447

Merged

lekss361 merged 1 commit from feat/tradein-cian-state-parser into main

2026-05-23 12:35:02 +00:00

lekss361 commented

2026-05-23 12:29:13 +00:00

Owner

Summary

Stage 2 / Wave 2 of CianScraper v1 — shared utility для извлечения Cian Redux state из _cianConfig + extension ScrapedLot model.

Files

NEW tradein-mvp/backend/app/services/scrapers/cian_state_parser.py (76 lines)
- extract_state(html, mfe, key) — regex-based extractor для window._cianConfig['mfe'].push({key, value, ...})
- 3 parse strategies: direct JSON → JSON.parse("…") form → demjson3 raw JS object fallback (optional dep)
- extract_all_states(html) — discovery helper
MODIFIED tradein-mvp/backend/app/services/scrapers/base.py
- Extended ScrapedLot with 13 Cian-specific fields:
  living_area_m2, bedrooms_count, balconies_count, loggias_count, description_minhash, cadastral_number, building_cadastral_number, phones, is_homeowner, is_pro_seller, bargain_allowed, sale_type, metro_stations
- Deviation: house_source / house_ext_id уже существовали в model (lines 81-82) — не дублируем, 13 fields добавлено вместо 15
NEW tradein-mvp/backend/tests/test_cian_state_parser.py — 4 unit tests covering all parse strategies + missing/mismatch cases

Used by (downstream)

Stage 3 — cian.py SERP refactor (extract_state(html, 'frontend-serp', 'initialState'))
Stage 5 — cian_detail.py (mfe='frontend-offer-card', key='defaultState')
Stage 6 — cian_newbuilding.py (mfe='newbuilding-card-desktop-fichering-frontend')
Stage 7 — cian_valuation.py (mfe='valuation-for-agent-frontend')

Test plan

Ruff clean on new file
4/4 tests pass
Verify ScrapedLot extension doesn't break existing Avito/DomRF scrapers (CI)

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 2
decisions/Schema_Cian_SERP_Inventory.md (sec 1-3 SERP state, sec 14 detail, sec 18 newbuilding)
PR #445 (Stage 1 SQL migrations merged)

## Summary Stage 2 / Wave 2 of CianScraper v1 — shared utility для извлечения Cian Redux state из `_cianConfig` + extension `ScrapedLot` model. ## Files 1. **NEW** `tradein-mvp/backend/app/services/scrapers/cian_state_parser.py` (76 lines) - `extract_state(html, mfe, key)` — regex-based extractor для `window._cianConfig['mfe'].push({key, value, ...})` - 3 parse strategies: direct JSON → `JSON.parse("…")` form → `demjson3` raw JS object fallback (optional dep) - `extract_all_states(html)` — discovery helper 2. **MODIFIED** `tradein-mvp/backend/app/services/scrapers/base.py` - Extended `ScrapedLot` with 13 Cian-specific fields: `living_area_m2, bedrooms_count, balconies_count, loggias_count, description_minhash, cadastral_number, building_cadastral_number, phones, is_homeowner, is_pro_seller, bargain_allowed, sale_type, metro_stations` - **Deviation**: `house_source` / `house_ext_id` уже существовали в model (lines 81-82) — не дублируем, 13 fields добавлено вместо 15 3. **NEW** `tradein-mvp/backend/tests/test_cian_state_parser.py` — 4 unit tests covering all parse strategies + missing/mismatch cases ## Used by (downstream) - Stage 3 — `cian.py` SERP refactor (`extract_state(html, 'frontend-serp', 'initialState')`) - Stage 5 — `cian_detail.py` (`mfe='frontend-offer-card'`, `key='defaultState'`) - Stage 6 — `cian_newbuilding.py` (`mfe='newbuilding-card-desktop-fichering-frontend'`) - Stage 7 — `cian_valuation.py` (`mfe='valuation-for-agent-frontend'`) ## Test plan - [x] Ruff clean on new file - [x] 4/4 tests pass - [ ] Verify ScrapedLot extension doesn't break existing Avito/DomRF scrapers (CI) ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 2 - `decisions/Schema_Cian_SERP_Inventory.md` (sec 1-3 SERP state, sec 14 detail, sec 18 newbuilding) - PR #445 (Stage 1 SQL migrations merged)

lekss361 added 1 commit 2026-05-23 12:29:13 +00:00

feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields 7f03a5b634

- Add cian_state_parser.py: regex-based extractor for window._cianConfig push() entries,
  supports direct JSON, JSON.parse() escaped form, and optional demjson3 fallback
- Extend ScrapedLot in base.py with 13 Cian-specific fields (living_area_m2,
  bedrooms_count, balconies_count, loggias_count, description_minhash,
  cadastral_number, building_cadastral_number, phones, is_homeowner, is_pro_seller,
  bargain_allowed, sale_type, metro_stations)
- Add tests/test_cian_state_parser.py with 4 unit tests covering all parse strategies

lekss361 merged commit 9ba776f2ca into main

2026-05-23 12:35:02 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 12:35:02 +00:00

feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447)

lekss361 commented

2026-05-23 12:35:24 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Verification summary:

ScrapedLot backward compat: all 13 new fields Optional with None / default_factory; all existing callers (avito.py:190, cian.py:232/363, yandex_realty.py:211, n1.py:182) use kwargs — no breakage
house_source / house_ext_id NOT duplicated (verified base.py:81-82 preserved)
demjson3 graceful degradation OK (ImportError caught → debug log → continue); not in pyproject — Strategies 1 & 2 cover documented Cian shapes
save_listings INSERT intentionally NOT updated (Stage 2 scope = model only; Stages 3-7 will wire persistence with scrapers)
Recovery clean: head 7f03a5b contains exactly cian_state_parser.py + base.py + test, no avito_houses.py contamination
Cross-PR: #449 (avito-houses-parser) built on top of 7f03a5b → will rebase cleanly post-merge

Non-blocking nits (address in Stage 3+):

JS string unescape incomplete (missing \uXXXX, \t, \r) — real Cian payloads may break Strategy 2. Consider json.loads('"' + escaped + '"')
phones / metro_stations typed as list[dict] — promote to TypedDict for type safety
Stages 3-7: remember to extend save_listings INSERT/UPDATE clause for new Cian columns when scrapers start populating them

Merged via deep-code-reviewer — verdict APPROVE. **Verification summary**: - ScrapedLot backward compat: all 13 new fields Optional with None / default_factory; all existing callers (avito.py:190, cian.py:232/363, yandex_realty.py:211, n1.py:182) use kwargs — no breakage - house_source / house_ext_id NOT duplicated (verified base.py:81-82 preserved) - demjson3 graceful degradation OK (ImportError caught → debug log → continue); not in pyproject — Strategies 1 & 2 cover documented Cian shapes - save_listings INSERT intentionally NOT updated (Stage 2 scope = model only; Stages 3-7 will wire persistence with scrapers) - Recovery clean: head 7f03a5b contains exactly cian_state_parser.py + base.py + test, no avito_houses.py contamination - Cross-PR: #449 (avito-houses-parser) built on top of 7f03a5b → will rebase cleanly post-merge **Non-blocking nits** (address in Stage 3+): - JS string unescape incomplete (missing \uXXXX, \t, \r) — real Cian payloads may break Strategy 2. Consider `json.loads('"' + escaped + '"')` - `phones` / `metro_stations` typed as `list[dict]` — promote to TypedDict for type safety - Stages 3-7: remember to extend `save_listings` INSERT/UPDATE clause for new Cian columns when scrapers start populating them

lekss361 referenced this pull request

2026-05-23 12:38:03 +00:00

feat(tradein): avito_houses.py — Houses Catalog parser (state + reviews + history + recs) #449

lekss361 referenced this pull request

2026-05-23 12:48:18 +00:00

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) #450

lekss361 referenced this pull request

2026-05-23 12:52:48 +00:00

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) #450

lekss361 referenced this pull request from a commit

2026-05-23 12:58:13 +00:00

feat(tradein): extend save_listings INSERT to persist Cian-specific columns (PR #450 follow-up)