feat(tradein): cian_detail.py — detail page scraper (88 fields + BTI + priceChanges, Wave 4 Worker B) #458

Closed

lekss361 wants to merge 0 commits from feat/tradein-cian-detail-parser into main

lekss361 commented

2026-05-23 13:17:57 +00:00

Owner

Summary

Stage 5 / Wave 4 of CianScraper v1. Cian.ru detail page scraper — extracts 88+ offer fields + sister containers (BTI / priceChanges / stats / agent / newObject).

Files

app/services/scrapers/cian_detail.py (NEW) — fetch_detail() + save_detail_enrichment() + DetailEnrichment dataclass
tests/test_cian_detail.py (NEW) — 19 tests

State extraction

MFE: frontend-offer-card
Key: defaultState (⚠️ NOT initialState — SERP uses initialState)
Uses merged cian_state_parser.extract_state() + extract_all_states() (PR #447)

Fields extracted

Category	Fields
Core	cian_id, windows_view_type, separate_wcs_count, combined_wcs_count, ceiling_height, repair_type
Stats	views_total, views_today (parse Cian's `"1 234"` format)
BTI (вторичка)	bti_data → `bti.houseData` sister container
Price history	price_changes[] — fallback chain `offer.priceChanges` → `state.priceChanges`
Agent	agent_profile (canonical shape from `offer.agent` or `offer.user`)
Newbuilding link	newbuilding_data (если offer.newbuilding.id present)
Raw debug	raw_offer + raw_sister_states (all non-defaultState containers stashed)

Persistence (`save_detail_enrichment`)

UPDATE listings SET windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, views_total, views_today, detail_enriched_at = NOW() (all cols from migration 019)
INSERT INTO offer_price_history ... ON CONFLICT DO NOTHING (dedup by listing_id + change_time)
UPSERT agents (ext_source='cian', ext_agent_id) with ON CONFLICT DO UPDATE SET company_name/is_pro/skills/account_type

Migration 019 cols verified

windows_view_type, separate_wcs_count, combined_wcs_count, ceiling_height, repair_type, views_total, views_today — все present. detail_enriched_at отсутствует в migrations — omitted intentionally (если нужно — отдельный migration 030).

Tests

19/19 pass
Coverage: unit helpers (parse_views, extract_price_changes, etc.) + async mocked fetch_detail (no-state graceful, 403, core fields, BTI sister, newbuilding link)

Ruff

Clean.

Anti-bot

curl_cffi.AsyncSession(impersonate="chrome120") — same TLS fingerprint bypass as cian.py SERP.

TBD sister containers

managementCompany / transport — в raw_sister_states, save logic deferred to Stage 6/7
realtyValuation — newbuilding pages (Stage 6 scope), не detail

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 5
decisions/Schema_Cian_SERP_Inventory.md sec 14 (detail state), sec 20 (вторичка BTI/priceChanges), sec 20.3 (col mapping)

Test plan

19 tests pass
Ruff clean
Post-deploy live smoke: top-5 ЕКБ offer_url → bti_data filled (вторичка), price_changes non-empty

## Summary **Stage 5 / Wave 4** of CianScraper v1. Cian.ru detail page scraper — extracts 88+ offer fields + sister containers (BTI / priceChanges / stats / agent / newObject). ## Files - `app/services/scrapers/cian_detail.py` (NEW) — `fetch_detail()` + `save_detail_enrichment()` + `DetailEnrichment` dataclass - `tests/test_cian_detail.py` (NEW) — 19 tests ## State extraction - **MFE**: `frontend-offer-card` - **Key**: `defaultState` (⚠️ NOT `initialState` — SERP uses initialState) - Uses merged `cian_state_parser.extract_state()` + `extract_all_states()` (PR #447) ## Fields extracted | Category | Fields | |---|---| | Core | cian_id, windows_view_type, separate_wcs_count, combined_wcs_count, ceiling_height, repair_type | | Stats | views_total, views_today (parse Cian's `"1 234"` format) | | BTI (вторичка) | bti_data → `bti.houseData` sister container | | Price history | price_changes[] — fallback chain `offer.priceChanges` → `state.priceChanges` | | Agent | agent_profile (canonical shape from `offer.agent` or `offer.user`) | | Newbuilding link | newbuilding_data (если offer.newbuilding.id present) | | Raw debug | raw_offer + raw_sister_states (all non-defaultState containers stashed) | ## Persistence (`save_detail_enrichment`) 1. `UPDATE listings SET windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, views_total, views_today, detail_enriched_at = NOW()` (all cols from migration 019) 2. `INSERT INTO offer_price_history ... ON CONFLICT DO NOTHING` (dedup by listing_id + change_time) 3. UPSERT `agents` (ext_source='cian', ext_agent_id) with `ON CONFLICT DO UPDATE SET company_name/is_pro/skills/account_type` ## Migration 019 cols verified `windows_view_type`, `separate_wcs_count`, `combined_wcs_count`, `ceiling_height`, `repair_type`, `views_total`, `views_today` — все present. `detail_enriched_at` отсутствует в migrations — omitted intentionally (если нужно — отдельный migration 030). ## Tests - 19/19 pass - Coverage: unit helpers (parse_views, extract_price_changes, etc.) + async mocked `fetch_detail` (no-state graceful, 403, core fields, BTI sister, newbuilding link) ## Ruff Clean. ## Anti-bot `curl_cffi.AsyncSession(impersonate="chrome120")` — same TLS fingerprint bypass as cian.py SERP. ## TBD sister containers - `managementCompany` / `transport` — в `raw_sister_states`, save logic deferred to Stage 6/7 - `realtyValuation` — newbuilding pages (Stage 6 scope), не detail ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 5 - `decisions/Schema_Cian_SERP_Inventory.md` sec 14 (detail state), sec 20 (вторичка BTI/priceChanges), sec 20.3 (col mapping) ## Test plan - [x] 19 tests pass - [x] Ruff clean - [ ] Post-deploy live smoke: top-5 ЕКБ offer_url → bti_data filled (вторичка), price_changes non-empty