feat(tradein): cian_detail.py — detail page scraper (88 fields + BTI + priceChanges, Wave 4 Worker B retry) #463

Closed

lekss361 wants to merge 1 commit from feat/tradein-cian-detail into main

lekss361 commented

2026-05-23 13:24:22 +00:00

Owner

Summary

Stage 5 / Wave 4 of CianScraper v1 — RETRY of previously-failed Worker B (#458 closed: push race condition stranded branch on Avito IMV merge commit). Same scope, new branch name (feat/tradein-cian-detail без "-parser" суффикса).

Detail page scraper — extracts 88+ offer fields + sister state containers (BTI / priceChanges / stats / agent / newObject).

Push verification

	SHA
LOCAL	`f0e1f7fb1662b1c334019c4a1687879fa415bef2`
REMOTE	`f0e1f7fb1662b1c334019c4a1687879fa415bef2`
main	`d28bf4e66a04905b8fb30ae68a49ec655caae00b`

LOCAL == REMOTE ✅, LOCAL != main ✅ — independent commit confirmed.

Files

app/services/scrapers/cian_detail.py (NEW) — fetch_detail() + save_detail_enrichment() + DetailEnrichment dataclass
tests/test_cian_detail.py (NEW) — 11 tests

State extraction

MFE: frontend-offer-card
Key: defaultState (⚠️ NOT initialState — SERP uses initialState)
Uses merged cian_state_parser.extract_state() + extract_all_states() (PR #447)

Fields extracted

Category	Fields
Core	cian_id, windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type
Stats	views_total, views_today (parse Cian's "1 234" format)
BTI	bti_data → `bti.houseData` sister container
Price history	price_changes[] (offer.priceChanges or state.priceChanges)
Agent	agent_profile (offer.agent or offer.user)
Newbuilding link	newbuilding_data (offer.newbuilding.id)
Raw debug	raw_offer + raw_sister_states

Persistence (`save_detail_enrichment`)

UPDATE listings SET windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, views_total/today (all cols from migration 019)
INSERT INTO offer_price_history ... ON CONFLICT DO NOTHING
UPSERT agents (ON CONFLICT ext_source, ext_agent_id)

Tests

11/11 pass. Coverage: unit helpers (parse_views/parse_float/extract_price_changes/extract_agent_profile) + dataclass defaults + async mocked fetch_detail (no-state graceful, 403, core fields parsing).

Ruff

Clean.

Anti-bot

curl_cffi.AsyncSession(impersonate="chrome120") — consistent с cian.py SERP / Stage 4/6.

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 5
decisions/Schema_Cian_SERP_Inventory.md sec 14, 20, 20.3
Replaces closed PR #458 (empty due to push race)

Test plan

11 tests pass
Ruff clean
Push verified (LOCAL==REMOTE, ≠ main)
Post-deploy live smoke: top-5 ЕКБ offer_url → bti_data filled, price_changes non-empty

## Summary **Stage 5 / Wave 4** of CianScraper v1 — RETRY of previously-failed Worker B (#458 closed: push race condition stranded branch on Avito IMV merge commit). Same scope, **new branch name** (`feat/tradein-cian-detail` без "-parser" суффикса). Detail page scraper — extracts 88+ offer fields + sister state containers (BTI / priceChanges / stats / agent / newObject). ## Push verification | | SHA | |---|---| | LOCAL | `f0e1f7fb1662b1c334019c4a1687879fa415bef2` | | REMOTE | `f0e1f7fb1662b1c334019c4a1687879fa415bef2` | | main | `d28bf4e66a04905b8fb30ae68a49ec655caae00b` | LOCAL == REMOTE ✅, LOCAL != main ✅ — independent commit confirmed. ## Files - `app/services/scrapers/cian_detail.py` (NEW) — `fetch_detail()` + `save_detail_enrichment()` + `DetailEnrichment` dataclass - `tests/test_cian_detail.py` (NEW) — 11 tests ## State extraction - **MFE**: `frontend-offer-card` - **Key**: `defaultState` (⚠️ NOT initialState — SERP uses initialState) - Uses merged `cian_state_parser.extract_state()` + `extract_all_states()` (PR #447) ## Fields extracted | Category | Fields | |---|---| | Core | cian_id, windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type | | Stats | views_total, views_today (parse Cian's "1 234" format) | | BTI | bti_data → `bti.houseData` sister container | | Price history | price_changes[] (offer.priceChanges or state.priceChanges) | | Agent | agent_profile (offer.agent or offer.user) | | Newbuilding link | newbuilding_data (offer.newbuilding.id) | | Raw debug | raw_offer + raw_sister_states | ## Persistence (`save_detail_enrichment`) 1. `UPDATE listings SET windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, views_total/today` (all cols from migration 019) 2. `INSERT INTO offer_price_history ... ON CONFLICT DO NOTHING` 3. UPSERT `agents` (ON CONFLICT ext_source, ext_agent_id) ## Tests 11/11 pass. Coverage: unit helpers (parse_views/parse_float/extract_price_changes/extract_agent_profile) + dataclass defaults + async mocked fetch_detail (no-state graceful, 403, core fields parsing). ## Ruff Clean. ## Anti-bot `curl_cffi.AsyncSession(impersonate="chrome120")` — consistent с cian.py SERP / Stage 4/6. ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 5 - `decisions/Schema_Cian_SERP_Inventory.md` sec 14, 20, 20.3 - Replaces closed PR #458 (empty due to push race) ## Test plan - [x] 11 tests pass - [x] Ruff clean - [x] Push verified (LOCAL==REMOTE, ≠ main) - [ ] Post-deploy live smoke: top-5 ЕКБ offer_url → bti_data filled, price_changes non-empty

lekss361 added 1 commit 2026-05-23 13:24:22 +00:00

feat(tradein): cian_detail.py — detail page scraper (88 fields + BTI + priceChanges, Wave 4 Worker B retry) f0e1f7fb16

lekss361 closed this pull request

2026-05-23 13:25:59 +00:00

lekss361 commented

2026-05-23 13:26:02 +00:00

Author

Owner

Closing — cian_detail.py уже в main через commit 2dd76ea (PR #457 Stage 4 cookies). Stage 4 worker's worktree подобрала Stage 5 файлы — worktree pollution. Main version (324 lines) более полная чем эта retry (210 lines, abbreviated spec) — has agent_id_fk linking + richer logging. No action needed. Stage 5 effectively merged.

Closing — `cian_detail.py` уже в main через commit 2dd76ea (PR #457 Stage 4 cookies). Stage 4 worker's worktree подобрала Stage 5 файлы — worktree pollution. Main version (324 lines) более полная чем эта retry (210 lines, abbreviated spec) — has `agent_id_fk` linking + richer logging. No action needed. Stage 5 effectively merged.

lekss361 referenced this pull request

2026-05-23 14:23:50 +00:00

feat(tradein): Phase 1.3 backfill scripts — fingerprints + house/listing_sources #475