lekss361/gendesign

Fork 0

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) #450

Merged

lekss361 merged 1 commit from feat/tradein-cian-serp-refactor into main

2026-05-23 12:52:04 +00:00

lekss361 commented

2026-05-23 12:48:18 +00:00

Owner

Summary

Stage 3 / Wave 3 of CianScraper v1. Refactors existing DOM + __NEXT_DATA__ parser в state-based extraction через newly-merged cian_state_parser (PR #447). Extracts 137 fields per Schema doc sec 3.

Diff

cian.py: 484 → 476 lines (full rewrite, +457/-335 net per diff stat)
tests/test_cian_serp_scraper.py: new (458 lines, 21 tests)

Key changes

Aspect	Before	After
State extraction	DOM scrape via selectolax + `__NEXT_DATA__` JSON	`extract_state(html, 'frontend-serp', 'initialState')`
Subdomain	`ekaterinburg.cian.ru`	`ekb.cian.ru` (per Schema sec 13 closed Q4)
Coords	Anchor jitter / approximate	Exact: `offer.geo.coordinates.{lat,lng}`
Anti-bot	`curl_cffi` chrome120	PRESERVED — same TLS fingerprint bypass
Fields extracted	Basic (~30)	137 per Schema sec 3

New fields populated в ScrapedLot

Geo: lat/lon (exact, no jitter), metro_stations[]
Building: year_built, materialType, floorsCount, deadline.year
Cian-specific (from PR #447 ScrapedLot ext): windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, living_area_m2, bedrooms_count, balconies_count, loggias_count, description_minhash, cadastral_number, building_cadastral_number, phones[], is_homeowner, is_pro_seller, bargain_allowed, sale_type, house_source, house_ext_id

Tests

21 new tests in test_cian_serp_scraper.py (all sync, no network — synthesized Cian state strings)
Full suite: 71/71 passed
Ruff: clean

Deviations from spec

kitchen_area_m2 — not in ScrapedLot (Stage 2 PR #447 didn't add it; base.py forbidden to touch from this PR). Stored in raw_payload["kitchen_area_m2"]. Future minor PR can promote to top-level if Stage 8 matching needs it.
Tests sync (not async) — _parse_serp_html() is sync method, no @pytest.mark.asyncio needed.

Anti-bot preserved

✅ curl_cffi.AsyncSession(impersonate="chrome120") untouched. Cian blocks plain httpx (TLS fingerprint detection → 403 "Подозрительный трафик").

Next stages unblocked (Wave 4 parallel)

Stage 4 — cian_session.py (cookies management)
Stage 5 — cian_detail.py (defaultState, mfe='frontend-offer-card')
Stage 6 — cian_newbuilding.py (initialState, mfe='newbuilding-card-desktop-fichering-frontend')

Все 3 могут запускаться parallel через isolation:worktree после merge этого PR.

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 3
decisions/Schema_Cian_SERP_Inventory.md sec 1-3 (state structure, 137 fields), sec 11 (schema mapping), sec 13 (subdomain decision)
PR #447 (cian_state_parser shared utility — merged)
PR #448 (Migration 029 — extended schema cols used here)

Test plan

21 new tests pass
Full backend test suite green
Ruff clean
Post-merge: live smoke на ЕКБ anchor → 28+ lots с точными coords, no 403

## Summary **Stage 3 / Wave 3** of CianScraper v1. Refactors existing DOM + `__NEXT_DATA__` parser в state-based extraction через newly-merged `cian_state_parser` (PR #447). Extracts 137 fields per Schema doc sec 3. ## Diff - `cian.py`: 484 → 476 lines (full rewrite, +457/-335 net per diff stat) - `tests/test_cian_serp_scraper.py`: new (458 lines, 21 tests) ## Key changes | Aspect | Before | After | |---|---|---| | State extraction | DOM scrape via selectolax + `__NEXT_DATA__` JSON | `extract_state(html, 'frontend-serp', 'initialState')` | | Subdomain | `ekaterinburg.cian.ru` | `ekb.cian.ru` (per Schema sec 13 closed Q4) | | Coords | Anchor jitter / approximate | Exact: `offer.geo.coordinates.{lat,lng}` | | Anti-bot | `curl_cffi` chrome120 | **PRESERVED** — same TLS fingerprint bypass | | Fields extracted | Basic (~30) | **137** per Schema sec 3 | ## New fields populated в ScrapedLot - Geo: lat/lon (exact, no jitter), metro_stations[] - Building: year_built, materialType, floorsCount, deadline.year - Cian-specific (from PR #447 ScrapedLot ext): windows_view_type, separate/combined_wcs_count, ceiling_height, repair_type, living_area_m2, bedrooms_count, balconies_count, loggias_count, description_minhash, cadastral_number, building_cadastral_number, phones[], is_homeowner, is_pro_seller, bargain_allowed, sale_type, house_source, house_ext_id ## Tests - 21 new tests in `test_cian_serp_scraper.py` (all sync, no network — synthesized Cian state strings) - Full suite: 71/71 passed - Ruff: clean ## Deviations from spec 1. **`kitchen_area_m2`** — not in `ScrapedLot` (Stage 2 PR #447 didn't add it; base.py forbidden to touch from this PR). Stored in `raw_payload["kitchen_area_m2"]`. Future minor PR can promote to top-level if Stage 8 matching needs it. 2. Tests sync (not async) — `_parse_serp_html()` is sync method, no `@pytest.mark.asyncio` needed. ## Anti-bot preserved ✅ `curl_cffi.AsyncSession(impersonate="chrome120")` untouched. Cian blocks plain httpx (TLS fingerprint detection → 403 "Подозрительный трафик"). ## Next stages unblocked (Wave 4 parallel) - Stage 4 — cian_session.py (cookies management) - Stage 5 — cian_detail.py (defaultState, mfe='frontend-offer-card') - Stage 6 — cian_newbuilding.py (initialState, mfe='newbuilding-card-desktop-fichering-frontend') Все 3 могут запускаться parallel через isolation:worktree после merge этого PR. ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 3 - `decisions/Schema_Cian_SERP_Inventory.md` sec 1-3 (state structure, 137 fields), sec 11 (schema mapping), sec 13 (subdomain decision) - PR #447 (cian_state_parser shared utility — merged) - PR #448 (Migration 029 — extended schema cols used here) ## Test plan - [x] 21 new tests pass - [x] Full backend test suite green - [x] Ruff clean - [ ] Post-merge: live smoke на ЕКБ anchor → 28+ lots с точными coords, no 403

lekss361 added 1 commit 2026-05-23 12:48:19 +00:00

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) b22ea0da80

Stage 3 of CianScraper v1 plan. Replaces DOM+__NEXT_DATA__ parsing with
cian_state_parser.extract_state(mfe='frontend-serp', key='initialState').
No anchor jitter — precise coords from offer.geo.coordinates.{lat,lng}.
Subdomain: ekb.cian.ru (per Schema sec 13, closed Q4).
curl_cffi anti-bot preserved (chrome120 TLS fingerprint).

Fields mapped: identity, geo, area, building, cadastral (flat+house),
bargainTerms, newbuilding link, seller (phones/is_homeowner/is_pro),
description_minhash (Cian native + SHA1 fallback), metro_stations,
balconies/loggias/bedrooms, listing_segment (vtorichka/novostroyki).
kitchen_area_m2 stored in raw_payload (not in ScrapedLot Stage 2).

Tests: 21 new unit tests in test_cian_serp_scraper.py (all passing).
Full suite: 71/71 passed.

lekss361 merged commit 4f0319297e into main

2026-05-23 12:52:04 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 12:52:05 +00:00

feat(tradein): cian.py SERP refactor — state-based extraction (137 fields, no jitter) (#450)

lekss361 commented

2026-05-23 12:52:48 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE (with one MEDIUM follow-up noted below).

Deep review summary

Verified:

Subdomain switch ekaterinburg.cian.ru → ekb.cian.ru consistent — production cron-scrape.sh calls CianScraper class (not URL strings), zero hardcoded references remain in tradein-mvp/backend/app/**. Only stale ref is tradein-mvp/docs/ROADMAP.md:68 (doc-only, non-blocking).
Dedup safe across subdomain switch: base.py:121 compute_dedup_hash uses source_id (cianId), NOT source_url — existing rows with old subdomain keep the same hash and will UPDATE, not duplicate.
curl_cffi.AsyncSession(impersonate="chrome120") preserved (anti-bot bypass intact).
21 tests cover state-extract, no-jitter coords, kitchen_area in raw_payload, minhash fallback, newbuilding vs vtorichka segmentation, cadastral numbers, address formatter, missing-state graceful empty list, missing-coords no-crash.
Old DOM/__NEXT_DATA__ parser fully removed; no orphan tests on deleted code (only test_cian_state_parser.py from PR #447 + new test_cian_serp_scraper.py).
_parse_serp_html() pure-sync, fixture-testable; fetch_around() async, network-isolated.
nb_id == 0 sentinel correctly maps to vtorichka per Schema sec 20.5.

MEDIUM follow-up — save_listings persistence gap (Stage 8 prerequisite)

base.py save_listings INSERT (lines 215–276) does NOT reference the new Cian-specific columns added to ScrapedLot (PR #447) or listings DDL (SQL 019):

living_area_m2, bedrooms_count, balconies_count, loggias_count
description_minhash, cadastral_number, building_cadastral_number
phones, is_homeowner, is_pro_seller, bargain_allowed, sale_type
metro_stations

Per intentional scope (base.py forbidden to touch this PR), but consequence:

All ~12 Stage-2 model fields populated by _offer_to_lot are silently dropped at INSERT.
Also NOT stored in raw_payload jsonb (raw_payload only carries cian_id, flat_type, kitchen_area_m2, mortgage flags, etc.).
cadastral_number column stays NULL (legacy kadastr_num is populated as before from same value).

Impact: Stage 8 cross-source matching that depends on description_minhash / cadastral_number / building_cadastral_number cannot use these signals until INSERT is extended. Cross-PR plan should add a Stage ~3.5 or ~7.5 base.py PR before Stage 8 ships.

Other observations (LOW)

_format_address always falls back to "Екатеринбург (Cian)" — fine for ЕКБ-only stage, but hardcoded city name will be incorrect when Cian scraper expands to other regions (future Stage). Worth a TODO comment.
metro_stations[*]["mode"] uses u.get("transportType") ("walk"/"transport") — verify naming matches downstream consumer expectations (e.g. UI labels).
description_minhash SHA1[:32] fallback is fine for short-term dedup, but Cian-native LSH minhash (when present) and SHA1 hex are not compatible for similarity comparison — Stage 8 needs to branch on which kind of hash is stored. Document in Schema sec 11 mapping notes.

Post-merge verification steps

Confirm deploy-tradein.yml triggers on tradein-mvp/backend/** push to main.
CI pytest expected green (71/71).
Live smoke: after next cron-scrape cycle, run on prod DB:
```
SELECT count(*), count(lat) AS with_coords
FROM listings
WHERE source='cian' AND scraped_at > NOW() - INTERVAL '1 hour';
```
Expectation: high coord coverage (state-based extract → near-100% lat filled, vs old DOM 0%).
Sanity-check no 403 spike in tradein-backend logs (anti-bot still works).
Update docs/ROADMAP.md:68 to ekb.cian.ru in a follow-up doc PR.

Merged via deep-code-reviewer — verdict APPROVE (with one MEDIUM follow-up noted below). ## Deep review summary **Verified:** - Subdomain switch `ekaterinburg.cian.ru` → `ekb.cian.ru` consistent — production `cron-scrape.sh` calls `CianScraper` class (not URL strings), zero hardcoded references remain in `tradein-mvp/backend/app/**`. Only stale ref is `tradein-mvp/docs/ROADMAP.md:68` (doc-only, non-blocking). - Dedup safe across subdomain switch: `base.py:121` `compute_dedup_hash` uses `source_id` (cianId), NOT `source_url` — existing rows with old subdomain keep the same hash and will UPDATE, not duplicate. - `curl_cffi.AsyncSession(impersonate="chrome120")` preserved (anti-bot bypass intact). - 21 tests cover state-extract, no-jitter coords, kitchen_area in raw_payload, minhash fallback, newbuilding vs vtorichka segmentation, cadastral numbers, address formatter, missing-state graceful empty list, missing-coords no-crash. - Old DOM/`__NEXT_DATA__` parser fully removed; no orphan tests on deleted code (only `test_cian_state_parser.py` from PR #447 + new `test_cian_serp_scraper.py`). - `_parse_serp_html()` pure-sync, fixture-testable; `fetch_around()` async, network-isolated. - `nb_id == 0` sentinel correctly maps to vtorichka per Schema sec 20.5. ## MEDIUM follow-up — save_listings persistence gap (Stage 8 prerequisite) `base.py` `save_listings` INSERT (lines 215–276) does NOT reference the new Cian-specific columns added to `ScrapedLot` (PR #447) or `listings` DDL (SQL 019): - `living_area_m2`, `bedrooms_count`, `balconies_count`, `loggias_count` - `description_minhash`, `cadastral_number`, `building_cadastral_number` - `phones`, `is_homeowner`, `is_pro_seller`, `bargain_allowed`, `sale_type` - `metro_stations` Per intentional scope (`base.py forbidden to touch this PR`), but consequence: - All ~12 Stage-2 model fields populated by `_offer_to_lot` are **silently dropped at INSERT**. - Also NOT stored in `raw_payload` jsonb (raw_payload only carries `cian_id`, `flat_type`, `kitchen_area_m2`, mortgage flags, etc.). - `cadastral_number` column stays NULL (legacy `kadastr_num` is populated as before from same value). **Impact:** Stage 8 cross-source matching that depends on `description_minhash` / `cadastral_number` / `building_cadastral_number` cannot use these signals until INSERT is extended. Cross-PR plan should add a Stage ~3.5 or ~7.5 `base.py` PR before Stage 8 ships. ## Other observations (LOW) - `_format_address` always falls back to "Екатеринбург (Cian)" — fine for ЕКБ-only stage, but hardcoded city name will be incorrect when Cian scraper expands to other regions (future Stage). Worth a TODO comment. - `metro_stations[*]["mode"]` uses `u.get("transportType")` ("walk"/"transport") — verify naming matches downstream consumer expectations (e.g. UI labels). - `description_minhash` SHA1[:32] fallback is fine for short-term dedup, but Cian-native LSH minhash (when present) and SHA1 hex are not compatible for similarity comparison — Stage 8 needs to branch on which kind of hash is stored. Document in Schema sec 11 mapping notes. ## Post-merge verification steps 1. Confirm `deploy-tradein.yml` triggers on `tradein-mvp/backend/**` push to main. 2. CI pytest expected green (71/71). 3. Live smoke: after next cron-scrape cycle, run on prod DB: ```sql SELECT count(*), count(lat) AS with_coords FROM listings WHERE source='cian' AND scraped_at > NOW() - INTERVAL '1 hour'; ``` Expectation: high coord coverage (state-based extract → near-100% lat filled, vs old DOM 0%). 4. Sanity-check no 403 spike in `tradein-backend` logs (anti-bot still works). 5. Update `docs/ROADMAP.md:68` to `ekb.cian.ru` in a follow-up doc PR.

lekss361 referenced this pull request

2026-05-23 12:55:49 +00:00

feat(tradein): scrape_pipeline.py — Avito full orchestrator (search → houses → detail) #451

lekss361 referenced this pull request from a commit

2026-05-23 12:58:13 +00:00

feat(tradein): extend save_listings INSERT to persist Cian-specific columns (PR #450 follow-up)

lekss361 referenced this pull request

2026-05-23 12:59:37 +00:00

feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) #455

lekss361 referenced this pull request from a commit

2026-05-23 13:05:22 +00:00

feat(tradein): extend base.py save_listings to persist Cian-specific columns (PR #450 follow-up) (#455)