lekss361/gendesign

Fork 0

feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache #465

Merged

lekss361 merged 1 commit from feat/tradein-cian-valuation into main

2026-05-23 13:43:34 +00:00

lekss361 commented

2026-05-23 13:38:27 +00:00

Owner

Summary

Stage 7 / Wave 5 of CianScraper v1. Cian.ru authenticated Valuation Calculator scraper — 6-й evaluation source для estimator (Stage 9).

Depends Stage 4 (cookies via app.services.cian_session.load_session) ✅ merged.

Files

app/services/scrapers/cian_valuation.py (NEW) — estimate_via_cian_valuation() + cache helpers + dataclass
tests/test_cian_valuation.py (NEW) — 28 tests

Push verify

	SHA
LOCAL	`6d99e6a41aef34554b775e0df26adfc98ceaab17`
REMOTE	`6d99e6a41aef34554b775e0df26adfc98ceaab17`

Match ✅.

State extraction

URL: https://www.cian.ru/kalkulator-nedvizhimosti/?address=...&totalArea=...
MFE: valuation-for-agent-frontend
Key: initialState
Auth: cookies via load_session(db) (Stage 4)
На isAuthenticated=false → mark_session_invalid() + return None gracefully

Real Cian state shape (verified)

estimation.sale.data.price (NOT valuation.sale.price)
estimationChart.data.chartData.data (unix ms → ISO date)
estimationChart.data.title.changeValue ("8,3%" → 8.3) — _parse_change_pct() handles comma decimal
estimation.sale.data.filtersHash also saved (dedup helper)
chart_change_direction: Cian's native increase/decrease/neutral

Cache (`external_valuations`)

24h TTL по cache_key = sha256(address + totalArea + rooms + floor + repair_type + deal_type).

Schema cols verified (migrations 026 + 029):

Base: source, cache_key, address, total_area, rooms_count, floor, total_floors, repair_type, deal_type, price_rub, accuracy, raw_payload, chart, fetched_at, expires_at
029 extension: sale_price_rub, sale_accuracy, rent_price_rub, rent_accuracy, chart_change_pct, chart_change_direction, filters_hash, external_house_id

ON CONFLICT (source, cache_key) DO UPDATE — re-fetch refreshes cache.

Auth flow

load_session(db) → cookies dict or None
If None → log warning + return None (graceful degradation, estimator continues without Cian)
Fetch с cookies via curl_cffi(impersonate="chrome120")
Parse state — verify state.user.isAuthenticated
If false → mark_session_invalid() + return None (forces re-upload prompt)

Tests

28/28 pass. Coverage:

cache_key stability (lowercase + strip normalization)
num parsing edge cases
valuation state shape (sale/rent split, chart, change direction up/down/flat)
async fetch (no cookies / cache hit / auth fail paths)

Ruff

Clean (auto-fixed UP017 datetime.UTC + import sort).

Anti-bot

curl_cffi.AsyncSession(impersonate="chrome120") — consistent across cian.py / cian_detail.py / cian_newbuilding.py.

Next: Stage 8 (matching) + Stage 9 (estimator integration)

Stage 8 (cross-source matching) и Stage 9 (estimator 6th source) теперь unblocked.

Refs

decisions/CianScraper_v1_Implementation_Plan.md Stage 7
decisions/Schema_Cian_SERP_Inventory.md sec 24 (URL/auth), sec 25 (response shape), sec 26 (cache DDL)
PR #457 (Stage 4 cookies — load_session API)
PR #448 (Migration 029 — external_valuations extension)

Test plan

28 unit tests pass
Ruff clean
Push verified
Post-deploy: upload cookies via /admin/scrape/cian/upload-cookies → POST /estimate с ЕКБ address → sources_used includes cian_valuation

## Summary **Stage 7 / Wave 5** of CianScraper v1. Cian.ru authenticated Valuation Calculator scraper — 6-й evaluation source для estimator (Stage 9). Depends Stage 4 (cookies via `app.services.cian_session.load_session`) ✅ merged. ## Files - `app/services/scrapers/cian_valuation.py` (NEW) — `estimate_via_cian_valuation()` + cache helpers + dataclass - `tests/test_cian_valuation.py` (NEW) — 28 tests ## Push verify | | SHA | |---|---| | LOCAL | `6d99e6a41aef34554b775e0df26adfc98ceaab17` | | REMOTE | `6d99e6a41aef34554b775e0df26adfc98ceaab17` | Match ✅. ## State extraction - **URL**: `https://www.cian.ru/kalkulator-nedvizhimosti/?address=...&totalArea=...` - **MFE**: `valuation-for-agent-frontend` - **Key**: `initialState` - Auth: cookies via `load_session(db)` (Stage 4) - На `isAuthenticated=false` → `mark_session_invalid()` + return None gracefully ## Real Cian state shape (verified) - `estimation.sale.data.price` (NOT `valuation.sale.price`) - `estimationChart.data.chartData.data` (unix ms → ISO date) - `estimationChart.data.title.changeValue` ("8,3%" → 8.3) — `_parse_change_pct()` handles comma decimal - `estimation.sale.data.filtersHash` also saved (dedup helper) - `chart_change_direction`: Cian's native `increase`/`decrease`/`neutral` ## Cache (`external_valuations`) 24h TTL по `cache_key = sha256(address + totalArea + rooms + floor + repair_type + deal_type)`. Schema cols verified (migrations 026 + 029): - Base: source, cache_key, address, total_area, rooms_count, floor, total_floors, repair_type, deal_type, price_rub, accuracy, raw_payload, chart, fetched_at, expires_at - 029 extension: sale_price_rub, sale_accuracy, rent_price_rub, rent_accuracy, chart_change_pct, chart_change_direction, filters_hash, external_house_id ON CONFLICT (source, cache_key) DO UPDATE — re-fetch refreshes cache. ## Auth flow 1. `load_session(db)` → cookies dict or None 2. If None → log warning + return None (graceful degradation, estimator continues without Cian) 3. Fetch с cookies via `curl_cffi(impersonate="chrome120")` 4. Parse state — verify `state.user.isAuthenticated` 5. If false → `mark_session_invalid()` + return None (forces re-upload prompt) ## Tests 28/28 pass. Coverage: - cache_key stability (lowercase + strip normalization) - num parsing edge cases - valuation state shape (sale/rent split, chart, change direction up/down/flat) - async fetch (no cookies / cache hit / auth fail paths) ## Ruff Clean (auto-fixed UP017 `datetime.UTC` + import sort). ## Anti-bot `curl_cffi.AsyncSession(impersonate="chrome120")` — consistent across cian.py / cian_detail.py / cian_newbuilding.py. ## Next: Stage 8 (matching) + Stage 9 (estimator integration) Stage 8 (cross-source matching) и Stage 9 (estimator 6th source) теперь unblocked. ## Refs - `decisions/CianScraper_v1_Implementation_Plan.md` Stage 7 - `decisions/Schema_Cian_SERP_Inventory.md` sec 24 (URL/auth), sec 25 (response shape), sec 26 (cache DDL) - PR #457 (Stage 4 cookies — `load_session` API) - PR #448 (Migration 029 — `external_valuations` extension) ## Test plan - [x] 28 unit tests pass - [x] Ruff clean - [x] Push verified - [ ] Post-deploy: upload cookies via `/admin/scrape/cian/upload-cookies` → POST /estimate с ЕКБ address → `sources_used` includes `cian_valuation`

lekss361 added 1 commit 2026-05-23 13:38:28 +00:00

feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache 6d99e6a41a

Stage 7 (Wave 5). Scraper fetches https://www.cian.ru/kalkulator-nedvizhimosti/
with curl_cffi (impersonate=chrome120) + Cian session cookies from DB (Stage 4).
Parses estimation.{sale,rent}.data, estimationChart 7-point time-series,
houseInfo 15 items, managementCompany, filters.houseId.
Writes to external_valuations (026+029 schema) with 24h TTL + ON CONFLICT upsert.
28 unit tests covering cache key, state parsing, floor/repair enums, cache hit/miss.

lekss361 merged commit b0a7235110 into main

2026-05-23 13:43:34 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 13:43:35 +00:00

feat(tradein): cian_valuation.py — auth-required Calculator scraper + 24h cache (#465)

lekss361 commented

2026-05-23 13:44:09 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Critical checks verified:

UNIQUE (source, cache_key): confirmed in 026_external_valuations.sql:34 — ON CONFLICT (source, cache_key) resolves correctly, no permanent cache-miss bug.
load_session(db) signature: matches cian_session.py:135 (db: Session) -> dict[str, str] | None). Returns None gracefully on no cookies / DB error / decryption fail.
mark_session_invalid(db, user_id): signature matches cian_session.py:186. Idempotent (simple UPDATE last_invalid_at = NOW()), safe for parallel auth-fail races.
cache_key determinism: address.strip().lower() + pipe-separated params + sha256. Tests cover normalization + distinct-on-deal_type/area. Caveat: caller must pass total_area consistently as float (int 45 vs float 45.0 produce different keys) — not exercised in tests but caller-side concern.
SQL UPSERT correctness: 21 columns / 21 VALUES placeholders / 18 named params — match verified. All numeric/jsonb via CAST(:x AS type) per psycopg v3 convention. No :x::type violations.
Auth flow: cache HIT short-circuits before load_session() (saves DB hit + token check). isAuthenticated=false → mark_session_invalid() only if user_id truthy (handles missing userId edge case).
Chart parsing: unix_ms → UTC ISO date via datetime.UTC (Python 3.12+ confirmed in pyproject.toml). 7 test points cover full deserialization.
_parse_change_pct: handles "8,3%" (RU comma), "-2,5%" (negative), "1.5%" (English dot), "" (None). All 4 cases tested.
Schema cols match: 14 base (026) + 8 extension (029) = 22 cols defined; PR writes 21 (skips legacy generic price_rub/accuracy in favor of sale_*/rent_* split). Intended per #448 design.
Anti-bot: curl_cffi(impersonate="chrome120") consistent with cian.py:20, avito_houses.py:29 sibling pattern.
Tests: 28/28 documented coverage; cache_key stability + state shape + sale/rent split + chart change direction up/down/flat + 3 async integration paths (no cookies / cache hit / use_cache=False bypass).

Minor (non-blocking, optional follow-ups):

Inline import datetime at line 294 inconsistent with sibling files that use top-level from datetime import UTC. Cosmetic only.
_load_from_cache re-uses cached raw_state only if dict; logged values pass-through _parse_num even when DB returns numeric (slight overhead, no correctness issue).
Integration test for real DB UPSERT roundtrip would catch param-typing regressions earlier (currently mock-only).

Stage 8 (matching) + Stage 9 (estimator integration) unblocked.

Post-merge: deploy-tradein.yml will trigger; verify CI pytest 28/28 green, then upload cookies via /admin/scrape/cian/upload-cookies (Stage 4 endpoint) → POST /estimate с ЕКБ address → expect log Cian valuation extracted: sale=... → repeat within 24h → expect Cian valuation cache HIT.

Merge commit: b0a72351.

Merged via deep-code-reviewer — verdict APPROVE. **Critical checks verified:** - **UNIQUE (source, cache_key)**: confirmed in `026_external_valuations.sql:34` — `ON CONFLICT (source, cache_key)` resolves correctly, no permanent cache-miss bug. - **load_session(db) signature**: matches `cian_session.py:135` (`db: Session) -> dict[str, str] | None`). Returns None gracefully on no cookies / DB error / decryption fail. - **mark_session_invalid(db, user_id)**: signature matches `cian_session.py:186`. Idempotent (simple `UPDATE last_invalid_at = NOW()`), safe for parallel auth-fail races. - **cache_key determinism**: `address.strip().lower()` + pipe-separated params + sha256. Tests cover normalization + distinct-on-deal_type/area. Caveat: caller must pass `total_area` consistently as float (int 45 vs float 45.0 produce different keys) — not exercised in tests but caller-side concern. - **SQL UPSERT correctness**: 21 columns / 21 VALUES placeholders / 18 named params — match verified. All numeric/jsonb via `CAST(:x AS type)` per psycopg v3 convention. No `:x::type` violations. - **Auth flow**: cache HIT short-circuits before `load_session()` (saves DB hit + token check). `isAuthenticated=false` → `mark_session_invalid()` only if `user_id` truthy (handles missing userId edge case). - **Chart parsing**: unix_ms → UTC ISO date via `datetime.UTC` (Python 3.12+ confirmed in pyproject.toml). 7 test points cover full deserialization. - **`_parse_change_pct`**: handles `"8,3%"` (RU comma), `"-2,5%"` (negative), `"1.5%"` (English dot), `""` (None). All 4 cases tested. - **Schema cols match**: 14 base (026) + 8 extension (029) = 22 cols defined; PR writes 21 (skips legacy generic `price_rub`/`accuracy` in favor of `sale_*`/`rent_*` split). Intended per #448 design. - **Anti-bot**: `curl_cffi(impersonate="chrome120")` consistent with `cian.py:20`, `avito_houses.py:29` sibling pattern. - **Tests**: 28/28 documented coverage; cache_key stability + state shape + sale/rent split + chart change direction up/down/flat + 3 async integration paths (no cookies / cache hit / use_cache=False bypass). **Minor (non-blocking, optional follow-ups):** - Inline `import datetime` at line 294 inconsistent with sibling files that use top-level `from datetime import UTC`. Cosmetic only. - `_load_from_cache` re-uses cached `raw_state` only if dict; logged values pass-through `_parse_num` even when DB returns numeric (slight overhead, no correctness issue). - Integration test for real DB UPSERT roundtrip would catch param-typing regressions earlier (currently mock-only). **Stage 8 (matching) + Stage 9 (estimator integration) unblocked.** Post-merge: `deploy-tradein.yml` will trigger; verify CI pytest 28/28 green, then upload cookies via `/admin/scrape/cian/upload-cookies` (Stage 4 endpoint) → POST `/estimate` с ЕКБ address → expect log `Cian valuation extracted: sale=...` → repeat within 24h → expect `Cian valuation cache HIT`. Merge commit: `b0a72351`.

lekss361 referenced this pull request

2026-05-23 14:15:22 +00:00

feat(tradein): estimator — Yandex Valuation as on-demand 6th source #473

lekss361 referenced this pull request

2026-05-23 14:25:12 +00:00

feat(tradein): estimator — Cian Valuation as on-demand evaluation source (Stage 9) #476

lekss361 referenced this pull request

2026-05-23 14:30:18 +00:00

feat(tradein): estimator — Cian Valuation as on-demand evaluation source (Stage 9) #476

lekss361 referenced this pull request

2026-05-23 14:49:07 +00:00

fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) #480