fix(yandex-valuation): curl_cffi chrome120 — Yandex gates SSR on Chrome TLS #545

Merged

lekss361 merged 1 commit from fix/tradein-yandex-curl-cffi into main

2026-05-24 16:42:08 +00:00

lekss361 commented

2026-05-24 16:38:07 +00:00

Owner

Summary

YandexValuationScraper overrides __aenter__/__aexit__/_http_get to use curl_cffi.requests.AsyncSession(impersonate='chrome120')
BaseScraper untouched (other scrapers don't need Chrome TLS gate)
Same pattern as sibling yandex_realty.py + scripts/local-sweep-ekb-yandex.py

Why (live evidence)

Probe today: plain httpx / curl GET to /otsenka-kvartiry-po-adresu-onlayn/ returns 200 OK with 1.27 MB HTML — but body contains ONLY the table header row (__head). Data rows are client-side rendered. Yandex anti-bot gates SSR on Chrome TLS+JA3 fingerprint.

With curl_cffi chrome120: same URL → 21 __row occurrences, 133 __cell, 23 м², 38 dates — full SSR data.

This explains why the DOM-based parser from PR #544 returned 0 rows on re-sweep: rows simply weren't in the HTML the scraper received.

Tests

40/40 pytest pass (all existing tests are pure-parser, no network)
ruff check + format clean
curl_cffi==0.15.0 already in deps (no pyproject.toml/uv.lock changes)

Followup

After merge: purge yandex_valuation history+cache, re-sweep Базовый 52 — expected plausible area_filled ≥90% with DOM parser now having actual DOM nodes to parse.

## Summary - `YandexValuationScraper` overrides `__aenter__`/`__aexit__`/`_http_get` to use `curl_cffi.requests.AsyncSession(impersonate='chrome120')` - `BaseScraper` untouched (other scrapers don't need Chrome TLS gate) - Same pattern as sibling `yandex_realty.py` + `scripts/local-sweep-ekb-yandex.py` ## Why (live evidence) Probe today: plain `httpx` / `curl` GET to `/otsenka-kvartiry-po-adresu-onlayn/` returns 200 OK with 1.27 MB HTML — but body contains ONLY the table header row (`__head`). Data rows are client-side rendered. Yandex anti-bot gates SSR on Chrome TLS+JA3 fingerprint. With curl_cffi `chrome120`: same URL → **21 `__row` occurrences, 133 `__cell`, 23 м², 38 dates** — full SSR data. This explains why the DOM-based parser from PR #544 returned 0 rows on re-sweep: rows simply weren't in the HTML the scraper received. ## Tests - 40/40 pytest pass (all existing tests are pure-parser, no network) - ruff check + format clean - `curl_cffi==0.15.0` already in deps (no `pyproject.toml`/`uv.lock` changes) ## Followup After merge: purge yandex_valuation history+cache, re-sweep Базовый 52 — expected `plausible area_filled ≥90%` with DOM parser now having actual DOM nodes to parse.

lekss361 added 1 commit 2026-05-24 16:38:08 +00:00

fix(yandex-valuation): use curl_cffi chrome120 — Yandex gates SSR on Chrome TLS ed6e52f393

Live probe today: plain httpx GET to /otsenka-kvartiry-po-adresu-onlayn/
returns 200 OK with full HTML, but body contains only the table header row
(__head). Data rows are CSR-rendered. curl_cffi with impersonate=chrome120
gets full SSR (21 rows, 133 cells, 38 dates verified).

Override __aenter__/__aexit__/_http_get only in YandexValuationScraper —
BaseScraper untouched (other scrapers don't need Chrome TLS).

Sibling pattern: yandex_realty.py + scripts/local-sweep-ekb-yandex.py
already use curl_cffi chrome120. This brings yandex_valuation in line.

Net effect: DOM-based parser from PR #544 will actually have DOM rows to
parse. Previous re-sweep returned 0 saves because rows weren't in SSR.

lekss361 commented

2026-05-24 16:41:59 +00:00

Author

Owner

Deep Code Review — PR #545 (yandex_valuation curl_cffi chrome120)

Status: APPROVE
Files: 1 (P1 scraper) | Lines: +26 / -0 | SHA verified: ed6e52f matches head.

Verdict summary

Targeted override of __aenter__/__aexit__/_http_get on YandexValuationScraper to use curl_cffi.requests.AsyncSession(impersonate="chrome120"). BaseScraper untouched — zero blast-radius on other scrapers (Avito, Cian, Yandex realty/detail/newbuilding). Live evidence in PR description is convincing: plain-httpx 1.27 MB HTML contains only table header (__head), curl_cffi same URL returns 21 __row × 133 __cell. This is the upstream cause of the #544 DOM parser returning 0 rows.

Verified contracts

Dependency: curl-cffi>=0.7.0 already in tradein-mvp/backend/pyproject.toml (no new dep, no lockfile churn).
Response duck-compat: caller fetch_house_history uses only .status_code + .text — both present on curl_cffi Response. ✓
Async lifecycle: AsyncSession.close() is a coroutine (verified against curl-cffi docs) — await self._cffi_session.close() correct. Session created bare without async with is supported by curl-cffi.
Caller invariant: both callers (admin.py:915, estimator.py:397) wrap in async with YandexValuationScraper() as scraper: — the RuntimeError guard in _http_get will not fire in practice.
Timeout default: kwargs.setdefault("timeout", 30) — explicit, slightly more lenient than BaseScraper's 20 s, reasonable for SSR Yandex.
SSL/cert: no verify=False — curl_cffi default verification preserved.

Acknowledged trade-offs (documented in PR)

Retry loss: BaseScraper _http_get has @retry(stop_after_attempt(3)). Override does NOT inherit it (decorator drops on method override). PR docstring states this explicitly. Caller path fetch_house_history already wraps in try/except Exception → return None, so a transient TLS/network blip produces the same outcome as before (logged exception, no rows for that page). Acceptable; same blast radius as pre-PR error path.
super().__aenter__() not called — httpx self._client stays None. Intentional (Yandex valuation never uses httpx). No inherited helper currently dereferences self._client. Future-proof note only.

Tests

PR description correct: all 40 existing tests in test_yandex_valuation.py / test_yandex_valuation_save.py are pure-parser (operate on raw HTML strings, never touch _http_get or session). No mocks needed; nothing to update.

Minor (non-blocking)

PR description says "Same pattern as sibling yandex_realty.py" — yandex_realty.py actually still uses BaseScraper httpx (no curl_cffi import). True sibling is scripts/local-sweep-ekb-yandex.py. Wording quibble only.
Future refactor (out of scope): if 2+ scrapers need Chrome TLS, extract a CurlCffiSessionMixin instead of duplicating the 3-method override.

Cross-file impact

Caller	Path	Impact
`admin.py:905` — `POST /scrape/yandex-valuation`	uses `async with`	none, picks up new session transparently
`estimator.py:397` — `_get_or_fetch_yandex_valuation_cached`	uses `async with`	none, cache layer untouched

Vault cross-check

Consistent with PR history #538 (area sanity cap), #541 (date-order flip), #544 (DOM parser rewrite). This PR completes the loop — DOM parser from #544 was correct, it just had no DOM to parse because TLS was the gate. After merge: PR followup says re-sweep Базовый 52 to verify area_filled ≥90%.

Pre-flight

Single-file, scoped change. ✓
No --no-verify / --amend / --force evidence.
mergeable=true, base=main, branch fix/tradein-yandex-curl-cffi.

Approving — merge unblocks #544 DOM parser working in production.

## Deep Code Review — PR #545 (yandex_valuation curl_cffi chrome120) **Status:** APPROVE **Files:** 1 (P1 scraper) | **Lines:** +26 / -0 | **SHA verified:** `ed6e52f` matches head. ### Verdict summary Targeted override of `__aenter__`/`__aexit__`/`_http_get` on `YandexValuationScraper` to use `curl_cffi.requests.AsyncSession(impersonate="chrome120")`. BaseScraper untouched — zero blast-radius on other scrapers (Avito, Cian, Yandex realty/detail/newbuilding). Live evidence in PR description is convincing: plain-httpx 1.27 MB HTML contains only table header (`__head`), curl_cffi same URL returns 21 `__row` × 133 `__cell`. This is the upstream cause of the #544 DOM parser returning 0 rows. ### Verified contracts - **Dependency:** `curl-cffi>=0.7.0` already in `tradein-mvp/backend/pyproject.toml` (no new dep, no lockfile churn). - **Response duck-compat:** caller `fetch_house_history` uses only `.status_code` + `.text` — both present on curl_cffi `Response`. ✓ - **Async lifecycle:** `AsyncSession.close()` is a coroutine (verified against curl-cffi docs) — `await self._cffi_session.close()` correct. Session created bare without `async with` is supported by curl-cffi. - **Caller invariant:** both callers (`admin.py:915`, `estimator.py:397`) wrap in `async with YandexValuationScraper() as scraper:` — the `RuntimeError` guard in `_http_get` will not fire in practice. - **Timeout default:** `kwargs.setdefault("timeout", 30)` — explicit, slightly more lenient than BaseScraper's 20 s, reasonable for SSR Yandex. - **SSL/cert:** no `verify=False` — curl_cffi default verification preserved. ### Acknowledged trade-offs (documented in PR) 1. **Retry loss:** BaseScraper `_http_get` has `@retry(stop_after_attempt(3))`. Override does NOT inherit it (decorator drops on method override). PR docstring states this explicitly. Caller path `fetch_house_history` already wraps in `try/except Exception → return None`, so a transient TLS/network blip produces the same outcome as before (logged exception, no rows for that page). Acceptable; same blast radius as pre-PR error path. 2. **`super().__aenter__()` not called** — httpx `self._client` stays `None`. Intentional (Yandex valuation never uses httpx). No inherited helper currently dereferences `self._client`. Future-proof note only. ### Tests PR description correct: all 40 existing tests in `test_yandex_valuation.py` / `test_yandex_valuation_save.py` are pure-parser (operate on raw HTML strings, never touch `_http_get` or session). No mocks needed; nothing to update. ### Minor (non-blocking) - PR description says "Same pattern as sibling `yandex_realty.py`" — `yandex_realty.py` actually still uses BaseScraper httpx (no curl_cffi import). True sibling is `scripts/local-sweep-ekb-yandex.py`. Wording quibble only. - Future refactor (out of scope): if 2+ scrapers need Chrome TLS, extract a `CurlCffiSessionMixin` instead of duplicating the 3-method override. ### Cross-file impact | Caller | Path | Impact | |---|---|---| | `admin.py:905` — `POST /scrape/yandex-valuation` | uses `async with` | none, picks up new session transparently | | `estimator.py:397` — `_get_or_fetch_yandex_valuation_cached` | uses `async with` | none, cache layer untouched | ### Vault cross-check Consistent with PR history #538 (area sanity cap), #541 (date-order flip), #544 (DOM parser rewrite). This PR completes the loop — DOM parser from #544 was correct, it just had no DOM to parse because TLS was the gate. After merge: PR followup says re-sweep Базовый 52 to verify `area_filled ≥90%`. ### Pre-flight - Single-file, scoped change. ✓ - No `--no-verify` / `--amend` / `--force` evidence. - mergeable=true, base=main, branch `fix/tradein-yandex-curl-cffi`. Approving — merge unblocks #544 DOM parser working in production.