feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) #1657

Merged

bot-reviewer merged 1 commit from feat/cian-browser-price-history into main

2026-06-16 19:47:16 +00:00

lekss361 commented

2026-06-16 19:43:20 +00:00

Owner

Summary

Makes the normal server-side cian scraper collect price history — no more dependency on the local Playwright runner.

Problem

offer_price_history was frozen at 2026-05-24 (7125 rows; 14274 active cian listings with zero history). cian_history_backfill ran nightly done, listings_succeeded:100 but wrote 0 rows — a silent no-op: curl_cffi fetch_detail can't see priceChanges (JS-rendered _cianConfig['frontend-offer-card'].defaultState). Proven on prod: a listing with 6 known history rows → curl_cffi price_changes=0.

Fix

Route the backfill's detail fetch through the existing BrowserFetcher (camoufox tradein-browser service), which renders JS. Proven on prod: BrowserFetcher().fetch(<cian detail>) returns HTML containing priceChanges ({changeTime, priceData:{price}}).

cian_detail.fetch_detail — new browser_fetcher: BrowserFetcher | None param; when provided, fetches via browser; curl_cffi path unchanged when None (back-compat).
_extract_price_changes — extended to read offerData.priceChanges (browser HTML location) + priceData.price nesting, in addition to the legacy offer.priceChanges flat form. (The pre-existing parser did NOT handle the browser format — this was necessary.)
backfill_cian_history — opens one BrowserFetcher per run, passes it to every fetch_detail. Drains the 14274 backlog over nightly runs + stays current. Cookieless (browser-mode doesn't need the DB cookies).

Prod verification

BrowserFetcher + updated parser on listing 328647045 → 10 price_changes extracted (DB had 6 older rows). Mechanism proven end-to-end.

Test plan

Full gate: 1840 passed, 2 deselected (added unit test mocking BrowserFetcher → asserts price_changes populated)
ruff clean
Post-deploy: cian_history_backfill run grows offer_price_history past 2026-05-24

Refs #1575, #759

## Summary Makes the normal server-side cian scraper collect price history — no more dependency on the local Playwright runner. ### Problem `offer_price_history` was frozen at 2026-05-24 (7125 rows; **14274 active cian listings with zero history**). `cian_history_backfill` ran nightly `done, listings_succeeded:100` but wrote **0 rows** — a silent no-op: curl_cffi `fetch_detail` can't see `priceChanges` (JS-rendered `_cianConfig['frontend-offer-card'].defaultState`). Proven on prod: a listing with 6 known history rows → curl_cffi `price_changes=0`. ### Fix Route the backfill's detail fetch through the existing `BrowserFetcher` (camoufox `tradein-browser` service), which renders JS. Proven on prod: `BrowserFetcher().fetch(<cian detail>)` returns HTML containing `priceChanges` (`{changeTime, priceData:{price}}`). - `cian_detail.fetch_detail` — new `browser_fetcher: BrowserFetcher | None` param; when provided, fetches via browser; curl_cffi path unchanged when None (back-compat). - `_extract_price_changes` — extended to read `offerData.priceChanges` (browser HTML location) + `priceData.price` nesting, in addition to the legacy `offer.priceChanges` flat form. **(The pre-existing parser did NOT handle the browser format — this was necessary.)** - `backfill_cian_history` — opens one `BrowserFetcher` per run, passes it to every `fetch_detail`. Drains the 14274 backlog over nightly runs + stays current. Cookieless (browser-mode doesn't need the DB cookies). ### Prod verification `BrowserFetcher` + updated parser on listing 328647045 → **10 price_changes** extracted (DB had 6 older rows). Mechanism proven end-to-end. ## Test plan - [x] Full gate: **1840 passed, 2 deselected** (added unit test mocking BrowserFetcher → asserts price_changes populated) - [x] ruff clean - [ ] Post-deploy: `cian_history_backfill` run grows `offer_price_history` past 2026-05-24 Refs #1575, #759

lekss361 added 1 commit 2026-06-16 19:43:21 +00:00

feat(cian): use BrowserFetcher in history backfill to populate priceChanges

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI / changes (push) Successful in 7s

Details

CI / changes (pull_request) Successful in 5s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

af189e7e40

- cian_detail.fetch_detail: add browser_fetcher keyword param (BrowserFetcher|None).
  When provided, HTML is fetched via camoufox instead of curl_cffi. Parse path
  is identical — only the HTML source changes.
- _extract_price_changes: extend to also look at offerData.priceChanges (browser
  HTML location) and handle nested priceData.price format (proven on prod 2026-06-16:
  listing 328647045 yields 10 entries).
- cian_history_backfill.backfill_cian_history: open one BrowserFetcher for the
  batch and pass browser_fetcher=bf to each fetch_detail call; single-listing
  failure does not abort the run.
- Tests: 6 new tests for browser_fetcher param + priceData.price format; 32 pass
  total. Full gate: 1840 passed, 2 deselected.