cian offer_price_history frozen since 2026-05-24 — server backfill is a silent no-op; needs local Playwright runner #1575

New issue

Closed

opened 2026-06-16 17:06:41 +00:00 by lekss361 · 1 comment

lekss361 commented

2026-06-16 17:06:41 +00:00

Owner

Corrected root cause (initial "stale cookies" premise was WRONG)

Cookies are valid, not stale: account 75544897, expires_at_estimate=2026-06-30, last_invalid_at=NULL, last_used_at=2026-06-16 (today). Verified on prod.

The real problem: offer_price_history has been frozen at 2026-05-24 (7125 rows total; 14274 active cian listings have zero history rows), because:

Server-side cian_history_backfill is a silent no-op. It runs nightly with status=done, listings_processed=100, listings_succeeded=100 — but inserts 0 new rows. Server-IP + curl_cffi cannot extract priceChanges: per runbook cian_local_playwright_backfill.md, Cian SSR HTML no longer contains _cianConfig.push() in raw form — it needs real-Chrome JS execution to populate _cianConfig['frontend-offer-card'].defaultState.priceChanges. So the server task "succeeds" while extracting nothing. (Also documented 2026-05-30: valid cookies are rejected on the server egress, failed_fetch.)
The only writer that has ever populated cian offer_price_history is the local Playwright runner tradein-mvp/scripts/local-cian/playwright_history.py — runs from a home/dev IP with a real Chromium, injects the existing valid DB cookies, reads _cianConfig via JS, extracts priceChanges. It wrote the 7125 rows in late-May sessions and hasn't run since ~05-24.

What does NOT need fixing

Cian listing coverage (novostroyki + vtorichka) — restored via PR #1568 (cian_city_sweep re-enabled, cookieless).
avito/yandex price-history — now captured via the derived trigger (PR #1559).
Cian cookies — valid through 2026-06-30, no re-upload needed yet.

Action options (price-history backfill for the 14274-listing backlog)

(a) Run the local Playwright backfill (runbook cian_local_playwright_backfill.md): SSH tunnel to tradein-postgres + COOKIE_ENCRYPTION_KEY + python playwright_history.py --batch 20. ~25 listings/min → ~9.5h for the full backlog. Can be run from the user's machine (cookies already valid). This is a recurring maintenance job, not a code fix.
(b) Build a server-side Chromium price-history path (Playwright-in-container) — heavy, headless-CAPTCHA risk; only if (a) is too manual.
(c) Accept the derived trigger going forward — PR #1559's trigger now records cian price changes on re-scrape (it currently excludes cian to protect the rich backfill; could be revisited). This captures changes from now on but not the historical timeline.

Re-labeled from needs-human (cookies are fine) — this is an ops/maintenance item.

Diagnosis: vault fixes/Fix_Deactivate_Stale_Segment_Aware_Ghost_Cleanup_Jun16.md. Refs #759, #1568.

## Corrected root cause (initial "stale cookies" premise was WRONG) Cookies are **valid**, not stale: account `75544897`, `expires_at_estimate=2026-06-30`, `last_invalid_at=NULL`, `last_used_at=2026-06-16` (today). Verified on prod. The real problem: **`offer_price_history` has been frozen at 2026-05-24** (7125 rows total; **14274 active cian listings have zero history rows**), because: 1. **Server-side `cian_history_backfill` is a silent no-op.** It runs nightly with `status=done, listings_processed=100, listings_succeeded=100` — but inserts **0** new rows. Server-IP + curl_cffi cannot extract `priceChanges`: per runbook `cian_local_playwright_backfill.md`, Cian SSR HTML no longer contains `_cianConfig.push()` in raw form — it needs **real-Chrome JS execution** to populate `_cianConfig['frontend-offer-card'].defaultState.priceChanges`. So the server task "succeeds" while extracting nothing. (Also documented 2026-05-30: valid cookies are rejected on the server egress, `failed_fetch`.) 2. The only writer that has ever populated cian `offer_price_history` is the **local Playwright runner** `tradein-mvp/scripts/local-cian/playwright_history.py` — runs from a home/dev IP with a real Chromium, injects the existing valid DB cookies, reads `_cianConfig` via JS, extracts `priceChanges`. It wrote the 7125 rows in late-May sessions and hasn't run since ~05-24. ## What does NOT need fixing - **Cian listing coverage** (novostroyki + vtorichka) — restored via PR #1568 (`cian_city_sweep` re-enabled, cookieless). - **avito/yandex price-history** — now captured via the derived trigger (PR #1559). - **Cian cookies** — valid through 2026-06-30, no re-upload needed yet. ## Action options (price-history backfill for the 14274-listing backlog) - **(a) Run the local Playwright backfill** (runbook `cian_local_playwright_backfill.md`): SSH tunnel to tradein-postgres + `COOKIE_ENCRYPTION_KEY` + `python playwright_history.py --batch 20`. ~25 listings/min → ~9.5h for the full backlog. Can be run from the user's machine (cookies already valid). **This is a recurring maintenance job**, not a code fix. - **(b) Build a server-side Chromium price-history path** (Playwright-in-container) — heavy, headless-CAPTCHA risk; only if (a) is too manual. - **(c) Accept the derived trigger going forward** — PR #1559's trigger now records cian price *changes* on re-scrape (it currently excludes cian to protect the rich backfill; could be revisited). This captures changes from now on but not the historical timeline. Re-labeled from needs-human (cookies are fine) — this is an ops/maintenance item. Diagnosis: vault `fixes/Fix_Deactivate_Stale_Segment_Aware_Ghost_Cleanup_Jun16.md`. Refs #759, #1568.

lekss361 added the

needs-human

scrapers

labels 2026-06-16 17:06:46 +00:00

lekss361 changed title from ~~Re-upload Cian session cookies (stale since 2026-05-31) — unblocks cian_history_backfill price-history~~ to cian offer_price_history frozen since 2026-05-24 — server backfill is a silent no-op; needs local Playwright runner

2026-06-16 19:00:58 +00:00

lekss361 referenced this issue

2026-06-16 19:43:20 +00:00

feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) #1657

lekss361 commented

2026-06-17 17:12:07 +00:00

Author

Owner

Проверка по forgejo/main: устранено в #1657 (commit 66433dfe) — cian_history_backfill.py:113-118 теперь тянет priceChanges через BrowserFetcher (JS-render), молчаливый no-op backfill пофикшен. Закрываю.

Проверка по `forgejo/main`: устранено в #1657 (commit `66433dfe`) — `cian_history_backfill.py:113-118` теперь тянет priceChanges через BrowserFetcher (JS-render), молчаливый no-op backfill пофикшен. Закрываю.

lekss361 closed this issue

2026-06-17 17:12:13 +00:00