feat(scrapers): enable cian_city_sweep (proxy alive, sweep is cookieless) #1568

Merged

lekss361 merged 1 commit from feat/enable-cian-city-sweep into main

2026-06-16 17:08:48 +00:00

lekss361 commented

2026-06-16 17:05:44 +00:00

Owner

Summary

Migration 118_enable_cian_city_sweep.sql re-enables the dormant cian_city_sweep schedule to restore novostroyki + vtorichka cian listing coverage.

Why it was off, and why it's safe to enable now

cian_city_sweep was seeded enabled=false (107) on a stale "proxy dead" assumption from 2026-05-31 (last attempt aborted on anchor timeouts). As a result cian's ~15k active rows became a frozen late-May snapshot — only ~28 of 15067 re-seen within 14 days, and the 9659 active novostroyki never refreshed.

Verified on prod 2026-06-16:

All 3 mobile-proxy egresses are ALIVE (rotate endpoints return status:OK, fresh IPs). The "proxy dead" premise is stale.
cian_city_sweep is cookieless: CianScraper self-warms its curl_cffi (chrome120) session via a homepage GET; cian_detail.fetch_detail opens its own AsyncSession (chrome120 + cian_proxy_url). Neither uses the DB cian_session_cookies. Only the separate cian_history_backfill (source-provided priceChanges) needs those cookies — they're stale since 05-31, tracked separately for human re-upload.

Prod smoke proof (controlled, via `scrape_schedules` next_run=now)

run 99 (SERP-only): lots_fetched=560, lots_inserted=89, lots_updated=471, 0 errors.
run 100 (detail+houses): detail_enriched=3/3, houses_enriched=4, 0 detail failures.

Both completed done. After verification, prod was set to the production config this migration encodes; the migration makes the enable reproducible (a fresh DB rebuild would otherwise re-apply 107's disabled seed).

Config

enabled=true, nightly 02:00-05:00 UTC (separate proxy egress from avito — no contention), params pages_per_anchor=3, detail_top_n=10, enrich_houses=true, request_delay_sec=5, radius_m=1500. next_run_at bootstrapped to tomorrow 02:30 UTC.

Test plan

Prod smoke (runs 99/100) — full sweep works cookieless
Idempotent UPDATE keyed on source
Post-deploy: confirm cian_city_sweep enabled; first nightly run refreshes cian last_seen_at

Refs #759

## Summary Migration `118_enable_cian_city_sweep.sql` re-enables the dormant `cian_city_sweep` schedule to restore **novostroyki + vtorichka** cian listing coverage. ### Why it was off, and why it's safe to enable now `cian_city_sweep` was seeded `enabled=false` (107) on a **stale "proxy dead"** assumption from 2026-05-31 (last attempt aborted on anchor timeouts). As a result cian's ~15k active rows became a **frozen late-May snapshot** — only ~28 of 15067 re-seen within 14 days, and the 9659 active novostroyki never refreshed. Verified on prod 2026-06-16: - **All 3 mobile-proxy egresses are ALIVE** (rotate endpoints return `status:OK`, fresh IPs). The "proxy dead" premise is stale. - **cian_city_sweep is cookieless**: `CianScraper` self-warms its curl_cffi (chrome120) session via a homepage GET; `cian_detail.fetch_detail` opens its own `AsyncSession` (chrome120 + `cian_proxy_url`). Neither uses the DB `cian_session_cookies`. Only the *separate* `cian_history_backfill` (source-provided priceChanges) needs those cookies — they're stale since 05-31, tracked separately for human re-upload. ### Prod smoke proof (controlled, via `scrape_schedules` next_run=now) - **run 99** (SERP-only): `lots_fetched=560, lots_inserted=89, lots_updated=471`, **0 errors**. - **run 100** (detail+houses): `detail_enriched=3/3, houses_enriched=4`, **0 detail failures**. Both completed `done`. After verification, prod was set to the production config this migration encodes; the migration makes the enable reproducible (a fresh DB rebuild would otherwise re-apply 107's disabled seed). ### Config `enabled=true`, nightly **02:00-05:00 UTC** (separate proxy egress from avito — no contention), params `pages_per_anchor=3, detail_top_n=10, enrich_houses=true, request_delay_sec=5, radius_m=1500`. `next_run_at` bootstrapped to tomorrow 02:30 UTC. ## Test plan - [x] Prod smoke (runs 99/100) — full sweep works cookieless - [x] Idempotent UPDATE keyed on source - [ ] Post-deploy: confirm `cian_city_sweep` enabled; first nightly run refreshes cian `last_seen_at` Refs #759

lekss361 added 1 commit 2026-06-16 17:05:45 +00:00

feat(scrapers): enable cian_city_sweep — proxy alive, sweep is cookieless

CI / frontend-tests (push) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / changes (push) Successful in 8s

Details

CI / backend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 7s

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

3853007e88

cian_city_sweep was dormant (107, enabled=false) on a stale 'proxy dead'
assumption from 2026-05-31. Verified 2026-06-16: all 3 mobile-proxy egresses
alive; CianScraper + cian_detail self-warm their curl_cffi sessions (no DB
cookies). Prod smoke proved the full sweep works cookieless:
  run 99 (SERP):   560 fetched / 89 inserted / 471 updated, 0 errors
  run 100 (detail): detail_enriched=3/3, houses_enriched=4, 0 failures
Only cian_history_backfill (source-provided priceChanges) needs the stale
cookies — tracked separately for human re-upload.

Restores novostroyki + vtorichka cian coverage (was a frozen late-May
snapshot: ~28 of 15067 active re-seen within 14d). Nightly 02-05 UTC.

Refs #759