feat(scrapers): enable cian_city_sweep — proxy alive, sweep is cookieless
All checks were successful
CI / frontend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 7s
CI / openapi-codegen-check (pull_request) Has been skipped

cian_city_sweep was dormant (107, enabled=false) on a stale 'proxy dead'
assumption from 2026-05-31. Verified 2026-06-16: all 3 mobile-proxy egresses
alive; CianScraper + cian_detail self-warm their curl_cffi sessions (no DB
cookies). Prod smoke proved the full sweep works cookieless:
  run 99 (SERP):   560 fetched / 89 inserted / 471 updated, 0 errors
  run 100 (detail): detail_enriched=3/3, houses_enriched=4, 0 failures
Only cian_history_backfill (source-provided priceChanges) needs the stale
cookies — tracked separately for human re-upload.

Restores novostroyki + vtorichka cian coverage (was a frozen late-May
snapshot: ~28 of 15067 active re-seen within 14d). Nightly 02-05 UTC.

Refs #759
This commit is contained in:
bot-backend 2026-06-16 20:05:21 +03:00
parent e46867fa7f
commit 3853007e88

View file

@ -0,0 +1,49 @@
-- 118_enable_cian_city_sweep.sql
-- Enable the cian_city_sweep schedule (novostroyki + vtorichka listing coverage).
--
-- Context:
-- cian_city_sweep was seeded enabled=false in 107_scrape_schedules_seed_cian_city_sweep.sql
-- with the note "включается оператором вручную после восстановления прокси" — it was
-- shipped DORMANT because the cian mobile-proxy egress was assumed dead (last attempt
-- 2026-05-31 aborted on "3 consecutive anchor failures / anchor timeout 240s").
--
-- As a result cian listing coverage collapsed: the ~15k active cian rows became a frozen
-- late-May snapshot (only ~28 of 15067 re-seen within 14 days as of 2026-06-16), and
-- the 9659 active novostroyki were never refreshed.
--
-- Diagnosis (2026-06-16, verified on prod):
-- - All three mobile-proxy egresses (AVITO/CIAN/YANDEX) are ALIVE — the rotate endpoints
-- return status=OK with fresh IPs. The "proxy dead" premise is stale.
-- - cian_city_sweep is COOKIELESS: CianScraper self-warms its curl_cffi (chrome120)
-- session via a homepage GET; cian_detail.fetch_detail opens its own AsyncSession
-- (chrome120 + cian_proxy_url). Neither uses the DB cian_session_cookies. Only the
-- separate cian_history_backfill (source-provided priceChanges backfill) needs those
-- cookies (which are stale since 2026-05-31 — tracked separately for human re-upload).
-- - Smoke runs on prod (run 99 SERP-only, run 100 with detail+houses) succeeded:
-- run 99: lots_fetched=560, lots_inserted=89, lots_updated=471, 0 errors
-- run 100: detail_enriched=3/3, houses_enriched=4, 0 detail failures
-- proving the full sweep (SERP → detail → houses) works cookieless via the live proxy.
--
-- What this migration does:
-- Enable cian_city_sweep with production params (pages_per_anchor=3, detail_top_n=10,
-- enrich_houses=true, request_delay_sec=5, radius_m=1500), nightly window 02:00-05:00 UTC
-- (same as avito; a separate proxy egress so no contention). next_run_at bootstrapped to
-- tomorrow 02:30 UTC so a fresh deploy does not fire immediately.
--
-- Idempotent: plain UPDATE keyed on source — safe to re-run (sets the same values).
--
-- Dependencies:
-- 052_scrape_schedules.sql (table), 107_scrape_schedules_seed_cian_city_sweep.sql (row),
-- scheduler.py trigger_cian_city_sweep_run (already deployed).
BEGIN;
UPDATE scrape_schedules
SET enabled = true,
window_start_hour = 2,
window_end_hour = 5,
default_params = '{"radius_m":1500,"detail_top_n":10,"enrich_houses":true,"pages_per_anchor":3,"request_delay_sec":5}'::jsonb,
next_run_at = ((CURRENT_DATE + INTERVAL '1 day') + INTERVAL '2 hours 30 minutes') AT TIME ZONE 'UTC'
WHERE source = 'cian_city_sweep';
COMMIT;