lekss361
b5829fa226
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 35s
Deploy Trade-In / build-backend (push) Successful in 26s
Deploy Trade-In / deploy (push) Successful in 42s
49 lines
2.8 KiB
PL/PgSQL
49 lines
2.8 KiB
PL/PgSQL
-- 118_enable_cian_city_sweep.sql
|
|
-- Enable the cian_city_sweep schedule (novostroyki + vtorichka listing coverage).
|
|
--
|
|
-- Context:
|
|
-- cian_city_sweep was seeded enabled=false in 107_scrape_schedules_seed_cian_city_sweep.sql
|
|
-- with the note "включается оператором вручную после восстановления прокси" — it was
|
|
-- shipped DORMANT because the cian mobile-proxy egress was assumed dead (last attempt
|
|
-- 2026-05-31 aborted on "3 consecutive anchor failures / anchor timeout 240s").
|
|
--
|
|
-- As a result cian listing coverage collapsed: the ~15k active cian rows became a frozen
|
|
-- late-May snapshot (only ~28 of 15067 re-seen within 14 days as of 2026-06-16), and
|
|
-- the 9659 active novostroyki were never refreshed.
|
|
--
|
|
-- Diagnosis (2026-06-16, verified on prod):
|
|
-- - All three mobile-proxy egresses (AVITO/CIAN/YANDEX) are ALIVE — the rotate endpoints
|
|
-- return status=OK with fresh IPs. The "proxy dead" premise is stale.
|
|
-- - cian_city_sweep is COOKIELESS: CianScraper self-warms its curl_cffi (chrome120)
|
|
-- session via a homepage GET; cian_detail.fetch_detail opens its own AsyncSession
|
|
-- (chrome120 + cian_proxy_url). Neither uses the DB cian_session_cookies. Only the
|
|
-- separate cian_history_backfill (source-provided priceChanges backfill) needs those
|
|
-- cookies (which are stale since 2026-05-31 — tracked separately for human re-upload).
|
|
-- - Smoke runs on prod (run 99 SERP-only, run 100 with detail+houses) succeeded:
|
|
-- run 99: lots_fetched=560, lots_inserted=89, lots_updated=471, 0 errors
|
|
-- run 100: detail_enriched=3/3, houses_enriched=4, 0 detail failures
|
|
-- proving the full sweep (SERP → detail → houses) works cookieless via the live proxy.
|
|
--
|
|
-- What this migration does:
|
|
-- Enable cian_city_sweep with production params (pages_per_anchor=3, detail_top_n=10,
|
|
-- enrich_houses=true, request_delay_sec=5, radius_m=1500), nightly window 02:00-05:00 UTC
|
|
-- (same as avito; a separate proxy egress so no contention). next_run_at bootstrapped to
|
|
-- tomorrow 02:30 UTC so a fresh deploy does not fire immediately.
|
|
--
|
|
-- Idempotent: plain UPDATE keyed on source — safe to re-run (sets the same values).
|
|
--
|
|
-- Dependencies:
|
|
-- 052_scrape_schedules.sql (table), 107_scrape_schedules_seed_cian_city_sweep.sql (row),
|
|
-- scheduler.py trigger_cian_city_sweep_run (already deployed).
|
|
|
|
BEGIN;
|
|
|
|
UPDATE scrape_schedules
|
|
SET enabled = true,
|
|
window_start_hour = 2,
|
|
window_end_hour = 5,
|
|
default_params = '{"radius_m":1500,"detail_top_n":10,"enrich_houses":true,"pages_per_anchor":3,"request_delay_sec":5}'::jsonb,
|
|
next_run_at = ((CURRENT_DATE + INTERVAL '1 day') + INTERVAL '2 hours 30 minutes') AT TIME ZONE 'UTC'
|
|
WHERE source = 'cian_city_sweep';
|
|
|
|
COMMIT;
|