gendesign/tradein-mvp/backend/data/sql/174_domclick_session_cookies.sql
bot-backend 0b9dfcc691
All checks were successful
CI / changes (pull_request) Successful in 7s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 7s
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 52s
feat(tradein/domclick): cookie-injection MVP для обхода QRATOR-блока (#2000)
Эмпирически подтверждено вживую 2026-07-04: инъекция cookies валидной
аутентифицированной test-аккаунт сессии (Sber ID) полностью обходит QRATOR
reputation-блок DomClick, даже на уже подозрительном прокси-IP (проверено:
проход через camoufox context.addCookies() → 200 + полный __SSR_STATE__ на
IP, который секундами ранее отдавал anti-bot страницу без cookies).

- domclick_session_cookies (миграция 174) + app/services/domclick_session.py —
  зеркалит существующий cian_session.py (pgp_sym_encrypt, account_cas_id вместо
  account_user_id), без verify_session (DomClick-верификация требует реального
  browser-фетча, не curl_cffi — future enhancement).
- POST /scrape/domclick/upload-cookies — ручная загрузка (зеркалит upload_cian_cookies).
- POST /scrape/domclick/debug/detail-fetch — единственный сейчас живой путь
  прогнать связку session→cookies→fetch_detail (боевой backfill-оркестратор
  для DomClick ещё не смёржен). source="domclick" — выделенный резидентный
  прокси-пул (id=1, provider_affinity='domclick'), построенный и verified
  в этой же сессии, а не устаревший source="cian" из старой рекомендации.
- _assert_domclick_host — SSRF-guard для *.domclick.ru поддоменов (реальные
  карточки живут на региональных поддоменах, не на голом domclick.ru).
- server.py/browser_fetcher.py: cookies-параметр расширяет origin-механизм
  (#2430) той же схемой — keyword-only, default None → byte-identical для
  avito/cian/yandex. Домен cookie выводится из hostname целевого URL
  (провайдер-агностично), не захардкожен под domclick.
2026-07-04 20:46:20 +03:00

46 lines
2.2 KiB
PL/PgSQL

-- 174_domclick_session_cookies.sql
-- Purpose: Encrypted storage for DomClick browser session cookies needed to
-- bypass QRATOR anti-bot reputation blocks on card-detail fetches.
-- A manually-uploaded authenticated DomClick session (Sber ID login)
-- whose cookies (notably qrator_jsid2, CAS_ID, CAS_ID_SIGNED,
-- sessionId) let scraper requests pass QRATOR's reputation check.
-- Empirically validated 2026-07-04: injecting these cookies via
-- Playwright context.addCookies() through a reputation-blocked
-- proxy IP fully bypassed the block and returned real
-- __SSR_STATE__ content.
-- Uses pgcrypto pgp_sym_encrypt for AES encryption at rest.
-- Dependencies:
-- - pgcrypto extension (installed here via CREATE EXTENSION IF NOT EXISTS)
-- Deploy order: Apply after 173.
--
-- Security notes:
-- - cookies_encrypted stores AES-encrypted JSON blob via pgp_sym_encrypt.
-- - Encryption key lives in .env.runtime (COOKIE_ENCRYPTION_KEY), never in DB.
-- - Access restricted to admin-token-gated API endpoint only.
--
-- Sources: issue #2000 (DomClick Layer B unblock)
BEGIN;
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE TABLE IF NOT EXISTS domclick_session_cookies (
account_cas_id bigint PRIMARY KEY, -- DomClick CAS_ID cookie (Sber-ID-derived user id)
cookies_encrypted bytea NOT NULL, -- pgp_sym_encrypt(json_cookies, key)
expires_at_estimate timestamptz NOT NULL, -- estimated expiry (~30 days from upload)
uploaded_at timestamptz NOT NULL DEFAULT NOW(),
last_used_at timestamptz, -- set on each successful card-detail fetch
last_invalid_at timestamptz, -- set when session no longer bypasses QRATOR
notes text -- e.g. 'Account: <sber id / email>'
);
-- Most recent upload first (admin dashboard ordering)
CREATE INDEX IF NOT EXISTS domclick_cookies_uploaded_idx
ON domclick_session_cookies (uploaded_at DESC);
-- Active session lookup (non-expired and not invalidated)
CREATE INDEX IF NOT EXISTS domclick_cookies_active_idx
ON domclick_session_cookies (expires_at_estimate)
WHERE last_invalid_at IS NULL;
COMMIT;