All checks were successful
CI / changes (pull_request) Successful in 7s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 7s
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 52s
Эмпирически подтверждено вживую 2026-07-04: инъекция cookies валидной аутентифицированной test-аккаунт сессии (Sber ID) полностью обходит QRATOR reputation-блок DomClick, даже на уже подозрительном прокси-IP (проверено: проход через camoufox context.addCookies() → 200 + полный __SSR_STATE__ на IP, который секундами ранее отдавал anti-bot страницу без cookies). - domclick_session_cookies (миграция 174) + app/services/domclick_session.py — зеркалит существующий cian_session.py (pgp_sym_encrypt, account_cas_id вместо account_user_id), без verify_session (DomClick-верификация требует реального browser-фетча, не curl_cffi — future enhancement). - POST /scrape/domclick/upload-cookies — ручная загрузка (зеркалит upload_cian_cookies). - POST /scrape/domclick/debug/detail-fetch — единственный сейчас живой путь прогнать связку session→cookies→fetch_detail (боевой backfill-оркестратор для DomClick ещё не смёржен). source="domclick" — выделенный резидентный прокси-пул (id=1, provider_affinity='domclick'), построенный и verified в этой же сессии, а не устаревший source="cian" из старой рекомендации. - _assert_domclick_host — SSRF-guard для *.domclick.ru поддоменов (реальные карточки живут на региональных поддоменах, не на голом domclick.ru). - server.py/browser_fetcher.py: cookies-параметр расширяет origin-механизм (#2430) той же схемой — keyword-only, default None → byte-identical для avito/cian/yandex. Домен cookie выводится из hostname целевого URL (провайдер-агностично), не захардкожен под domclick.
46 lines
2.2 KiB
PL/PgSQL
46 lines
2.2 KiB
PL/PgSQL
-- 174_domclick_session_cookies.sql
|
|
-- Purpose: Encrypted storage for DomClick browser session cookies needed to
|
|
-- bypass QRATOR anti-bot reputation blocks on card-detail fetches.
|
|
-- A manually-uploaded authenticated DomClick session (Sber ID login)
|
|
-- whose cookies (notably qrator_jsid2, CAS_ID, CAS_ID_SIGNED,
|
|
-- sessionId) let scraper requests pass QRATOR's reputation check.
|
|
-- Empirically validated 2026-07-04: injecting these cookies via
|
|
-- Playwright context.addCookies() through a reputation-blocked
|
|
-- proxy IP fully bypassed the block and returned real
|
|
-- __SSR_STATE__ content.
|
|
-- Uses pgcrypto pgp_sym_encrypt for AES encryption at rest.
|
|
-- Dependencies:
|
|
-- - pgcrypto extension (installed here via CREATE EXTENSION IF NOT EXISTS)
|
|
-- Deploy order: Apply after 173.
|
|
--
|
|
-- Security notes:
|
|
-- - cookies_encrypted stores AES-encrypted JSON blob via pgp_sym_encrypt.
|
|
-- - Encryption key lives in .env.runtime (COOKIE_ENCRYPTION_KEY), never in DB.
|
|
-- - Access restricted to admin-token-gated API endpoint only.
|
|
--
|
|
-- Sources: issue #2000 (DomClick Layer B unblock)
|
|
|
|
BEGIN;
|
|
|
|
CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
|
|
|
CREATE TABLE IF NOT EXISTS domclick_session_cookies (
|
|
account_cas_id bigint PRIMARY KEY, -- DomClick CAS_ID cookie (Sber-ID-derived user id)
|
|
cookies_encrypted bytea NOT NULL, -- pgp_sym_encrypt(json_cookies, key)
|
|
expires_at_estimate timestamptz NOT NULL, -- estimated expiry (~30 days from upload)
|
|
uploaded_at timestamptz NOT NULL DEFAULT NOW(),
|
|
last_used_at timestamptz, -- set on each successful card-detail fetch
|
|
last_invalid_at timestamptz, -- set when session no longer bypasses QRATOR
|
|
notes text -- e.g. 'Account: <sber id / email>'
|
|
);
|
|
|
|
-- Most recent upload first (admin dashboard ordering)
|
|
CREATE INDEX IF NOT EXISTS domclick_cookies_uploaded_idx
|
|
ON domclick_session_cookies (uploaded_at DESC);
|
|
|
|
-- Active session lookup (non-expired and not invalidated)
|
|
CREATE INDEX IF NOT EXISTS domclick_cookies_active_idx
|
|
ON domclick_session_cookies (expires_at_estimate)
|
|
WHERE last_invalid_at IS NULL;
|
|
|
|
COMMIT;
|