feat(tradein/domclick): cookie-injection MVP для обхода QRATOR-блока (#2000)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 7s
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 52s
All checks were successful
CI / changes (pull_request) Successful in 7s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 7s
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 52s
Эмпирически подтверждено вживую 2026-07-04: инъекция cookies валидной аутентифицированной test-аккаунт сессии (Sber ID) полностью обходит QRATOR reputation-блок DomClick, даже на уже подозрительном прокси-IP (проверено: проход через camoufox context.addCookies() → 200 + полный __SSR_STATE__ на IP, который секундами ранее отдавал anti-bot страницу без cookies). - domclick_session_cookies (миграция 174) + app/services/domclick_session.py — зеркалит существующий cian_session.py (pgp_sym_encrypt, account_cas_id вместо account_user_id), без verify_session (DomClick-верификация требует реального browser-фетча, не curl_cffi — future enhancement). - POST /scrape/domclick/upload-cookies — ручная загрузка (зеркалит upload_cian_cookies). - POST /scrape/domclick/debug/detail-fetch — единственный сейчас живой путь прогнать связку session→cookies→fetch_detail (боевой backfill-оркестратор для DomClick ещё не смёржен). source="domclick" — выделенный резидентный прокси-пул (id=1, provider_affinity='domclick'), построенный и verified в этой же сессии, а не устаревший source="cian" из старой рекомендации. - _assert_domclick_host — SSRF-guard для *.domclick.ru поддоменов (реальные карточки живут на региональных поддоменах, не на голом domclick.ru). - server.py/browser_fetcher.py: cookies-параметр расширяет origin-механизм (#2430) той же схемой — keyword-only, default None → byte-identical для avito/cian/yandex. Домен cookie выводится из hostname целевого URL (провайдер-агностично), не захардкожен под domclick.
This commit is contained in:
parent
acc1e05357
commit
0b9dfcc691
12 changed files with 938 additions and 19 deletions
|
|
@ -69,6 +69,7 @@ from app.core.config import settings
|
|||
from app.core.db import SessionLocal, get_db
|
||||
from app.schemas.trade_in import ScheduleConfig, ScheduleConfigUpdate
|
||||
from app.services import cian_session as cian_session_svc
|
||||
from app.services import domclick_session as domclick_session_svc
|
||||
from app.services import scrape_runs as runs_mod
|
||||
from app.services.geocoder import geocode
|
||||
from app.services.scheduler import has_running_run
|
||||
|
|
@ -110,6 +111,21 @@ def _assert_allowed_url(url: str) -> None:
|
|||
raise HTTPException(status_code=400, detail="host not allowed")
|
||||
|
||||
|
||||
def _assert_domclick_host(url: str) -> None:
|
||||
"""SSRF guard для DomClick card-URL — вариант _assert_allowed_url для поддоменов.
|
||||
|
||||
settings.scrape_allowed_hosts содержит только точные "domclick.ru"/"www.domclick.ru",
|
||||
а реальные карточки живут на региональных поддоменах (ekaterinburg.domclick.ru,
|
||||
spb.domclick.ru, ...) — точное сравнение из _assert_allowed_url их бы отклонило.
|
||||
Разрешаем domclick.ru и любой *.domclick.ru поддомен; относительные пути
|
||||
(netloc пуст) пропускаем как и _assert_allowed_url.
|
||||
"""
|
||||
parsed = urlparse(url)
|
||||
host = parsed.netloc.lower()
|
||||
if host and host != "domclick.ru" and not host.endswith(".domclick.ru"):
|
||||
raise HTTPException(status_code=400, detail="host not allowed")
|
||||
|
||||
|
||||
def _safe_http_error(status_code: int, public_detail: str, log_context: str) -> HTTPException:
|
||||
"""Defense-in-depth: не отдаём текст исключения в HTTP-ответ.
|
||||
|
||||
|
|
@ -486,6 +502,136 @@ async def test_cian_auth(
|
|||
return {"authenticated": True, "userId": user_id, "reason": None}
|
||||
|
||||
|
||||
# ── DomClick session cookie management (cookie-injection MVP, #2000) ─────────
|
||||
#
|
||||
# Эмпирически подтверждено вживую (2026-07-04): QRATOR-блок DomClick (даже на
|
||||
# уже подозрительном прокси-IP) полностью обходится инъекцией cookies валидной
|
||||
# аутентифицированной test-аккаунт сессии (Sber ID) перед навигацией на карточку.
|
||||
# В отличие от Cian, здесь нет verify_session (потребовал бы реального browser-
|
||||
# фетча, не curl_cffi) — MVP просто фильтрует+сохраняет, доверяя caller'у.
|
||||
|
||||
|
||||
@router.post("/scrape/domclick/upload-cookies", status_code=200)
|
||||
async def upload_domclick_cookies(
|
||||
cookies: dict[str, str],
|
||||
db: Annotated[Session, Depends(get_db)],
|
||||
) -> dict:
|
||||
"""Upload DomClick session cookies (Sber ID) для обхода QRATOR anti-bot блока.
|
||||
|
||||
Body: JSON object вида { "CAS_ID": "...", "qrator_jsid2": "...", ... } — плоский
|
||||
name→value dict (caller уже флэттенит сырой Cookie-Editor export до этой формы;
|
||||
полный JSON-массив формат Cookie-Editor здесь НЕ парсим).
|
||||
Cookies экспортируются из Chrome DevTools → Application → Cookies →
|
||||
*.domclick.ru или через расширение Cookie-Editor → Export → JSON.
|
||||
|
||||
Шаги:
|
||||
1. Фильтрует payload по DOMCLICK_REQUIRED_COOKIES.
|
||||
2. Извлекает account_cas_id из cookies["CAS_ID"] (int).
|
||||
3. Сохраняет зашифрованно (pgp_sym_encrypt) в domclick_session_cookies.
|
||||
|
||||
MVP: без verify — в отличие от Cian, DomClick verification требует реального
|
||||
browser-фетча (future enhancement), не простого curl_cffi-запроса.
|
||||
|
||||
Returns: {"ok": true, "accountCasId": <int>, "cookieCount": <int>}
|
||||
"""
|
||||
if not settings.cookie_encryption_key:
|
||||
raise HTTPException(status_code=503, detail="COOKIE_ENCRYPTION_KEY not configured")
|
||||
|
||||
cleaned = {
|
||||
k: v for k, v in cookies.items() if k in domclick_session_svc.DOMCLICK_REQUIRED_COOKIES
|
||||
}
|
||||
if not cleaned:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=(
|
||||
f"No recognized DomClick cookies in payload. "
|
||||
f"Expected any of: {sorted(domclick_session_svc.DOMCLICK_REQUIRED_COOKIES)}"
|
||||
),
|
||||
)
|
||||
|
||||
raw_cas_id = cookies.get("CAS_ID")
|
||||
if raw_cas_id is None:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Missing CAS_ID cookie — cannot derive account id"
|
||||
)
|
||||
try:
|
||||
account_cas_id = int(raw_cas_id)
|
||||
except (TypeError, ValueError):
|
||||
raise HTTPException(status_code=400, detail="CAS_ID cookie is not numeric") from None
|
||||
|
||||
domclick_session_svc.save_session(db, account_cas_id=account_cas_id, cookies=cleaned)
|
||||
return {"ok": True, "accountCasId": account_cas_id, "cookieCount": len(cleaned)}
|
||||
|
||||
|
||||
class DomClickDebugDetailFetchRequest(BaseModel):
|
||||
card_url: str
|
||||
|
||||
|
||||
class DomClickDebugDetailFetchResponse(BaseModel):
|
||||
ok: bool
|
||||
card_url: str
|
||||
cookies_injected: bool
|
||||
item_id: str | None
|
||||
repair_state: str | None
|
||||
living_area_m2: float | None
|
||||
year_built: int | None
|
||||
price_changes_count: int
|
||||
raw_extra: dict[str, Any]
|
||||
|
||||
|
||||
@router.post("/scrape/domclick/debug/detail-fetch", response_model=DomClickDebugDetailFetchResponse)
|
||||
async def debug_domclick_detail_fetch(
|
||||
body: DomClickDebugDetailFetchRequest,
|
||||
db: Annotated[Session, Depends(get_db)],
|
||||
) -> DomClickDebugDetailFetchResponse:
|
||||
"""Ad-hoc fetch одной DomClick detail-карточки с cookie-инъекцией (debug, #2000).
|
||||
|
||||
Единственный СЕЙЧАС живой путь прогнать связку domclick_session.load_session +
|
||||
BrowserFetcher(cookies=...) + scraper_kit.providers.domclick.detail.fetch_detail —
|
||||
боевой backfill-оркестратор для DomClick ещё не смёржен (issue #2000).
|
||||
|
||||
cookies=None (валидной session нет в БД) — fetch без инъекции, остаётся только
|
||||
origin-warmup через SERP (#2430). Debug-only, ничего не пишет в БД.
|
||||
"""
|
||||
_assert_domclick_host(body.card_url)
|
||||
|
||||
# Local import — зеркалит паттерн scrape_cian_detail/scrape_cian_newbuilding в
|
||||
# этом же файле: domclick.detail.fetch_detail тёзка module-level avito fetch_detail
|
||||
# (строка 51), local import ограничивает shadowing только этой функцией.
|
||||
from scraper_kit.providers.domclick.detail import fetch_detail as domclick_fetch_detail
|
||||
|
||||
cookies = domclick_session_svc.load_session(db)
|
||||
|
||||
# source="domclick" — выделенный резидентный прокси (scrape_proxies.id=1,
|
||||
# provider_affinity='domclick', см. 173_scrape_proxies_add_domclick_affinity.sql)
|
||||
# построен и verified live именно 2026-07-04 (815КБ __SSR_STATE__ через свежий IP).
|
||||
# Заменяет прежний source="cian" (docstring domclick/detail.py — устаревшая
|
||||
# рекомендация от 2026-06-27, до появления выделенного пула).
|
||||
async with BrowserFetcher(source="domclick", endpoint=settings.browser_http_endpoint) as bf:
|
||||
try:
|
||||
enrichment = await domclick_fetch_detail(
|
||||
body.card_url, browser_fetcher=bf, cookies=cookies
|
||||
)
|
||||
except Exception as e:
|
||||
raise _safe_http_error(
|
||||
502,
|
||||
"fetch failed",
|
||||
f"domclick-debug-detail-fetch: fetch failed for {body.card_url}",
|
||||
) from e
|
||||
|
||||
return DomClickDebugDetailFetchResponse(
|
||||
ok=True,
|
||||
card_url=body.card_url,
|
||||
cookies_injected=cookies is not None,
|
||||
item_id=enrichment.item_id,
|
||||
repair_state=enrichment.repair_state,
|
||||
living_area_m2=enrichment.living_area_m2,
|
||||
year_built=enrichment.year_built,
|
||||
price_changes_count=len(enrichment.price_changes),
|
||||
raw_extra=enrichment.raw_extra,
|
||||
)
|
||||
|
||||
|
||||
# ── Geocode backfill: batch address-dedup geocoder (all sources) ─────────────
|
||||
|
||||
|
||||
|
|
|
|||
161
tradein-mvp/backend/app/services/domclick_session.py
Normal file
161
tradein-mvp/backend/app/services/domclick_session.py
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
"""DomClick session cookie management — load/save/invalidate encrypted cookies.
|
||||
|
||||
Empirically verified live (2026-07-04): DomClick's QRATOR anti-bot reputation-block
|
||||
(даже на прокси-IP, уже помеченном как suspicious) полностью обходится инъекцией
|
||||
cookies валидной аутентифицированной test-аккаунт сессии (Sber ID login) перед
|
||||
навигацией на карточку. Cookies хранятся зашифрованно (pgp_sym_encrypt) в
|
||||
domclick_session_cookies (зеркалит cian_session_cookies, но account_cas_id вместо
|
||||
account_user_id).
|
||||
|
||||
MVP: без verify_session (в отличие от app.services.cian_session). DomClick-
|
||||
верификация требует реального browser-фетча (SSR-стейт страницы), а не простого
|
||||
curl_cffi-запроса как у Cian Valuation Calculator — вне рамок этой итерации,
|
||||
future enhancement.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Cookies критичные для DomClick auth (Sber ID) — фильтр перед сохранением.
|
||||
# Список составлен по реальному DevTools/Cookie-Editor дампу авторизованной
|
||||
# test-аккаунт сессии (Sber ID login), 2026-07-04.
|
||||
DOMCLICK_REQUIRED_COOKIES: set[str] = {
|
||||
"qrator_jsid2",
|
||||
"CAS_ID",
|
||||
"CAS_ID_SIGNED",
|
||||
"sessionId",
|
||||
"UNIQ_SESSION_ID",
|
||||
"_visitId",
|
||||
"ns_session",
|
||||
"RETENTION_COOKIES_NAME",
|
||||
"regionName",
|
||||
"region",
|
||||
"currentRegionGuid",
|
||||
"currentLocalityGuid",
|
||||
"adtech_uid",
|
||||
"ftgl_cookie_id",
|
||||
"_ym_uid",
|
||||
"_ym_d",
|
||||
}
|
||||
|
||||
|
||||
def save_session(
|
||||
db: Session,
|
||||
account_cas_id: int,
|
||||
cookies: dict[str, str],
|
||||
ttl_days: int = 30,
|
||||
) -> None:
|
||||
"""Encrypt cookies via pgp_sym_encrypt and UPSERT into domclick_session_cookies.
|
||||
|
||||
Uses settings.cookie_encryption_key as the encryption secret.
|
||||
Никогда не логирует сырые значения cookies.
|
||||
"""
|
||||
cookies_json = json.dumps(cookies)
|
||||
db.execute(
|
||||
text("""
|
||||
INSERT INTO domclick_session_cookies (
|
||||
account_cas_id,
|
||||
cookies_encrypted,
|
||||
expires_at_estimate,
|
||||
uploaded_at
|
||||
) VALUES (
|
||||
CAST(:cas_id AS bigint),
|
||||
pgp_sym_encrypt(:cookies_json, :key),
|
||||
NOW() + (CAST(:ttl_days AS int) || ' days')::interval,
|
||||
NOW()
|
||||
)
|
||||
ON CONFLICT (account_cas_id) DO UPDATE SET
|
||||
cookies_encrypted = EXCLUDED.cookies_encrypted,
|
||||
expires_at_estimate = EXCLUDED.expires_at_estimate,
|
||||
uploaded_at = NOW(),
|
||||
last_invalid_at = NULL
|
||||
"""),
|
||||
{
|
||||
"cas_id": account_cas_id,
|
||||
"cookies_json": cookies_json,
|
||||
"key": settings.cookie_encryption_key,
|
||||
"ttl_days": ttl_days,
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
logger.info(
|
||||
"DomClick cookies saved for casId=%s (count=%d, ttl=%d days)",
|
||||
account_cas_id,
|
||||
len(cookies),
|
||||
ttl_days,
|
||||
)
|
||||
|
||||
|
||||
def load_session(db: Session) -> dict[str, str] | None:
|
||||
"""Load most-recently-uploaded valid DomClick cookies (decrypt).
|
||||
|
||||
Returns dict[cookie_name, cookie_value] или None если нет валидной session.
|
||||
Выбирает только записи где expires_at_estimate > NOW() и сессия не
|
||||
была инвалидирована после последнего upload'а.
|
||||
"""
|
||||
row = (
|
||||
db.execute(
|
||||
text("""
|
||||
SELECT
|
||||
account_cas_id,
|
||||
pgp_sym_decrypt(cookies_encrypted, :key)::text AS cookies_json,
|
||||
expires_at_estimate
|
||||
FROM domclick_session_cookies
|
||||
WHERE expires_at_estimate > NOW()
|
||||
AND (last_invalid_at IS NULL OR last_invalid_at < uploaded_at)
|
||||
ORDER BY uploaded_at DESC
|
||||
LIMIT 1
|
||||
"""),
|
||||
{"key": settings.cookie_encryption_key},
|
||||
)
|
||||
.mappings()
|
||||
.first()
|
||||
)
|
||||
|
||||
if row is None:
|
||||
logger.warning("No valid DomClick session cookies in DB")
|
||||
return None
|
||||
|
||||
cookies: dict[str, str] = json.loads(row["cookies_json"])
|
||||
|
||||
# Обновляем last_used_at — не критично, игнорируем ошибки.
|
||||
try:
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE domclick_session_cookies SET last_used_at = NOW()"
|
||||
" WHERE account_cas_id = CAST(:cas_id AS bigint)"
|
||||
),
|
||||
{"cas_id": row["account_cas_id"]},
|
||||
)
|
||||
db.commit()
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to update last_used_at for casId=%s: %s", row["account_cas_id"], exc)
|
||||
|
||||
logger.info(
|
||||
"DomClick cookies loaded for casId=%s (count=%d)",
|
||||
row["account_cas_id"],
|
||||
len(cookies),
|
||||
)
|
||||
return cookies
|
||||
|
||||
|
||||
def mark_session_invalid(db: Session, account_cas_id: int) -> None:
|
||||
"""Flag session как expired/invalid (например после блока во время scrape)."""
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE domclick_session_cookies SET last_invalid_at = NOW()"
|
||||
" WHERE account_cas_id = CAST(:cas_id AS bigint)"
|
||||
),
|
||||
{"cas_id": account_cas_id},
|
||||
)
|
||||
db.commit()
|
||||
logger.warning("DomClick session marked invalid for casId=%s", account_cas_id)
|
||||
|
|
@ -0,0 +1,46 @@
|
|||
-- 174_domclick_session_cookies.sql
|
||||
-- Purpose: Encrypted storage for DomClick browser session cookies needed to
|
||||
-- bypass QRATOR anti-bot reputation blocks on card-detail fetches.
|
||||
-- A manually-uploaded authenticated DomClick session (Sber ID login)
|
||||
-- whose cookies (notably qrator_jsid2, CAS_ID, CAS_ID_SIGNED,
|
||||
-- sessionId) let scraper requests pass QRATOR's reputation check.
|
||||
-- Empirically validated 2026-07-04: injecting these cookies via
|
||||
-- Playwright context.addCookies() through a reputation-blocked
|
||||
-- proxy IP fully bypassed the block and returned real
|
||||
-- __SSR_STATE__ content.
|
||||
-- Uses pgcrypto pgp_sym_encrypt for AES encryption at rest.
|
||||
-- Dependencies:
|
||||
-- - pgcrypto extension (installed here via CREATE EXTENSION IF NOT EXISTS)
|
||||
-- Deploy order: Apply after 173.
|
||||
--
|
||||
-- Security notes:
|
||||
-- - cookies_encrypted stores AES-encrypted JSON blob via pgp_sym_encrypt.
|
||||
-- - Encryption key lives in .env.runtime (COOKIE_ENCRYPTION_KEY), never in DB.
|
||||
-- - Access restricted to admin-token-gated API endpoint only.
|
||||
--
|
||||
-- Sources: issue #2000 (DomClick Layer B unblock)
|
||||
|
||||
BEGIN;
|
||||
|
||||
CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS domclick_session_cookies (
|
||||
account_cas_id bigint PRIMARY KEY, -- DomClick CAS_ID cookie (Sber-ID-derived user id)
|
||||
cookies_encrypted bytea NOT NULL, -- pgp_sym_encrypt(json_cookies, key)
|
||||
expires_at_estimate timestamptz NOT NULL, -- estimated expiry (~30 days from upload)
|
||||
uploaded_at timestamptz NOT NULL DEFAULT NOW(),
|
||||
last_used_at timestamptz, -- set on each successful card-detail fetch
|
||||
last_invalid_at timestamptz, -- set when session no longer bypasses QRATOR
|
||||
notes text -- e.g. 'Account: <sber id / email>'
|
||||
);
|
||||
|
||||
-- Most recent upload first (admin dashboard ordering)
|
||||
CREATE INDEX IF NOT EXISTS domclick_cookies_uploaded_idx
|
||||
ON domclick_session_cookies (uploaded_at DESC);
|
||||
|
||||
-- Active session lookup (non-expired and not invalidated)
|
||||
CREATE INDEX IF NOT EXISTS domclick_cookies_active_idx
|
||||
ON domclick_session_cookies (expires_at_estimate)
|
||||
WHERE last_invalid_at IS NULL;
|
||||
|
||||
COMMIT;
|
||||
|
|
@ -316,7 +316,9 @@ async def test_fetch_detail_parses_via_browser() -> None:
|
|||
assert e.item_id == "2075729321"
|
||||
assert e.repair_state == "good"
|
||||
bf.fetch.assert_called_once_with(
|
||||
_CARD_URL, origin="https://ekaterinburg.domclick.ru/pokupka/kvartiry/vtorichka"
|
||||
_CARD_URL,
|
||||
origin="https://ekaterinburg.domclick.ru/pokupka/kvartiry/vtorichka",
|
||||
cookies=None,
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -335,7 +337,30 @@ async def test_fetch_detail_passes_derived_vtorichka_origin() -> None:
|
|||
await fetch_detail(other_card_url, browser_fetcher=bf)
|
||||
|
||||
bf.fetch.assert_called_once_with(
|
||||
other_card_url, origin="https://spb.domclick.ru/pokupka/kvartiry/vtorichka"
|
||||
other_card_url,
|
||||
origin="https://spb.domclick.ru/pokupka/kvartiry/vtorichka",
|
||||
cookies=None,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_fetch_detail_passes_cookies_through_to_browser_fetcher() -> None:
|
||||
"""fetch_detail(cookies=...) пробрасывает cookies в browser_fetcher.fetch без изменений.
|
||||
|
||||
cookies — сессия из app.services.domclick_session.load_session (см. admin debug-роут
|
||||
/scrape/domclick/debug/detail-fetch) — этот модуль сам её не загружает (kit не
|
||||
импортирует app.*), только пробрасывает то, что передал caller.
|
||||
"""
|
||||
bf = MagicMock()
|
||||
bf.fetch = AsyncMock(return_value=_HTML)
|
||||
session_cookies = {"CAS_ID": "12345", "qrator_jsid2": "abc"}
|
||||
|
||||
await fetch_detail(_CARD_URL, browser_fetcher=bf, cookies=session_cookies)
|
||||
|
||||
bf.fetch.assert_called_once_with(
|
||||
_CARD_URL,
|
||||
origin="https://ekaterinburg.domclick.ru/pokupka/kvartiry/vtorichka",
|
||||
cookies=session_cookies,
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
174
tradein-mvp/backend/tests/test_domclick_admin_apis.py
Normal file
174
tradein-mvp/backend/tests/test_domclick_admin_apis.py
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
"""Offline tests для DomClick cookie-injection admin-эндпоинтов (#2000 MVP).
|
||||
|
||||
Покрытие 2 эндпоинтов (db/BrowserFetcher.fetch мокаются, NO live network/DB), зеркалит
|
||||
паттерн test_scraper_admin_apis.py (dependency_overrides[get_db] + TestClient):
|
||||
- POST /api/v1/admin/scrape/domclick/upload-cookies
|
||||
- POST /api/v1/admin/scrape/domclick/debug/detail-fetch
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")
|
||||
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI
|
||||
from fastapi.testclient import TestClient
|
||||
from scraper_kit.providers.domclick.detail import DomClickDetailEnrichment
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client() -> TestClient:
|
||||
from app.api.v1 import admin as admin_module
|
||||
from app.core.db import get_db
|
||||
|
||||
app = FastAPI()
|
||||
app.include_router(admin_module.router, prefix="/api/v1/admin")
|
||||
|
||||
def fake_db():
|
||||
yield MagicMock()
|
||||
|
||||
app.dependency_overrides[get_db] = fake_db
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
_UPLOAD_URL = "/api/v1/admin/scrape/domclick/upload-cookies"
|
||||
_DEBUG_URL = "/api/v1/admin/scrape/domclick/debug/detail-fetch"
|
||||
|
||||
|
||||
# ── POST /scrape/domclick/upload-cookies ──────────────────────────────────────
|
||||
|
||||
|
||||
def test_upload_cookies_503_when_no_encryption_key(client: TestClient) -> None:
|
||||
with patch("app.api.v1.admin.settings.cookie_encryption_key", ""):
|
||||
resp = client.post(_UPLOAD_URL, json={"CAS_ID": "12345"})
|
||||
assert resp.status_code == 503
|
||||
|
||||
|
||||
def test_upload_cookies_400_when_no_recognized_cookies(client: TestClient) -> None:
|
||||
with patch("app.api.v1.admin.settings.cookie_encryption_key", "test-key"):
|
||||
resp = client.post(_UPLOAD_URL, json={"unrelated_cookie": "x"})
|
||||
assert resp.status_code == 400
|
||||
assert "No recognized DomClick cookies" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_upload_cookies_400_when_cas_id_missing(client: TestClient) -> None:
|
||||
with patch("app.api.v1.admin.settings.cookie_encryption_key", "test-key"):
|
||||
resp = client.post(_UPLOAD_URL, json={"qrator_jsid2": "abc"})
|
||||
assert resp.status_code == 400
|
||||
assert "CAS_ID" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_upload_cookies_400_when_cas_id_non_numeric(client: TestClient) -> None:
|
||||
with patch("app.api.v1.admin.settings.cookie_encryption_key", "test-key"):
|
||||
resp = client.post(_UPLOAD_URL, json={"CAS_ID": "not-a-number", "qrator_jsid2": "abc"})
|
||||
assert resp.status_code == 400
|
||||
assert "not numeric" in resp.json()["detail"]
|
||||
|
||||
|
||||
def test_upload_cookies_success_filters_and_calls_save_session(client: TestClient) -> None:
|
||||
with (
|
||||
patch("app.api.v1.admin.settings.cookie_encryption_key", "test-key"),
|
||||
patch("app.api.v1.admin.domclick_session_svc.save_session") as mock_save,
|
||||
):
|
||||
resp = client.post(
|
||||
_UPLOAD_URL,
|
||||
json={"CAS_ID": "12345", "qrator_jsid2": "abc", "unrelated_field": "y"},
|
||||
)
|
||||
assert resp.status_code == 200
|
||||
assert resp.json() == {"ok": True, "accountCasId": 12345, "cookieCount": 2}
|
||||
|
||||
mock_save.assert_called_once()
|
||||
_, kwargs = mock_save.call_args
|
||||
assert kwargs["account_cas_id"] == 12345
|
||||
assert kwargs["cookies"] == {"CAS_ID": "12345", "qrator_jsid2": "abc"}
|
||||
assert "unrelated_field" not in kwargs["cookies"]
|
||||
|
||||
|
||||
# ── POST /scrape/domclick/debug/detail-fetch ──────────────────────────────────
|
||||
|
||||
_CARD_URL = "https://ekaterinburg.domclick.ru/card/sale__flat__2075729321"
|
||||
|
||||
|
||||
def test_debug_detail_fetch_400_when_host_not_allowed(client: TestClient) -> None:
|
||||
"""SSRF guard: хост не *.domclick.ru → 400, никакого fetch не происходит."""
|
||||
resp = client.post(_DEBUG_URL, json={"card_url": "https://evil.example.com/x"})
|
||||
assert resp.status_code == 400
|
||||
|
||||
|
||||
def test_debug_detail_fetch_success_without_session_cookies(client: TestClient) -> None:
|
||||
"""load_session()→None (нет сохранённой сессии) → fetch без инъекции, cookies_injected=False."""
|
||||
enrichment = DomClickDetailEnrichment(
|
||||
item_id="2075729321",
|
||||
source_url=_CARD_URL,
|
||||
repair_state="good",
|
||||
living_area_m2=45.5,
|
||||
year_built=2015,
|
||||
price_changes=[{"change_time": "x", "price_rub": 5_000_000}],
|
||||
raw_extra={"wall_type": "brick"},
|
||||
)
|
||||
with (
|
||||
patch("app.api.v1.admin.domclick_session_svc.load_session", return_value=None),
|
||||
patch(
|
||||
"scraper_kit.providers.domclick.detail.fetch_detail",
|
||||
new=AsyncMock(return_value=enrichment),
|
||||
) as mock_fetch,
|
||||
):
|
||||
resp = client.post(_DEBUG_URL, json={"card_url": _CARD_URL})
|
||||
|
||||
assert resp.status_code == 200
|
||||
body = resp.json()
|
||||
assert body["ok"] is True
|
||||
assert body["cookies_injected"] is False
|
||||
assert body["item_id"] == "2075729321"
|
||||
assert body["repair_state"] == "good"
|
||||
assert body["price_changes_count"] == 1
|
||||
assert body["raw_extra"] == {"wall_type": "brick"}
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
_, kwargs = mock_fetch.call_args
|
||||
assert kwargs["cookies"] is None
|
||||
|
||||
|
||||
def test_debug_detail_fetch_success_with_session_cookies(client: TestClient) -> None:
|
||||
"""load_session() возвращает сессию → cookies_injected=True, cookies проброшены в fetch_detail.
|
||||
|
||||
Не завязан на конкретные HTML/enrichment-детали — cookies-проброс проверяется
|
||||
напрямую через call_args мока fetch_detail.
|
||||
"""
|
||||
session_cookies = {"CAS_ID": "12345", "qrator_jsid2": "abc"}
|
||||
enrichment = DomClickDetailEnrichment(item_id="2075729321", source_url=_CARD_URL)
|
||||
with (
|
||||
patch(
|
||||
"app.api.v1.admin.domclick_session_svc.load_session",
|
||||
return_value=session_cookies,
|
||||
),
|
||||
patch(
|
||||
"scraper_kit.providers.domclick.detail.fetch_detail",
|
||||
new=AsyncMock(return_value=enrichment),
|
||||
) as mock_fetch,
|
||||
):
|
||||
resp = client.post(_DEBUG_URL, json={"card_url": _CARD_URL})
|
||||
|
||||
assert resp.status_code == 200
|
||||
assert resp.json()["cookies_injected"] is True
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
_, kwargs = mock_fetch.call_args
|
||||
assert kwargs["cookies"] == session_cookies
|
||||
|
||||
|
||||
def test_debug_detail_fetch_502_on_fetch_failure(client: TestClient) -> None:
|
||||
with (
|
||||
patch("app.api.v1.admin.domclick_session_svc.load_session", return_value=None),
|
||||
patch(
|
||||
"scraper_kit.providers.domclick.detail.fetch_detail",
|
||||
new=AsyncMock(side_effect=RuntimeError("blocked")),
|
||||
),
|
||||
):
|
||||
resp = client.post(_DEBUG_URL, json={"card_url": _CARD_URL})
|
||||
|
||||
assert resp.status_code == 502
|
||||
167
tradein-mvp/backend/tests/test_domclick_session.py
Normal file
167
tradein-mvp/backend/tests/test_domclick_session.py
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
"""Tests for domclick_session — cookie management service (MVP, #2000).
|
||||
|
||||
Зеркалит test_cian_session.py (MagicMock db, никакого live DB) — без verify_session,
|
||||
т.к. domclick_session MVP её не реализует (см. модуль docstring).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
import pytest
|
||||
|
||||
from app.services.domclick_session import (
|
||||
DOMCLICK_REQUIRED_COOKIES,
|
||||
load_session,
|
||||
mark_session_invalid,
|
||||
save_session,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def mock_db() -> MagicMock:
|
||||
db = MagicMock()
|
||||
# Default: no row found (load_session → None)
|
||||
db.execute.return_value.mappings.return_value.first.return_value = None
|
||||
return db
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DOMCLICK_REQUIRED_COOKIES
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_required_cookies_set_not_empty() -> None:
|
||||
assert isinstance(DOMCLICK_REQUIRED_COOKIES, set)
|
||||
assert len(DOMCLICK_REQUIRED_COOKIES) >= 5
|
||||
|
||||
|
||||
def test_required_cookies_contains_key_names() -> None:
|
||||
assert "CAS_ID" in DOMCLICK_REQUIRED_COOKIES
|
||||
assert "qrator_jsid2" in DOMCLICK_REQUIRED_COOKIES
|
||||
assert "sessionId" in DOMCLICK_REQUIRED_COOKIES
|
||||
assert "_ym_uid" in DOMCLICK_REQUIRED_COOKIES
|
||||
|
||||
|
||||
def test_required_cookies_includes_real_sber_id_dump() -> None:
|
||||
"""Real Sber-ID authenticated test-account DevTools export (2026-07-04)."""
|
||||
real_cookies_from_browser = {
|
||||
"qrator_jsid2",
|
||||
"CAS_ID",
|
||||
"CAS_ID_SIGNED",
|
||||
"sessionId",
|
||||
"UNIQ_SESSION_ID",
|
||||
"_visitId",
|
||||
"ns_session",
|
||||
}
|
||||
missing = real_cookies_from_browser - DOMCLICK_REQUIRED_COOKIES
|
||||
assert not missing, f"Filter missing real cookies: {missing}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# save_session
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_save_session_calls_pgp_sym_encrypt(mock_db: MagicMock) -> None:
|
||||
save_session(mock_db, account_cas_id=12345, cookies={"CAS_ID": "12345"})
|
||||
args, _ = mock_db.execute.call_args
|
||||
sql_text = str(args[0])
|
||||
assert "pgp_sym_encrypt" in sql_text
|
||||
assert "domclick_session_cookies" in sql_text
|
||||
|
||||
|
||||
def test_save_session_uses_cast_not_colon_colon(mock_db: MagicMock) -> None:
|
||||
"""Verify psycopg v3 compatibility — no ::bigint syntax in SQL."""
|
||||
save_session(mock_db, account_cas_id=99, cookies={"CAS_ID": "99"})
|
||||
args, _ = mock_db.execute.call_args
|
||||
sql_text = str(args[0])
|
||||
assert "CAST(" in sql_text
|
||||
assert "::bigint" not in sql_text
|
||||
|
||||
|
||||
def test_save_session_commits(mock_db: MagicMock) -> None:
|
||||
save_session(mock_db, account_cas_id=12345, cookies={"CAS_ID": "12345"})
|
||||
assert mock_db.commit.called
|
||||
|
||||
|
||||
def test_save_session_on_conflict_do_update(mock_db: MagicMock) -> None:
|
||||
save_session(mock_db, account_cas_id=1, cookies={"CAS_ID": "1"})
|
||||
args, _ = mock_db.execute.call_args
|
||||
sql_text = str(args[0])
|
||||
assert "ON CONFLICT" in sql_text
|
||||
assert "DO UPDATE" in sql_text
|
||||
|
||||
|
||||
def test_save_session_passes_correct_params(mock_db: MagicMock) -> None:
|
||||
save_session(mock_db, account_cas_id=777, cookies={"qrator_jsid2": "x"}, ttl_days=14)
|
||||
call_args = mock_db.execute.call_args
|
||||
params = call_args[0][1] if len(call_args[0]) > 1 else call_args[1].get("params", {})
|
||||
assert params["cas_id"] == 777
|
||||
assert params["ttl_days"] == 14
|
||||
cookies_payload = json.loads(params["cookies_json"])
|
||||
assert cookies_payload == {"qrator_jsid2": "x"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# load_session
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_load_session_returns_none_when_empty(mock_db: MagicMock) -> None:
|
||||
result = load_session(mock_db)
|
||||
assert result is None
|
||||
|
||||
|
||||
def test_load_session_decodes_json(mock_db: MagicMock) -> None:
|
||||
mock_row = {
|
||||
"account_cas_id": 12345,
|
||||
"cookies_json": json.dumps({"CAS_ID": "12345", "qrator_jsid2": "abc"}),
|
||||
"expires_at_estimate": None,
|
||||
}
|
||||
mock_db.execute.return_value.mappings.return_value.first.return_value = mock_row
|
||||
result = load_session(mock_db)
|
||||
assert result == {"CAS_ID": "12345", "qrator_jsid2": "abc"}
|
||||
|
||||
|
||||
def test_load_session_updates_last_used_at(mock_db: MagicMock) -> None:
|
||||
mock_row = {
|
||||
"account_cas_id": 42,
|
||||
"cookies_json": json.dumps({"sessionId": "s"}),
|
||||
"expires_at_estimate": None,
|
||||
}
|
||||
mock_db.execute.return_value.mappings.return_value.first.return_value = mock_row
|
||||
load_session(mock_db)
|
||||
# Должно быть минимум 2 вызова execute: SELECT + UPDATE last_used_at
|
||||
assert mock_db.execute.call_count >= 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# mark_session_invalid
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_mark_session_invalid_updates_table(mock_db: MagicMock) -> None:
|
||||
mark_session_invalid(mock_db, account_cas_id=12345)
|
||||
args, _ = mock_db.execute.call_args
|
||||
sql = str(args[0])
|
||||
assert "last_invalid_at" in sql
|
||||
assert "domclick_session_cookies" in sql
|
||||
|
||||
|
||||
def test_mark_session_invalid_commits(mock_db: MagicMock) -> None:
|
||||
mark_session_invalid(mock_db, account_cas_id=99)
|
||||
assert mock_db.commit.called
|
||||
|
||||
|
||||
def test_mark_session_invalid_uses_cast(mock_db: MagicMock) -> None:
|
||||
mark_session_invalid(mock_db, account_cas_id=5)
|
||||
args, _ = mock_db.execute.call_args
|
||||
sql = str(args[0])
|
||||
assert "CAST(" in sql
|
||||
assert "::bigint" not in sql
|
||||
|
|
@ -198,3 +198,37 @@ async def test_fetch_with_origin_sends_it_in_payload() -> None:
|
|||
|
||||
body = client.post.call_args.kwargs["json"]
|
||||
assert body["origin"] == "https://ekaterinburg.domclick.ru/pokupka/kvartiry/vtorichka"
|
||||
|
||||
|
||||
# ── fetch() cookies passthrough (DomClick cookie-injection, #2000) ───────────────
|
||||
|
||||
|
||||
async def test_fetch_without_cookies_sends_none() -> None:
|
||||
"""fetch(url) без cookies → payload содержит "cookies": None (parity avito/cian/yandex).
|
||||
|
||||
Ни один провайдер кроме domclick debug-fetch не передаёт cookies — сервер
|
||||
(body.get("cookies")) трактует None как «инъекции не делать», поведение не меняется.
|
||||
"""
|
||||
client = _mock_client({"html": "<ok>"})
|
||||
bf = _fetcher(client, source="avito")
|
||||
|
||||
html = await bf.fetch("https://avito.ru/x")
|
||||
|
||||
assert html == "<ok>"
|
||||
body = client.post.call_args.kwargs["json"]
|
||||
assert body["cookies"] is None
|
||||
|
||||
|
||||
async def test_fetch_with_cookies_sends_them_in_payload() -> None:
|
||||
"""fetch(url, cookies=X) → payload несёт cookies=X (DomClick session cookie-injection)."""
|
||||
client = _mock_client({"html": "<ok>"})
|
||||
bf = _fetcher(client, source="cian")
|
||||
session_cookies = {"CAS_ID": "12345", "qrator_jsid2": "abc"}
|
||||
|
||||
await bf.fetch(
|
||||
"https://ekaterinburg.domclick.ru/card/sale__flat__1",
|
||||
cookies=session_cookies,
|
||||
)
|
||||
|
||||
body = client.post.call_args.kwargs["json"]
|
||||
assert body["cookies"] == session_cookies
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
экспонирует простой HTTP API на базе aiohttp:
|
||||
|
||||
GET /health → {"status": "ok", "browsers": {"avito": bool, ...}}
|
||||
POST /fetch → {"url","origin"(опц.)} → {"html": "..."}
|
||||
POST /fetch → {"url","origin"(опц.),"cookies"(опц.)} → {"html": "..."}
|
||||
POST /fetch-json → {"url","method","headers","body","origin"} → {"status","body"}
|
||||
POST /login → {"url": "...", "email": "...", "password": "...", ...} → {"cookies": [...]}
|
||||
|
||||
|
|
@ -617,6 +617,13 @@ async def fetch_handler(request: web.Request) -> web.Response:
|
|||
# поведение идентично прежнему (avito/cian/yandex его никогда не передают).
|
||||
origin: str | None = body.get("origin")
|
||||
|
||||
# cookies (опционально) — dict name→value для инъекции в контекст страницы ПЕРЕД
|
||||
# навигацией (обход QRATOR-блока DomClick при валидной test-аккаунт сессии,
|
||||
# эмпирически подтверждено вживую 2026-07-04). Явного auto-derive нет — caller не
|
||||
# передал → остаётся None, инъекции нет, поведение идентично прежнему (avito/cian/
|
||||
# yandex его никогда не передают).
|
||||
cookies: dict | None = body.get("cookies")
|
||||
|
||||
provider = _resolve_provider(body, url)
|
||||
proxy_override = _resolve_proxy_override(body, provider)
|
||||
|
||||
|
|
@ -637,7 +644,7 @@ async def fetch_handler(request: web.Request) -> web.Response:
|
|||
)
|
||||
|
||||
try:
|
||||
html = await _do_fetch(provider, url, origin=origin)
|
||||
html = await _do_fetch(provider, url, origin=origin, cookies=cookies)
|
||||
except Exception as exc:
|
||||
logger.error(
|
||||
"tradein-browser[%s]: fetch error url=%r: %s: %s",
|
||||
|
|
@ -776,17 +783,23 @@ async def _pace_provider(provider: str) -> None:
|
|||
_last_goto_at[provider] = asyncio.get_event_loop().time()
|
||||
|
||||
|
||||
async def _do_fetch(provider: str, url: str, *, origin: str | None = None) -> str:
|
||||
async def _do_fetch(
|
||||
provider: str,
|
||||
url: str,
|
||||
*,
|
||||
origin: str | None = None,
|
||||
cookies: dict | None = None,
|
||||
) -> str:
|
||||
"""Одна попытка навигации; при краше браузера — relaunch и один retry.
|
||||
|
||||
Caller держит _locks[provider] (нет параллельных страниц на этом инстансе),
|
||||
поэтому relaunch безопасен. Crash-relaunch сохраняет текущий прокси инстанса
|
||||
(_relaunch_browser reuse _launched_proxy) — динамический прокси пула не теряется.
|
||||
|
||||
origin — см. _fetch_once. None (дефолт) не меняет поведение.
|
||||
origin/cookies — см. _fetch_once. None (дефолт) не меняет поведение.
|
||||
"""
|
||||
try:
|
||||
return await _fetch_once(provider, url, origin=origin)
|
||||
return await _fetch_once(provider, url, origin=origin, cookies=cookies)
|
||||
except Exception as exc:
|
||||
if _is_browser_crash(exc):
|
||||
logger.warning(
|
||||
|
|
@ -798,11 +811,17 @@ async def _do_fetch(provider: str, url: str, *, origin: str | None = None) -> st
|
|||
await _relaunch_browser(provider)
|
||||
if _browsers.get(provider) is None:
|
||||
raise
|
||||
return await _fetch_once(provider, url, origin=origin)
|
||||
return await _fetch_once(provider, url, origin=origin, cookies=cookies)
|
||||
raise
|
||||
|
||||
|
||||
async def _fetch_once(provider: str, url: str, *, origin: str | None = None) -> str:
|
||||
async def _fetch_once(
|
||||
provider: str,
|
||||
url: str,
|
||||
*,
|
||||
origin: str | None = None,
|
||||
cookies: dict | None = None,
|
||||
) -> str:
|
||||
"""Открывает СОБСТВЕННУЮ страницу, переходит по URL, ждёт JS, возвращает HTML.
|
||||
|
||||
Caller держит _locks[provider], поэтому страницы на этом инстансе не
|
||||
|
|
@ -813,12 +832,30 @@ async def _fetch_once(provider: str, url: str, *, origin: str | None = None) ->
|
|||
вместо холодного прямого захода (зеркалит _fetch_json_once, #1917 — DomClick
|
||||
card-fetch, эмпирически подтверждено вживую 2026-07-04). None (дефолт) → поведение
|
||||
не меняется, ровно один goto(url) как раньше (avito/cian/yandex не передают origin).
|
||||
|
||||
cookies (опционально) — dict cookie_name→value для инъекции в контекст страницы
|
||||
ДО любой навигации (обход QRATOR-блока DomClick при валидной test-аккаунт сессии,
|
||||
эмпирически подтверждено вживую 2026-07-04). Провайдер-агностично: домен НЕ
|
||||
захардкожен, а выводится из hostname целевого url (с ведущей точкой — покрывает
|
||||
поддомены, зеркалит реальную DomClick cookie-scope ".domclick.ru" для
|
||||
ekaterinburg.domclick.ru/spb.domclick.ru/etc, но работает для любого хоста, не
|
||||
только DomClick) — механизм пригоден для будущих caller'ов (Cian/Avito/Yandex).
|
||||
None (дефолт) → без инъекции, поведение не меняется (avito/cian/yandex сейчас
|
||||
cookies не передают).
|
||||
"""
|
||||
browser = _browsers.get(provider)
|
||||
assert browser is not None, "browser not launched"
|
||||
|
||||
page = await browser.new_page() # type: ignore[attr-defined]
|
||||
try:
|
||||
if cookies:
|
||||
cookie_domain = f".{urlparse(url).hostname or ''}"
|
||||
await page.context.add_cookies( # type: ignore[attr-defined]
|
||||
[
|
||||
{"name": name, "value": value, "domain": cookie_domain, "path": "/"}
|
||||
for name, value in cookies.items()
|
||||
]
|
||||
)
|
||||
await _apply_resource_block(page)
|
||||
await _pace_provider(provider)
|
||||
if origin:
|
||||
|
|
|
|||
|
|
@ -202,7 +202,14 @@ class _OverlapProbe:
|
|||
self.global_active = 0
|
||||
self.global_peak = 0
|
||||
|
||||
async def __call__(self, provider: str, url: str, *, origin: str | None = None) -> str:
|
||||
async def __call__(
|
||||
self,
|
||||
provider: str,
|
||||
url: str,
|
||||
*,
|
||||
origin: str | None = None,
|
||||
cookies: dict | None = None,
|
||||
) -> str:
|
||||
self.active[provider] = self.active.get(provider, 0) + 1
|
||||
self.peak[provider] = max(self.peak.get(provider, 0), self.active[provider])
|
||||
self.global_active += 1
|
||||
|
|
|
|||
|
|
@ -294,12 +294,23 @@ def test_login_closes_page_on_cancelled_error(
|
|||
# domclick detail) должен остаться РОВНО одним goto(url), без регрессии.
|
||||
|
||||
|
||||
class _FakeCookieContext:
|
||||
"""Поддельный BrowserContext: фиксирует add_cookies-вызовы (порядок + payload)."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.add_cookies_calls: list[list[dict]] = []
|
||||
|
||||
async def add_cookies(self, cookies: list[dict]) -> None:
|
||||
self.add_cookies_calls.append(cookies)
|
||||
|
||||
|
||||
class _OriginTrackingPage:
|
||||
"""Поддельная page: фиксирует ВСЕ goto-URL по порядку вызова."""
|
||||
"""Поддельная page: фиксирует ВСЕ goto-URL по порядку вызова + cookie-инъекции."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.goto_urls: list[str] = []
|
||||
self.closed = 0
|
||||
self.context = _FakeCookieContext()
|
||||
|
||||
async def route(self, pattern: str, handler: Any) -> None:
|
||||
return None
|
||||
|
|
@ -356,6 +367,85 @@ def test_fetch_once_single_goto_when_origin_none(monkeypatch: pytest.MonkeyPatch
|
|||
assert page.goto_urls == ["https://www.avito.ru/card/1"]
|
||||
|
||||
|
||||
# ── _fetch_once: cookies-инъекция ТОЛЬКО когда передана (DomClick QRATOR-bypass) ──
|
||||
# Провайдер-агностично: domain НЕ захардкожен ".domclick.ru" — выводится из
|
||||
# urlparse(url).hostname (с ведущей точкой) прямо в _fetch_once, поэтому механизм
|
||||
# годится для будущих caller'ов (Cian/Avito/Yandex), не только DomClick.
|
||||
# cookies=None (дефолт, все providers сейчас) должен остаться без add_cookies-вызова.
|
||||
|
||||
|
||||
def test_fetch_once_injects_cookies_before_goto_with_derived_domain(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""cookies передан → context.add_cookies() ДО goto(url), domain = "." + hostname."""
|
||||
page = _OriginTrackingPage()
|
||||
server._browsers["cian"] = _OriginTrackingBrowser(page)
|
||||
monkeypatch.setattr(server, "BROWSER_RECYCLE_PAGES", 10_000)
|
||||
|
||||
html = asyncio.run(
|
||||
server._fetch_once(
|
||||
"cian",
|
||||
"https://ekaterinburg.domclick.ru/card/sale__flat__1",
|
||||
cookies={"CAS_ID": "12345", "qrator_jsid2": "abc"},
|
||||
)
|
||||
)
|
||||
|
||||
assert html == "<html>ok</html>"
|
||||
assert len(page.context.add_cookies_calls) == 1
|
||||
injected = page.context.add_cookies_calls[0]
|
||||
assert {
|
||||
"name": "CAS_ID",
|
||||
"value": "12345",
|
||||
"domain": ".ekaterinburg.domclick.ru",
|
||||
"path": "/",
|
||||
} in injected
|
||||
assert {
|
||||
"name": "qrator_jsid2",
|
||||
"value": "abc",
|
||||
"domain": ".ekaterinburg.domclick.ru",
|
||||
"path": "/",
|
||||
} in injected
|
||||
# Инъекция ДО навигации на целевой url — единственный goto остаётся ровно url.
|
||||
assert page.goto_urls == ["https://ekaterinburg.domclick.ru/card/sale__flat__1"]
|
||||
|
||||
|
||||
def test_fetch_once_no_cookie_injection_when_cookies_none(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""cookies=None (дефолт) → add_cookies НЕ вызывается, поведение не меняется."""
|
||||
page = _OriginTrackingPage()
|
||||
server._browsers["avito"] = _OriginTrackingBrowser(page)
|
||||
monkeypatch.setattr(server, "BROWSER_RECYCLE_PAGES", 10_000)
|
||||
|
||||
asyncio.run(server._fetch_once("avito", "https://www.avito.ru/card/1"))
|
||||
|
||||
assert page.context.add_cookies_calls == []
|
||||
|
||||
|
||||
def test_fetch_once_cookie_domain_derivation_is_generic_not_domclick_hardcoded(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Domain derivation работает для ЛЮБОГО хоста, не только domclick.ru.
|
||||
|
||||
Доказывает, что механизм провайдер-агностичен: hardcode ".domclick.ru" НЕ
|
||||
используется — тот же код с avito.ru url выводит ".www.avito.ru".
|
||||
"""
|
||||
page = _OriginTrackingPage()
|
||||
server._browsers["avito"] = _OriginTrackingBrowser(page)
|
||||
monkeypatch.setattr(server, "BROWSER_RECYCLE_PAGES", 10_000)
|
||||
|
||||
asyncio.run(
|
||||
server._fetch_once(
|
||||
"avito", "https://www.avito.ru/items/1", cookies={"sessionid": "xyz"}
|
||||
)
|
||||
)
|
||||
|
||||
injected = page.context.add_cookies_calls[0]
|
||||
assert injected == [
|
||||
{"name": "sessionid", "value": "xyz", "domain": ".www.avito.ru", "path": "/"}
|
||||
]
|
||||
|
||||
|
||||
async def _coro(value: Any) -> Any:
|
||||
"""Хелпер: оборачивает значение в awaitable для подмены request.json()."""
|
||||
return value
|
||||
|
|
|
|||
|
|
@ -95,7 +95,13 @@ class BrowserFetcher:
|
|||
|
||||
# ── public API ─────────────────────────────────────────────────────────────
|
||||
|
||||
async def fetch(self, url: str, *, origin: str | None = None) -> str:
|
||||
async def fetch(
|
||||
self,
|
||||
url: str,
|
||||
*,
|
||||
origin: str | None = None,
|
||||
cookies: dict[str, str] | None = None,
|
||||
) -> str:
|
||||
"""Запрашивает HTML страницы через tradein-browser HTTP-сервис.
|
||||
|
||||
origin — same-site якорь, на который камуфокс зайдёт ПЕРЕД url (органическая
|
||||
|
|
@ -103,6 +109,12 @@ class BrowserFetcher:
|
|||
``origin`` в server.py. None (дефолт) → поведение не меняется (ровно один
|
||||
goto(url), как раньше) — таков путь всех providers кроме domclick detail.
|
||||
|
||||
cookies — dict cookie_name→value для инъекции в browser-контекст ПЕРЕД
|
||||
навигацией (обходит QRATOR-блок DomClick при валидной test-аккаунт
|
||||
сессии, empирически подтверждено вживую 2026-07-04, см. app.services.
|
||||
domclick_session). None (дефолт) → поведение не меняется, инъекции нет —
|
||||
таков путь всех providers кроме domclick detail-debug.
|
||||
|
||||
При HTTPError или ConnectError делает одну повторную попытку после
|
||||
короткой паузы. Остальные исключения всплывают к вызывающему коду.
|
||||
|
||||
|
|
@ -113,7 +125,7 @@ class BrowserFetcher:
|
|||
assert self._endpoint is not None, "BrowserFetcher: endpoint не задан"
|
||||
|
||||
try:
|
||||
return await self._post_fetch(url, origin)
|
||||
return await self._post_fetch(url, origin, cookies)
|
||||
except (httpx.HTTPError, httpx.TransportError) as exc:
|
||||
logger.warning(
|
||||
"BrowserFetcher: ошибка запроса (%s), retry через %.1fs: %s",
|
||||
|
|
@ -122,7 +134,7 @@ class BrowserFetcher:
|
|||
url,
|
||||
)
|
||||
await asyncio.sleep(_RETRY_SLEEP_S)
|
||||
return await self._post_fetch(url, origin)
|
||||
return await self._post_fetch(url, origin, cookies)
|
||||
|
||||
async def fetch_json(
|
||||
self,
|
||||
|
|
@ -262,17 +274,28 @@ class BrowserFetcher:
|
|||
exc_info=True,
|
||||
)
|
||||
|
||||
async def _post_fetch(self, url: str, origin: str | None = None) -> str:
|
||||
async def _post_fetch(
|
||||
self,
|
||||
url: str,
|
||||
origin: str | None = None,
|
||||
cookies: dict[str, str] | None = None,
|
||||
) -> str:
|
||||
"""Один HTTP POST к /fetch эндпоинту сервиса.
|
||||
|
||||
origin всегда кладём в payload (даже None) — зеркалит _post_fetch_json,
|
||||
сервер (body.get("origin")) корректно обрабатывает оба случая.
|
||||
origin/cookies всегда кладём в payload (даже None) — зеркалит
|
||||
_post_fetch_json, сервер (body.get("origin")/body.get("cookies"))
|
||||
корректно обрабатывает оба случая.
|
||||
"""
|
||||
assert self._client is not None
|
||||
assert self._endpoint is not None
|
||||
|
||||
with self._pool_proxy() as (proxy_url, proxy_kind):
|
||||
payload: dict = {"url": url, "source": self._source, "origin": origin}
|
||||
payload: dict = {
|
||||
"url": url,
|
||||
"source": self._source,
|
||||
"origin": origin,
|
||||
"cookies": cookies,
|
||||
}
|
||||
if proxy_url:
|
||||
payload["proxy"] = proxy_url
|
||||
if proxy_kind:
|
||||
|
|
|
|||
|
|
@ -466,12 +466,21 @@ async def fetch_detail(
|
|||
card_url: str,
|
||||
*,
|
||||
browser_fetcher: BrowserFetcher,
|
||||
cookies: dict[str, str] | None = None,
|
||||
) -> DomClickDetailEnrichment:
|
||||
"""Fetch абсолютного URL карточки через BrowserFetcher → parse_detail_html.
|
||||
|
||||
card_url уже абсолютный (listings.source_url из Layer A), URL не строим.
|
||||
Сессией владеет оркестратор (передаёт уже открытый browser_fetcher).
|
||||
|
||||
cookies — опционально: dict cookie_name→value авторизованной test-аккаунт
|
||||
сессии (Sber ID) для инъекции в browser-контекст ДО навигации — полностью
|
||||
обходит QRATOR-блок даже на подозрительном прокси-IP (эмпирически подтверждено
|
||||
вживую 2026-07-04). Caller (app.api.v1.admin debug-роут) сам вызывает
|
||||
app.services.domclick_session.load_session и передаёт результат сюда — этот
|
||||
модуль НЕ импортирует app.* (scraper_kit invariant). None (дефолт, сессии нет
|
||||
в БД) → fetch без инъекции, остаётся только SERP-warmup ниже.
|
||||
|
||||
Raises:
|
||||
DomClickBlockedError: anti-bot challenge ИЛИ ошибка браузерного fetch
|
||||
(нет curl-fallback — DataDome даёт 401 на curl). Оркестратор считает
|
||||
|
|
@ -483,7 +492,7 @@ async def fetch_detail(
|
|||
try:
|
||||
# Заход на same-site vtorichka-SERP ПЕРЕД карточкой снижает подозрительность
|
||||
# холодной навигации для QRATOR (эмпирически подтверждено вживую 2026-07-04).
|
||||
html = await browser_fetcher.fetch(card_url, origin=origin)
|
||||
html = await browser_fetcher.fetch(card_url, origin=origin, cookies=cookies)
|
||||
except DomClickBlockedError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue