feat(tradein/domclick): production Layer B detail-backfill orchestrator (#2000) #2436

Merged

lekss361 merged 1 commit from feat/tradein-domclick-layerb-orchestrator into main

2026-07-04 19:49:07 +00:00

lekss361 commented

2026-07-04 19:47:20 +00:00

Owner

Summary

Production scheduled orchestrator for DomClick detail-enrichment (Layer B), porting the manual POST /scrape/domclick/debug/detail-fetch wiring (session -> cookies -> fetch_detail) into a nightly scrape_schedules task. Modeled closely on the proven app/tasks/avito_detail_backfill.py.

New app/tasks/domclick_detail_backfill.py -- single-snapshot SELECT (guarantees termination), one BrowserFetcher(source="domclick") per run via async CM, cookies loaded ONCE via domclick_session.load_session(db) and threaded into every fetch_detail(cookies=...). Budget guard + SIGTERM-drain + heartbeat mirror avito.
Handler registration in app/services/product_handlers.py (domclick_detail_backfill).
Migration data/sql/175_scrape_schedules_seed_domclick_detail_backfill.sql -- seeds schedule enabled=false (activation is a deliberate separate manual step post-deploy), window 15:00-18:00 UTC (offset from avito 09-12 / yandex 12-15), ON CONFLICT (source) DO NOTHING. Idiom identical to prod-proven migration 112.
Docstring-only clarification in scraper_kit/providers/domclick/detail.py (QRATOR reputation-block semantics).

Naming split (intentional, not a bug): listings.source = 'domklik' (data identifier, matches Layer A serp.py inserts) in the snapshot WHERE clause; BrowserFetcher(source="domclick") (infra identifier -- proxy affinity, cookie tables) for the fetcher. Both correct in their own context.

Exception triad: DomClickBlockedError -> consecutive-block counter -> abort via mark_done (block = expected operational outcome, not a failure); DomClickParseError (schema drift) -> failed++, does NOT trip the block-breaker; other exceptions -> failed++ + rollback + continue.

Test plan

9 new unit tests pass (tests/tasks/test_domclick_detail_backfill.py).
Broader pass green (103 passed) across domclick / product_handlers / detail_backfill.
ruff clean; CAST discipline OK (CAST(:batch_size AS int), no :x::type).
Post-activation smoke against a live cookie session (deferred -- schedule ships disabled).

Refs #2000

## Summary Production scheduled orchestrator for DomClick detail-enrichment (Layer B), porting the manual `POST /scrape/domclick/debug/detail-fetch` wiring (session -> cookies -> fetch_detail) into a nightly `scrape_schedules` task. Modeled closely on the proven `app/tasks/avito_detail_backfill.py`. - New `app/tasks/domclick_detail_backfill.py` -- single-snapshot SELECT (guarantees termination), one `BrowserFetcher(source="domclick")` per run via async CM, cookies loaded ONCE via `domclick_session.load_session(db)` and threaded into every `fetch_detail(cookies=...)`. Budget guard + SIGTERM-drain + heartbeat mirror avito. - Handler registration in `app/services/product_handlers.py` (`domclick_detail_backfill`). - Migration `data/sql/175_scrape_schedules_seed_domclick_detail_backfill.sql` -- seeds schedule **`enabled=false`** (activation is a deliberate separate manual step post-deploy), window 15:00-18:00 UTC (offset from avito 09-12 / yandex 12-15), `ON CONFLICT (source) DO NOTHING`. Idiom identical to prod-proven migration 112. - Docstring-only clarification in `scraper_kit/providers/domclick/detail.py` (QRATOR reputation-block semantics). **Naming split (intentional, not a bug):** `listings.source = 'domklik'` (data identifier, matches Layer A `serp.py` inserts) in the snapshot WHERE clause; `BrowserFetcher(source="domclick")` (infra identifier -- proxy affinity, cookie tables) for the fetcher. Both correct in their own context. **Exception triad:** `DomClickBlockedError` -> consecutive-block counter -> abort via `mark_done` (block = expected operational outcome, not a failure); `DomClickParseError` (schema drift) -> `failed++`, does NOT trip the block-breaker; other exceptions -> `failed++` + rollback + continue. ## Test plan - [x] 9 new unit tests pass (`tests/tasks/test_domclick_detail_backfill.py`). - [x] Broader pass green (103 passed) across `domclick` / `product_handlers` / `detail_backfill`. - [x] ruff clean; CAST discipline OK (`CAST(:batch_size AS int)`, no `:x::type`). - [ ] Post-activation smoke against a live cookie session (deferred -- schedule ships disabled). Refs #2000

lekss361 added 1 commit 2026-07-04 19:47:20 +00:00

feat(tradein/domclick): production Layer B detail-backfill orchestrator (#2000 )

CI Trade-In / changes (pull_request) Successful in 8s

Details

CI / changes (pull_request) Successful in 8s

Details

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI Trade-In / backend-tests (pull_request) Successful in 53s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

f627b0fa4d

Ports the cookie-injection + organic SERP-origin session->cookies->fetch_detail
wiring (PR #2430, PR #2433), previously only reachable via the manual debug
endpoint, into a scheduled orchestrator mirroring avito_detail_backfill.py's
shape: snapshot-once-then-loop, budget guard, SIGTERM-drain, jittered delay,
heartbeat every 25 attempts.

DomClickBlockedError (QRATOR challenge / browser-fetch failure) increments a
consecutive-block counter and aborts via mark_done (not mark_failed) past
max_consecutive_blocks -- a block is an expected operational outcome for a
QRATOR reputation-based anti-bot, not a task failure. DomClickParseError
(__SSR_STATE__ schema drift) is a plain failed++, neutral to the breaker.
Unlike Avito there is no curl fallback (one BrowserFetcher per run) and no
IP-rotation recovery (single dedicated residential proxy).

Migration 175 seeds the schedule with enabled=false -- activation is a
deliberate manual step after smoke-testing against a live cookie session.
default_params are more conservative than Avito's (batch_size=200,
request_delay_sec=12, max_consecutive_blocks=3) since a burned QRATOR
reputation/session is costlier to recover than an Avito IP-block.

Also fixes a stale docstring in scraper_kit's domclick/detail.py that
incorrectly named DataDome as the anti-bot mechanism -- it's QRATOR
(exit-IP reputation-based), per the extensive live verification in #2000.