fix(scrapers): disable catalog-object SSR beat-cron — DOM.РФ WAF cooldown #516

Merged

lekss361 merged 1 commit from fix/disable-catalog-ssr-waf-cooldown into main

2026-05-24 12:21:11 +00:00

lekss361 commented

2026-05-24 12:16:29 +00:00

Owner

Summary

backend/app/workers/beat_schedule.py:243: закомментирован entry scrape-kn-catalog-objects-weekly (вт 04:00 UTC). Возврат после WAF cooldown 24-48h.

Root cause / context

DOM.РФ дал hard-ban на VPS IP после серии failed extras-сессий:

Run 26 (06:14-07:11 UTC) — 15 340 failures, базовые objects/flats прошли
Run 27 (11:41 UTC, user revoked at 380/1534) — ~4180 failures
Run 28 (12:01-12:03 UTC, targeted 1 obj) — 10 failures на 1 obj, flats тоже WAF-блок (раньше работал!)

Warm_up из PR #503 получает cookies успешно (got WAF cookies ['___dmpkit___', 'domain_sid'] в worker logs), но WAF всё равно возвращает challenge HTML на /portal-kn/api/sales/portal/table и /сервисы/api/object/* для нашего IP. Phase A list /portal-kn/api/kn/objects пока пропускает.

Catalog-object SSR (tasks.scrape_kn_catalog_objects) использует ТОТ ЖЕ BrowserSession + navigates на /сервисы/каталог-новостроек/объект/{id} — точно те же endpoints под WAF. Next beat-tick вт 26.05 04:00 UTC насыпет 300 failed SSR fetches и углубит WAF reputation penalty.

What's already done

UPDATE job_settings SET enabled=false WHERE job_type IN ('scrape_kn','objective_sync') — оба DOM.РФ jobs disabled через DB
Этот PR — закрывает третий DOM.РФ scraper (hardcoded в beat_schedule.py, через DB нельзя)

Re-enable plan

Через 24-48h — targeted test через /admin/scrape Force run на 1 объект
Если 0 failures + extras filled → uncomment этот entry + revert disable в job_settings (UPDATE enabled=true для scrape_kn, objective_sync)
Если по-прежнему WAF → нужен другой подход (proxy / referer fix / contact DOM.РФ)

Test plan

Pre-commit hooks passed (ruff/ruff-format passed)
После merge: следующий deploy успешный
Beat container перезагружен с новым schedule (beat читает build_beat_schedule на старте)
Вт 26.05 04:00 UTC — НЕТ celery task tasks.scrape_kn_catalog_objects.* enqueued (можно проверить через /api/v1/admin/scrape/queue или beat logs)

Risk: LOW

Removed только cron entry, task code не тронут — manual trigger через celery всё ещё работает
Не затрагивает другие schedule entries (NSPD harvest, OSM POI/noise, ekburg permits, PZZ — все остаются на расписании)
Reversible: uncomment 5 строк

## Summary - `backend/app/workers/beat_schedule.py:243`: закомментирован entry `scrape-kn-catalog-objects-weekly` (вт 04:00 UTC). Возврат после WAF cooldown 24-48h. ## Root cause / context DOM.РФ дал hard-ban на VPS IP после серии failed extras-сессий: - **Run 26** (06:14-07:11 UTC) — 15 340 failures, базовые objects/flats прошли - **Run 27** (11:41 UTC, user revoked at 380/1534) — ~4180 failures - **Run 28** (12:01-12:03 UTC, targeted 1 obj) — 10 failures на 1 obj, **flats тоже WAF-блок** (раньше работал!) Warm_up из PR #503 получает cookies успешно (`got WAF cookies ['___dmpkit___', 'domain_sid']` в worker logs), но WAF всё равно возвращает challenge HTML на `/portal-kn/api/sales/portal/table` и `/сервисы/api/object/*` для нашего IP. Phase A list `/portal-kn/api/kn/objects` пока пропускает. Catalog-object SSR (`tasks.scrape_kn_catalog_objects`) использует ТОТ ЖЕ `BrowserSession` + navigates на `/сервисы/каталог-новостроек/объект/{id}` — точно те же endpoints под WAF. Next beat-tick **вт 26.05 04:00 UTC** насыпет 300 failed SSR fetches и углубит WAF reputation penalty. ## What's already done - `UPDATE job_settings SET enabled=false WHERE job_type IN ('scrape_kn','objective_sync')` — оба DOM.РФ jobs disabled через DB - Этот PR — закрывает третий DOM.РФ scraper (hardcoded в beat_schedule.py, через DB нельзя) ## Re-enable plan 1. Через 24-48h — targeted test через `/admin/scrape` Force run на 1 объект 2. Если 0 failures + extras filled → uncomment этот entry + revert disable в `job_settings` (UPDATE enabled=true для scrape_kn, objective_sync) 3. Если по-прежнему WAF → нужен другой подход (proxy / referer fix / contact DOM.РФ) ## Test plan - [x] Pre-commit hooks passed (ruff/ruff-format passed) - [ ] После merge: следующий deploy успешный - [ ] Beat container перезагружен с новым schedule (beat читает build_beat_schedule на старте) - [ ] Вт 26.05 04:00 UTC — НЕТ celery task `tasks.scrape_kn_catalog_objects.*` enqueued (можно проверить через `/api/v1/admin/scrape/queue` или beat logs) ## Risk: LOW - Removed только cron entry, task code не тронут — manual trigger через celery всё ещё работает - Не затрагивает другие schedule entries (NSPD harvest, OSM POI/noise, ekburg permits, PZZ — все остаются на расписании) - Reversible: uncomment 5 строк

lekss361 added 1 commit 2026-05-24 12:16:30 +00:00

fix(scrapers): disable catalog-object SSR beat-cron — DOM.РФ WAF cooldown b0ffca4c41

DOM.РФ накатил WAF hard-ban на VPS IP после серии failed extras-сессий
(run 26/27/28 — суммарно ~15400+ requests с body '<!-- waf -->').

Catalog-object SSR (`tasks.scrape_kn_catalog_objects`) использует тот же
BrowserSession и navigates на /сервисы/каталог-новостроек/объект/{id} —
точно те же endpoints под WAF. Next beat-tick вт 26.05 04:00 UTC насыпет
300 failed SSR fetches и углубит WAF reputation penalty.

Commented out беспрепятственно (без infrastructure change). Возврат после
24-48h WAF cooldown + targeted test через admin/scrape.

Параллельные disable'ы через DB job_settings уже сделаны:
- scrape_kn (enabled=false)
- objective_sync (enabled=false)

lekss361 commented

2026-05-24 12:21:01 +00:00

Author

Owner

Deep Code Review — verdict: APPROVE

Summary

Status: APPROVE (low-risk defensive op)
Files: 1 (P1: backend/app/workers/beat_schedule.py)
Lines: +12 / -6 · pure comment-out, no logic change

Cross-file impact (verified)

backend/app/workers/celery_app.py:52 — task module app.workers.tasks.scrape_kn_catalog_objects still in include= list. Worker can execute on demand, no import-error risk on beat reload.
backend/app/api/v1/admin_scrape.py:1254 — admin endpoint apply_async path still intact. Manual trigger via /admin/scrape works for re-enable test (per PR plan step 1).
backend/tests/workers/test_scrape_kn_catalog_objects.py — untouched, task itself unchanged. No regression coverage needed for beat-toggle.
Neighboring beat entries (pzz-sync-monthly, nspd-quarter-dump-refresh-weekly) intact — clean surgical edit.

What this disables (confirmed)

Beat entry scrape-kn-catalog-objects-weekly (вт 04:00 UTC, batch 300, region 66).
Hits same /сервисы/каталог-новостроек/объект/{id} SSR + shared BrowserSession as failing extras runs 26/27/28 → continuing would deepen WAF reputation penalty. Correct call.
Third DOM.РФ surface closed (along with job_settings rows for scrape_kn + objective_sync already toggled via DB).

Data freshness impact

Catalog SSR fills ~25 nice-to-have columns (wall_type, energy_eff, ceiling_height_m, parking_*). Not on core SF read path. Stale window 24-48h acceptable.
Existing domrf_kn_objects rows continue to be served as-is.

Re-enable plan (well-formed)

Targeted single-object test via /admin/scrape Force run → 0 failures gate → uncomment + revert DB toggles. Reasonable.
Recommend adding follow-up: vault fix-note documenting run 26/27/28 + WAF cooldown threshold for next session continuity (no blocker for this PR).

Positive

Excellent inline rationale block (date, runs cited, mechanism, re-enable steps) — matches house style of NSPD comment above. Future maintainer has full context without git archeology.
Comment-out (vs. delete) is the right pattern for a short cooldown with re-enable intent.

Risk / reversibility

Risk: Very low. Reversible by uncomment + deploy.
Blast radius: scheduler only — no migrations, no API contract, no frontend.
Beat container will pick up new schedule on next deploy restart (per test plan checklist).

Merging.

## Deep Code Review — verdict: APPROVE ### Summary - **Status:** APPROVE (low-risk defensive op) - **Files:** 1 (P1: `backend/app/workers/beat_schedule.py`) - **Lines:** +12 / -6 · pure comment-out, no logic change ### Cross-file impact (verified) - `backend/app/workers/celery_app.py:52` — task module `app.workers.tasks.scrape_kn_catalog_objects` still in `include=` list. Worker can execute on demand, no import-error risk on beat reload. - `backend/app/api/v1/admin_scrape.py:1254` — admin endpoint `apply_async` path still intact. Manual trigger via `/admin/scrape` works for re-enable test (per PR plan step 1). - `backend/tests/workers/test_scrape_kn_catalog_objects.py` — untouched, task itself unchanged. No regression coverage needed for beat-toggle. - Neighboring beat entries (`pzz-sync-monthly`, `nspd-quarter-dump-refresh-weekly`) intact — clean surgical edit. ### What this disables (confirmed) - Beat entry `scrape-kn-catalog-objects-weekly` (вт 04:00 UTC, batch 300, region 66). - Hits same `/сервисы/каталог-новостроек/объект/{id}` SSR + shared `BrowserSession` as failing extras runs 26/27/28 → continuing would deepen WAF reputation penalty. Correct call. - Third DOM.РФ surface closed (along with `job_settings` rows for `scrape_kn` + `objective_sync` already toggled via DB). ### Data freshness impact - Catalog SSR fills ~25 nice-to-have columns (`wall_type`, `energy_eff`, `ceiling_height_m`, `parking_*`). Not on core SF read path. Stale window 24-48h acceptable. - Existing `domrf_kn_objects` rows continue to be served as-is. ### Re-enable plan (well-formed) - Targeted single-object test via `/admin/scrape` Force run → 0 failures gate → uncomment + revert DB toggles. Reasonable. - Recommend adding follow-up: vault fix-note documenting run 26/27/28 + WAF cooldown threshold for next session continuity (no blocker for this PR). ### Positive - Excellent inline rationale block (date, runs cited, mechanism, re-enable steps) — matches house style of NSPD comment above. Future maintainer has full context without git archeology. - Comment-out (vs. delete) is the right pattern for a short cooldown with re-enable intent. ### Risk / reversibility - Risk: **Very low**. Reversible by uncomment + deploy. - Blast radius: scheduler only — no migrations, no API contract, no frontend. - Beat container will pick up new schedule on next deploy restart (per test plan checklist). Merging.