gendesign/backend
lekss361 1ccb44687c fix(scrapers): bypass DOM.РФ WAF + update renamed extras endpoints
DOM.РФ накатил WAF между 2026-05-17 и 2026-05-24 на /сервисы/api/object/*/*.
Без session cookies (___dmpkit___, domain_sid) — все запросы → 403 + WAF
challenge HTML. Cookies выдаются после visit на /сервисы/каталог-новостроек/.

Дополнительно DOM.РФ переименовал 6 endpoints (photos, 5 × doc_*) — старые
URLs возвращают тот же WAF catch-all.

Run 26 (2026-05-24): 15,340 fail requests (1534 obj × 10 endpoints).
Extras snapshots не обновлялись с 2026-05-17.

- stealth.py: BrowserSession.warm_up() — visit catalog listing, кэшит WAF cookies
- domrf_kn.py: warm_up() перед Phase A/B/C (покрывает все extras запросы)
- domrf_catalog_object.py: warm_up() перед SSR HTML fetch batch
2026-05-24 13:26:22 +03:00
..
alembic ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
app fix(scrapers): bypass DOM.РФ WAF + update renamed extras endpoints 2026-05-24 13:26:22 +03:00
db/init ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
output feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
tests feat(22d): catalog scraper — skip-today filter + force flag (#348) 2026-05-17 23:15:22 +00:00
.dockerignore fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
.env.example refactor(security): убрать X-Admin-Token (Caddy basic_auth достаточен) (#437) 2026-05-23 10:41:22 +00:00
.env.runtime.example sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
alembic.ini ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
debug.log feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
Dockerfile fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
pyproject.toml feat(sf-b7): GET /parcels/{cad}/snapshot.pdf — 1-page WeasyPrint export (#334) 2026-05-17 21:11:03 +00:00
uv.lock feat(sf-b7): GET /parcels/{cad}/snapshot.pdf — 1-page WeasyPrint export (#334) 2026-05-17 21:11:03 +00:00