Commit graph

5 commits

Author SHA1 Message Date
68d0575080 fix(scrapers): WAF re-trigger guard + throttled BrowserSession для DOM.РФ catalog (#2445-D1/D2)
Some checks failed
CI Trade-In / changes (pull_request) Successful in 12s
CI / changes (pull_request) Successful in 11s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Failing after 3m1s
CI / backend-tests (pull_request) Successful in 15m39s
D1: admin-эндпоинты /kn-catalog-objects + /kn-catalog-flats теперь требуют
явный i_understand_waf_risk=true перед dispatch (иначе 400 со ссылкой на #2443),
т.к. их beat отключён после WAF-бана 2026-05-24 и случайный re-trigger углубит бан.

D2: обе catalog-BrowserSession (domrf_catalog_object.py + domrf_catalog.py flat-level)
переиспользуют throttled settings.scrape_kn_* (concurrency=2, jitter 1200-3000ms,
как scrape_kn после #1945) вместо молчаливого наследования дефолта stealth.py
(concurrency=8, 600-1500ms). Обновлены устаревшие комментарии config.py/stealth.py.

Refs #2445
2026-07-06 00:44:19 +05:00
099b8e4b59 fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)
Some checks failed
Deploy / changes (push) Successful in 4s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Failing after 35s
2026-05-24 10:53:28 +00:00
9ffeee7ed2 feat(22d): admin UI buttons (skip/force) + reduce catalog jitter 800→300ms (#352)
All checks were successful
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 1m48s
Deploy / build-worker (push) Successful in 3m1s
Deploy / build-frontend (push) Successful in 3m20s
Deploy / deploy (push) Successful in 55s
2026-05-17 23:38:22 +00:00
lekss361
e5498682ba fix(22d): commit outer transaction in scrape_catalog_objects orchestrator
PR #335 использовал `with db.begin_nested():` SAVEPOINT для per-row UPDATE,
но outer transaction никогда не commit'ился. После цикла Celery task
закрывал session (`db.close()` в finally) → autobegin'd outer tx
rollback → все UPDATE'ы теряются.

Симптом: task завершается с processed=3 succeeded=3, в Glitchtip нет ошибок,
но в БД ни одного catalog_scraped_at != NULL.

Fix: добавлен db.commit() после цикла + rollback на ошибку commit'а.
2026-05-18 01:10:06 +03:00
0567ad2130 feat(22d): domrf_catalog_object scraper — fill ~25 NULL kn_objects cols from SSR __NEXT_DATA__ (#335)
Some checks failed
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m43s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m49s
Deploy / deploy (push) Failing after 44s
2026-05-17 21:26:22 +00:00