8 commits

Author	SHA1	Message	Date
Light1YT	4571938b56	fix(scrapers): throttle + optional proxy levers for KN flats loader (#1945) ... DOM.РФ WAF банит на full sweep (~17k req @ concurrency=8): сессия, тронувшая gated /сервисы/api/object/* extras, получает 403 → cookies сессии флагуются → последующие запросы (incl. flats) тоже 403. Эмпирически (prod worker, throttle@concurrency=2, без прокси): • flats-only, 100 объектов / 104 req — 0 WAF-бан; • mixed flats+extras — extras 403 сразу, затем травит сессию (flats→403). Значит throttle снижает rate, но НЕ устраняет session-poisoning от extras; прокси (mobile-proxy, как в tradein) — следующий рычаг, off-by-default. - config: scrape_kn_browser_concurrency=2, scrape_kn_request_jitter_{min,max}_ms, scrape_kn_proxy_url=None (прокси опционален, дефолт — прямое подключение). - stealth.BrowserSession: per-instance concurrency / jitter / proxy_url; parse_proxy_url(http://user:pass@host:port → Playwright proxy-dict). Модульный _BROWSER_CONCURRENCY=8 не тронут — catalog/nspd без изменений. - domrf_kn.run_region_sweep + workers/tasks/scrape_kn: прокидывают новые settings ТОЛЬКО в KN-sweep BrowserSession. - tests: parse_proxy_url, threading concurrency/jitter/proxy, defaults unchanged.	2026-06-28 17:10:22 +05:00
bot-backend	f091554942	fix(scrapers): named BrowserSession concurrency const, fix Semaphore(3) doc drift (#1347) (#1913) ... named BrowserSession concurrency const + fix Semaphore(3) doc drift (#1347): keeps 8 (issue-sanctioned doc option). Refs #1347	2026-06-26 07:44:29 +00:00
bot-backend	14f3ef2019	fix(week-review): backend-аудит v2 — 82 фиксов (#1660) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-17 17:13:38 +00:00
lekss361	099b8e4b59	fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)	2026-05-24 10:53:28 +00:00
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather ... Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 ... PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00
lekss361	6d3fa8cbd2	add domrf extras parser (sale_graph + sales_agg + infra + photos) + ... fix Docker build for playwright - Regenerate uv.lock with playwright + tenacity (was missing → uv sync --frozen skipped them, hence playwright not in /app/.venv/bin/). - Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more) + `playwright install chromium` BEFORE apt-lists cleanup. - Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales, current aggregates, POI infrastructure, photo metadata, failure log. - Scraper: 4 new fetch helpers + upserts; photo binary download via Playwright APIRequest; failure-logging with full_url to kn_scrape_failures. - CLI: --no-extras / --download-photos / --no-flats / --probe URL. - Admin UI: extras checkboxes + failures table with copy-URL button. - PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg matches probe (Парк Победы 145/291 = 49%).	2026-04-27 18:35:46 +03:00
lekss361	79070bfe99	add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright ... - Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table. Server-side filter ignored — local filter by developer.companyGroup. - Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp). - Saved fingerprint state in data/playwright_state.json for server reuse. - Celery beat (configurable cron + jitter) + admin trigger endpoint + /admin/scrape UI page. - Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots, kn_scrape_runs journal. - Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.	2026-04-27 17:58:40 +03:00