Commit graph

5 commits

Author SHA1 Message Date
099b8e4b59 fix(scrapers): DOM.РФ KN extras — bypass WAF + update renamed endpoints (#503)
Some checks failed
Deploy / changes (push) Successful in 4s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m22s
Deploy / build-worker (push) Successful in 2m41s
Deploy / deploy (push) Failing after 35s
2026-05-24 10:53:28 +00:00
lekss361
2aa33e0eb8 fix(stealth): bump Semaphore 6->8 for higher concurrency in gather
Per-object 7 endpoints через asyncio.gather упираются в semaphore limit.
6 -> 8 — ещё +33% concurrency для batch fetch'а.
2026-05-17 23:04:51 +03:00
lekss361
0f289c05b1 fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6
PR #322 used placeholder URLs:
- /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints:
  document/rpd / developer/report / project/documentation /
  documentation/other / document/permits). Replaced with 5 parallel
  fetches + merge into single upsert_documents call.
- /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk
  kn/object payload; endpoint /checks does not exist. Commented out
  _fetch_obj_checks_safe with TODO for separate investigation PR.
- BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency
  for intra-object gather.

GlitchTip: kn_scrape_failures run #19 showed 40 failures each for
both placeholder endpoints across first 30 objects of 1532.
2026-05-17 22:59:44 +03:00
lekss361
6d3fa8cbd2 add domrf extras parser (sale_graph + sales_agg + infra + photos) +
fix Docker build for playwright

- Regenerate uv.lock with playwright + tenacity (was missing → uv sync
  --frozen skipped them, hence playwright not in /app/.venv/bin/).
- Dockerfile: explicit Chromium runtime libs in apt (libnss3 + 14 more)
  + `playwright install chromium` BEFORE apt-lists cleanup.
- Schema 51_schema_kn_extras.sql: 5 new tables for time-series sales,
  current aggregates, POI infrastructure, photo metadata, failure log.
- Scraper: 4 new fetch helpers + upserts; photo binary download via
  Playwright APIRequest; failure-logging with full_url to kn_scrape_failures.
- CLI: --no-extras / --download-photos / --no-flats / --probe URL.
- Admin UI: extras checkboxes + failures table with copy-URL button.
- PRINZIP smoke: 28 objs, 5409 POI, 5150 photos metadata, sales_agg
  matches probe (Парк Победы 145/291 = 49%).
2026-04-27 18:35:46 +03:00
lekss361
79070bfe99 add interactive parser of наш.дом.рф ЖК+flats with stealth Playwright
- Discovered real endpoints via chrome-devtools: /сервисы/api/kn/object
  (offset/limit/place/objStatus) + /portal-kn/api/sales/portal/table.
  Server-side filter ignored — local filter by developer.companyGroup.
- Pure Playwright in-context fetch (httpx blocked by ServicePipe TLS-fp).
- Saved fingerprint state in data/playwright_state.json for server reuse.
- Celery beat (configurable cron + jitter) + admin trigger endpoint
  + /admin/scrape UI page.
- Schema migration: UNIQUE(id, snapshot_date) for versioned snapshots,
  kn_scrape_runs journal.
- Smoke-tested on PRINZIP: 28 ЖК / 826 квартир as expected.
2026-04-27 17:58:40 +03:00