fix(scrape-kn): real documents endpoints (5x) + skip obj_checks + sem 3→8 #325

Merged

lekss361 merged 2 commits from fix/domrf-real-doc-endpoints-skip-checks-sem6 into main

2026-05-17 20:08:51 +00:00

Author	SHA1	Message	Date
lekss361	2aa33e0eb8	fix(stealth): bump Semaphore 6->8 for higher concurrency in gather Per-object 7 endpoints через asyncio.gather упираются в semaphore limit. 6 -> 8 — ещё +33% concurrency для batch fetch'а.	2026-05-17 23:04:51 +03:00
lekss361	0f289c05b1	fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 PR #322 used placeholder URLs: - /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints: document/rpd / developer/report / project/documentation / documentation/other / document/permits). Replaced with 5 parallel fetches + merge into single upsert_documents call. - /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk kn/object payload; endpoint /checks does not exist. Commented out _fetch_obj_checks_safe with TODO for separate investigation PR. - BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency for intra-object gather. GlitchTip: kn_scrape_failures run #19 showed 40 failures each for both placeholder endpoints across first 30 objects of 1532.	2026-05-17 22:59:44 +03:00

Author

SHA1

Message

Date

lekss361

2aa33e0eb8

fix(stealth): bump Semaphore 6->8 for higher concurrency in gather

Per-object 7 endpoints через asyncio.gather упираются в semaphore limit.
6 -> 8 — ещё +33% concurrency для batch fetch'а.

2026-05-17 23:04:51 +03:00

lekss361

0f289c05b1

fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6

PR #322 used placeholder URLs:
- /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints:
  document/rpd / developer/report / project/documentation /
  documentation/other / document/permits). Replaced with 5 parallel
  fetches + merge into single upsert_documents call.
- /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk
  kn/object payload; endpoint /checks does not exist. Commented out
  _fetch_obj_checks_safe with TODO for separate investigation PR.
- BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency
  for intra-object gather.

GlitchTip: kn_scrape_failures run #19 showed 40 failures each for
both placeholder endpoints across first 30 objects of 1532.

2026-05-17 22:59:44 +03:00