lekss361/gendesign

Fork 0

fix(scrape-kn): real documents endpoints (5x) + skip obj_checks + sem 3→8 #325

Merged

lekss361 merged 2 commits from fix/domrf-real-doc-endpoints-skip-checks-sem6 into main

2026-05-17 20:08:51 +00:00

lekss361 commented

2026-05-17 20:05:24 +00:00

Owner

Bug — run #19 (release `c80d79e`) showed 2 endpoint placeholders fail 404

kn_scrape_failures group_by endpoint:

endpoint	failures	URL	status
`documents`	40	`/сервисы/api/object/{id}/documents`	404
`obj_checks`	40	`/сервисы/api/object/{id}/checks`	404

Per Playwright network inspection (наш.дом.рф/.../объект/65136):

/documents (plural) — не существует, документы это 5 разных endpoints
/checks — не существует, 6 chekboxов "Проверено" inline в основном payload

Fix — 3 hunks

1. Documents → 5 параллельных endpoints

PATH_DOC_RPD                    = "/сервисы/api/object/{obj_id}/document/rpd"
PATH_DOC_DEVELOPER_REPORT       = "/сервисы/api/object/{obj_id}/developer/report"
PATH_DOC_PROJECT_DOCUMENTATION  = "/сервисы/api/object/{obj_id}/project/documentation"
PATH_DOC_DOCUMENTATION_OTHER    = "/сервисы/api/object/{obj_id}/documentation/other"
PATH_DOC_PERMITS                = "/сервисы/api/object/{obj_id}/document/permits"

Каждый — _fetch_doc_*_safe wrapper. В parallel asyncio.gather — 5 corutines вместо 1. После gather — merge всех результатов через extract_documents + единый upsert_documents per obj.

2. obj_checks — skip (commented + TODO)

/checks endpoint не существует. Закомментил _fetch_obj_checks_safe вызов + imports extract_obj_checks / upsert_obj_checks. Файл services/scrapers/obj_checks.py оставлен для future re-wire когда найдём реальный источник (вероятно inline в main object payload).

3. BrowserSession `Semaphore(3) → Semaphore(8)`

Концurrency limit поднят с 3 до 8 для intra-object asyncio.gather (12 endpoints/obj после fix). Run #19 показал что 3 одновременных request — слишком мало, gather'у нечего распараллеливать.

Ожидаемый speedup: ~2.5× относительно run #19 (4.4 sec/obj → ~1.8 sec/obj) → ~50 мин на 1532 obj vs 3.5 ч в run #14.

Acceptance после merge + deploy

kn_scrape_failures для нового run: 0 documents 404, 0 obj_checks 404 (закомментирован)
domrf_kn_documents count growing (cumulative across 5 doc types)
Speed: ~1.5-2 sec/obj в Progress log
Sweep duration ~50 мин для 1532 obj

Risk

🟡 Semaphore(8): 8 параллельных fetch на dom.рф из одной browser session. Rate-limit dom.рф не задокументирован. Если 429 — снизим обратно до 6.
🟢 5 doc endpoints: верифицированы через Playwright network inspection.
🟢 obj_checks skip: пустая таблица остаётся пустой, но больше нет error spam.

Tech debt

obj_checks endpoint URL — отдельный PR после investigation основного payload объекта. 6 chekboxов вероятно поля типа has_problems, documents_complete, bankruptcy_flag в bulk kn/object response.

## Bug — run #19 (release `c80d79e`) showed 2 endpoint placeholders fail 404 `kn_scrape_failures` group_by endpoint: | endpoint | failures | URL | status | |---|---|---|---| | `documents` | **40** | `/сервисы/api/object/{id}/documents` | 404 | | `obj_checks` | **40** | `/сервисы/api/object/{id}/checks` | 404 | Per Playwright network inspection (`наш.дом.рф/.../объект/65136`): - `/documents` (plural) — **не существует**, документы это **5 разных endpoints** - `/checks` — **не существует**, 6 chekboxов "Проверено" inline в основном payload ## Fix — 3 hunks ### 1. Documents → 5 параллельных endpoints ```python PATH_DOC_RPD = "/сервисы/api/object/{obj_id}/document/rpd" PATH_DOC_DEVELOPER_REPORT = "/сервисы/api/object/{obj_id}/developer/report" PATH_DOC_PROJECT_DOCUMENTATION = "/сервисы/api/object/{obj_id}/project/documentation" PATH_DOC_DOCUMENTATION_OTHER = "/сервисы/api/object/{obj_id}/documentation/other" PATH_DOC_PERMITS = "/сервисы/api/object/{obj_id}/document/permits" ``` Каждый — `_fetch_doc_*_safe` wrapper. В parallel `asyncio.gather` — 5 corutines вместо 1. После gather — merge всех результатов через `extract_documents` + единый `upsert_documents` per obj. ### 2. obj_checks — skip (commented + TODO) `/checks` endpoint не существует. Закомментил `_fetch_obj_checks_safe` вызов + imports `extract_obj_checks` / `upsert_obj_checks`. Файл `services/scrapers/obj_checks.py` оставлен для future re-wire когда найдём реальный источник (вероятно inline в main object payload). ### 3. BrowserSession `Semaphore(3) → Semaphore(8)` Концurrency limit поднят с 3 до **8** для intra-object `asyncio.gather` (12 endpoints/obj после fix). Run #19 показал что 3 одновременных request — слишком мало, gather'у нечего распараллеливать. Ожидаемый speedup: ~2.5× относительно run #19 (4.4 sec/obj → ~1.8 sec/obj) → **~50 мин на 1532 obj** vs 3.5 ч в run #14. ## Acceptance после merge + deploy - [ ] `kn_scrape_failures` для нового run: 0 documents 404, 0 obj_checks 404 (закомментирован) - [ ] `domrf_kn_documents` count growing (cumulative across 5 doc types) - [ ] Speed: ~1.5-2 sec/obj в Progress log - [ ] Sweep duration ~50 мин для 1532 obj ## Risk - 🟡 **`Semaphore(8)`**: 8 параллельных fetch на dom.рф из одной browser session. Rate-limit dom.рф не задокументирован. Если 429 — снизим обратно до 6. - 🟢 5 doc endpoints: верифицированы через Playwright network inspection. - 🟢 obj_checks skip: пустая таблица остаётся пустой, но больше нет error spam. ## Tech debt obj_checks endpoint URL — отдельный PR после investigation основного payload объекта. 6 chekboxов вероятно поля типа `has_problems`, `documents_complete`, `bankruptcy_flag` в bulk `kn/object` response.

lekss361 added 2 commits 2026-05-17 20:05:25 +00:00

fix(scrape-kn): real documents endpoints (5x) + skip obj_checks 404 + sem 3->6 0f289c05b1

PR #322 used placeholder URLs:
- /api/object/{id}/documents -> 404 (Playwright showed: 5 real endpoints:
  document/rpd / developer/report / project/documentation /
  documentation/other / document/permits). Replaced with 5 parallel
  fetches + merge into single upsert_documents call.
- /api/object/{id}/checks -> 404. 6 checkboxes likely inline in bulk
  kn/object payload; endpoint /checks does not exist. Commented out
  _fetch_obj_checks_safe with TODO for separate investigation PR.
- BrowserSession Semaphore(3) -> Semaphore(6) — double concurrency
  for intra-object gather.

GlitchTip: kn_scrape_failures run #19 showed 40 failures each for
both placeholder endpoints across first 30 objects of 1532.

fix(stealth): bump Semaphore 6->8 for higher concurrency in gather 2aa33e0eb8

Per-object 7 endpoints через asyncio.gather упираются в semaphore limit.
6 -> 8 — ещё +33% concurrency для batch fetch'а.

lekss361 commented

2026-05-17 20:07:58 +00:00

Author

Owner

Deep Code Review — verdict APPROVE

Status: APPROVE
Files: 2 (P1: backend/app/services/scrapers/domrf_kn.py +128/-24, backend/app/services/scrapers/stealth.py +1/-1)
Head SHA: 2aa33e0e
Mergeable: true

Hunk 1 — Documents → 5 endpoints

5 new path constants follow established Cyrillic-path pattern.
_fetch_doc_endpoint helper DRYs the 5 wrappers; each correctly returns (kind_tag, full_url, items_or_exception) and 404/JSON failure routes through _classify_and_log via the existing exception path.
Edge: _fetch_doc_endpoint handles 3 payload shapes (list, dict.data=list, dict.data missing or wrong type → []) — strictly safer than old fetch_documents which .get("data") or [] would crash if payload were None (httpx returns dict, but defensive form is correct).
Merge logic: all_docs.extend(extract_documents(doc_items or [])) is correct because extract_documents is pure-per-payload (no obj_id baked in — that's passed to upsert_documents).
Schema domrf_kn_documents (UNIQUE obj_id, doc_type, doc_num, file_url) — verified at data/sql/112_22i_domrf_documents.sql:18. doc_type discriminator is payload-derived (docTypeName → canonical via _DOC_TYPE_MAP), NOT endpoint-derived. This means same doc_type can come from different endpoints — UNIQUE still holds because file_url is part of the key.
Single upsert_documents after the loop is correct and efficient (one transaction with begin_nested SAVEPOINT per row).

Hunk 2 — obj_checks skip

Import commented cleanly (line 31-33), no F401 risk.
Verified no leftover references to extract_obj_checks / upsert_obj_checks on the PR branch (only obj_checks.py module retained for future re-wire — good).
extras_counts["checks_rows"] stays initialized at 0 and is never incremented. Log lines still print checks=0 — cosmetic only, not worth touching in this PR.
TODO comment block at the gather site documents the rationale clearly.

Hunk 3 — Semaphore 3 → 8

Single site change at stealth.py:107, per-instance attribute (not module-level) — correct.
Risk analysis matches author's own assessment: 8 parallel through one Playwright BrowserContext shares the connection pool, low WAF risk. Fallback plan (drop to 6 on 429) is documented.

Convention checks

No psycopg2 imports.
No requests, all httpx through existing BrowserSession.
No print(), only logger.*.
No :x::type SQL casts (extract_documents uses pure Python).
No SQL injection vectors (parametrized via text(...) + :params).
No hardcoded credentials.

Speedup math sanity

12 endpoints/obj after fix (8 original + 5 doc - 1 obj_checks = 12).
Semaphore(8) → 2 batches (~8 + ~4) per obj → theoretical ~1.5×.
Additional speedup from removing 2× 404 exception paths (each had httpx backoff) → claim of 2.5× total is plausible.

Pre-flight

Branch fix/domrf-real-doc-endpoints-skip-checks-sem6, not main.
No --no-verify / --force / --amend indicators.
mergeable=true, no conflicts.

Risk / reversibility

Risk: Low. All changes scoped to a scraper, no DB schema change, no API contract change.
Reversibility: Trivial revert (no migrations).

Merging via squash.

## Deep Code Review — verdict APPROVE **Status**: APPROVE **Files**: 2 (P1: `backend/app/services/scrapers/domrf_kn.py` +128/-24, `backend/app/services/scrapers/stealth.py` +1/-1) **Head SHA**: `2aa33e0e` **Mergeable**: true ### Hunk 1 — Documents → 5 endpoints - 5 new path constants follow established Cyrillic-path pattern. - `_fetch_doc_endpoint` helper DRYs the 5 wrappers; each correctly returns `(kind_tag, full_url, items_or_exception)` and 404/JSON failure routes through `_classify_and_log` via the existing exception path. - Edge: `_fetch_doc_endpoint` handles 3 payload shapes (`list`, `dict.data=list`, `dict.data` missing or wrong type → `[]`) — strictly safer than old `fetch_documents` which `.get("data") or []` would crash if payload were `None` (httpx returns dict, but defensive form is correct). - Merge logic: `all_docs.extend(extract_documents(doc_items or []))` is correct because `extract_documents` is pure-per-payload (no obj_id baked in — that's passed to `upsert_documents`). - Schema `domrf_kn_documents (UNIQUE obj_id, doc_type, doc_num, file_url)` — verified at `data/sql/112_22i_domrf_documents.sql:18`. `doc_type` discriminator is **payload-derived** (`docTypeName` → canonical via `_DOC_TYPE_MAP`), NOT endpoint-derived. This means same doc_type can come from different endpoints — UNIQUE still holds because `file_url` is part of the key. - Single `upsert_documents` after the loop is correct and efficient (one transaction with `begin_nested` SAVEPOINT per row). ### Hunk 2 — obj_checks skip - Import commented cleanly (line 31-33), no F401 risk. - Verified no leftover references to `extract_obj_checks` / `upsert_obj_checks` on the PR branch (only `obj_checks.py` module retained for future re-wire — good). - `extras_counts["checks_rows"]` stays initialized at 0 and is never incremented. Log lines still print `checks=0` — cosmetic only, not worth touching in this PR. - TODO comment block at the gather site documents the rationale clearly. ### Hunk 3 — Semaphore 3 → 8 - Single site change at `stealth.py:107`, per-instance attribute (not module-level) — correct. - Risk analysis matches author's own assessment: 8 parallel through one Playwright `BrowserContext` shares the connection pool, low WAF risk. Fallback plan (drop to 6 on 429) is documented. ### Convention checks - No `psycopg2` imports. - No `requests`, all `httpx` through existing `BrowserSession`. - No `print()`, only `logger.*`. - No `:x::type` SQL casts (`extract_documents` uses pure Python). - No SQL injection vectors (parametrized via `text(...)` + `:params`). - No hardcoded credentials. ### Speedup math sanity - 12 endpoints/obj after fix (8 original + 5 doc - 1 obj_checks = 12). - `Semaphore(8)` → 2 batches (~8 + ~4) per obj → theoretical ~1.5×. - Additional speedup from removing 2× 404 exception paths (each had httpx backoff) → claim of 2.5× total is plausible. ### Pre-flight - Branch `fix/domrf-real-doc-endpoints-skip-checks-sem6`, not main. - No `--no-verify` / `--force` / `--amend` indicators. - `mergeable=true`, no conflicts. ### Risk / reversibility - **Risk**: Low. All changes scoped to a scraper, no DB schema change, no API contract change. - **Reversibility**: Trivial revert (no migrations). Merging via squash.