fix(cadastre): count snapshot+per-cat requests in cadastre_jobs.requests_count #182

Merged

lekss361 merged 1 commit from fix/cadastre-metrics-snapshot-count into main

2026-05-15 14:17:02 +00:00

lekss361 commented

2026-05-15 14:10:31 +00:00

(Migrated from github.com)

Summary

Fix metrics bug: cadastre_jobs.requests_count previously counted ONLY grid_walk_requests, so:

Snapshot-only quarters reported 0 NSPD requests
After PR #179 (skip grid-walk on broken geom), entire ekb_full job #8 reported 4950 reqs while real total was ~14000+ (2408 quarters × snapshot + per-cat probes from #180)

Changes

HarvestResult.snapshot_requests — increments at Phase 1 (1 search) + Phase 1.5 (per-cat probes from #180)
HarvestResult.total_requests property — snapshot_requests + grid_walk_requests
bulk_harvest_quarter_task writes total_requests to cadastre_jobs.requests_count
Both counters returned in result_dict for downstream observability

Why this matters

Without accurate request count, можно easily:

Underestimate NSPD load (sustained 3 req/s budget rough)
Miss anomalies (rate spikes, WAF blocks correlated with overall request volume)
Trust misleading metric: pilot v6 showed requests_count=0 despite 50 quarters worth of NSPD work

Test plan

pytest tests/services/test_cadastre_bulk.py → 29 passed (added 1 new + 1 assertion in existing)
After deploy → next pilot/ekb_full job's cadastre_jobs.requests_count should ≈ targets_total × (1 + avg per-cat probes + avg grid-walks/quarter).

Closes part of #168.

## Summary Fix metrics bug: `cadastre_jobs.requests_count` previously counted ONLY `grid_walk_requests`, so: - Snapshot-only quarters reported 0 NSPD requests - After PR #179 (skip grid-walk on broken geom), entire `ekb_full` job #8 reported 4950 reqs while real total was ~14000+ (2408 quarters × snapshot + per-cat probes from #180) ## Changes - `HarvestResult.snapshot_requests` — increments at Phase 1 (1 search) + Phase 1.5 (per-cat probes from #180) - `HarvestResult.total_requests` property — `snapshot_requests + grid_walk_requests` - `bulk_harvest_quarter_task` writes `total_requests` to `cadastre_jobs.requests_count` - Both counters returned in result_dict for downstream observability ## Why this matters Without accurate request count, можно easily: - Underestimate NSPD load (sustained 3 req/s budget rough) - Miss anomalies (rate spikes, WAF blocks correlated with overall request volume) - Trust misleading metric: pilot v6 showed `requests_count=0` despite 50 quarters worth of NSPD work ## Test plan - [x] `pytest tests/services/test_cadastre_bulk.py` → 29 passed (added 1 new + 1 assertion in existing) - [ ] After deploy → next pilot/ekb_full job's `cadastre_jobs.requests_count` should ≈ `targets_total × (1 + avg per-cat probes + avg grid-walks/quarter)`. Closes part of #168.

lekss361 commented

2026-05-15 14:16:02 +00:00

(Migrated from github.com)

Review

CI зелёный (backend 1m24s + frontend 1m34s) + cross-check (agent):

✅ Increment correctness

Phase 1 (bulk_harvest.py:136): result.snapshot_requests += 1 после await — на raise increment скипается + propagates правильно.
Phase 1.5 (bulk_harvest.py:189): increment inside try, на raise — except swallow + continue → не считается.
Empty-response 404 счётся (client возвращает empty snapshot, не raise) — корректно для rate-limit budget.
total_requests = snapshot + grid_walk — disjoint counters, no double-count.

✅ Worker switch

scrape_cadastre.py:225 — requests_count_inc=result_dict.get("total_requests", 0) с защитным дефолтом. Оба counters returned в result_dict для observability.

✅ Backward-compat

HarvestResult server-only dataclass; новое поле default=0. grid_walk_requests сохранено — grep подтвердил, нет внешних consumers (только bulk_harvest.py + scrape_cadastre.py + tests).

✅ Test coverage

test_harvest_result_total_requests_property — direct property арифметика (3+225=228 + default 0).
test_harvest_quarter_calls_upsert_features augmented с assertions snapshot_requests=1, grid_walk_requests=0, total_requests=1.
Минор: explicit assertion на Phase 1.5 increment отсутствует, но pattern identical к Phase 1 (одна строка после await), property тест покрывает арифметику. Не блокер.

✅ No conflict с PR #181

#181 трогает Phase 4 (~L237) + новая функция (~L946).
#182 трогает dataclass (L69-82) + Phase 1 (L133) + Phase 1.5 (L188).
Zero line overlap. Clean 3-way merge независимо от порядка.

✅ Drift отсутствует

HarvestResult server-side only, no TS callers.

approve merge

## Review CI зелёный (backend 1m24s + frontend 1m34s) + cross-check (agent): ### ✅ Increment correctness - **Phase 1** (`bulk_harvest.py:136`): `result.snapshot_requests += 1` после `await` — на raise increment скипается + propagates правильно. - **Phase 1.5** (`bulk_harvest.py:189`): increment inside `try`, на raise — `except` swallow + `continue` → не считается. - Empty-response 404 счётся (client возвращает empty snapshot, не raise) — корректно для rate-limit budget. - `total_requests = snapshot + grid_walk` — disjoint counters, no double-count. ### ✅ Worker switch `scrape_cadastre.py:225` — `requests_count_inc=result_dict.get("total_requests", 0)` с защитным дефолтом. Оба counters returned в result_dict для observability. ### ✅ Backward-compat `HarvestResult` server-only dataclass; новое поле default=0. `grid_walk_requests` сохранено — grep подтвердил, нет внешних consumers (только bulk_harvest.py + scrape_cadastre.py + tests). ### ✅ Test coverage - `test_harvest_result_total_requests_property` — direct property арифметика (3+225=228 + default 0). - `test_harvest_quarter_calls_upsert_features` augmented с assertions `snapshot_requests=1, grid_walk_requests=0, total_requests=1`. - **Минор**: explicit assertion на Phase 1.5 increment отсутствует, но pattern identical к Phase 1 (одна строка после await), property тест покрывает арифметику. Не блокер. ### ✅ No conflict с PR #181 - #181 трогает Phase 4 (~L237) + новая функция (~L946). - #182 трогает dataclass (L69-82) + Phase 1 (L133) + Phase 1.5 (L188). - Zero line overlap. Clean 3-way merge независимо от порядка. ### ✅ Drift отсутствует `HarvestResult` server-side only, no TS callers. approve merge