feat: fuzzy matcher v2 — Objective mapping coverage 8.5%→20% #286

Merged

lekss361 merged 1 commit from feat/fuzzy-matcher-v2-coverage into main

2026-05-17 13:39:29 +00:00

lekss361 commented

2026-05-17 13:33:27 +00:00

Owner

Summary

Task #44 Part A — re-trigger Objective↔DOM.РФ mapping pipeline + добавить v2 pruned-threshold mode.

Discovery: existing objective_backfill.py (pg_trgm in-DB similarity, не rapidfuzz) никогда не запускался на проде — zero fuzzy_trgm rows. DB audit 2026-05-17: 1068 EKB obj unmapped.

Audit (DB, EKB region_cd=66)

Score band	Count	Action
≥ 0.90	109	v1 auto-accept
0.85–0.90	24	v1 auto-accept
0.80–0.85	47	v2 auto-accept (`is_reviewed=false`)
0.70–0.80	88	below v2 threshold, skip

Expected after both runs: 129 (existing) + 133 (v1) + 47 (v2) ≈ 309 / 1516 ≈ 20% coverage (вместо текущих 8.5%).

Changes

backend/app/services/etl/objective_backfill.py:
- AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant
- min_threshold param on find_match_candidates() (default 0.6)
- match_method param on auto_apply_matches() (default 'fuzzy_trgm')
backend/app/api/v1/admin_etl.py:
- ?v2=true query param на POST /api/v1/admin/etl/objective-backfill — threshold=0.80, method=fuzzy_v2, search range 0.80–1.0
- Response now includes threshold_used + match_method_used для audit trail
Vault: fixes/Fix_Mapping_Coverage_Fuzzy_v2_May17.md

How to trigger after merge (sequential)

# Step 1: v1 run (~133 new rows, score ≥ 0.85, match_method='fuzzy_trgm')
curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?refresh_mv=false" \
  -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN"

# Step 2: v2 run (~47 more rows, score ≥ 0.80, match_method='fuzzy_v2')
curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?v2=true&refresh_mv=true" \
  -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN"

Verify after run

SELECT match_method, COUNT(*), ROUND(AVG(match_score)::numeric, 2)
FROM objective_complex_mapping
GROUP BY match_method;
-- Expected: fuzzy=127, fuzzy_trgm~133, fuzzy_v2~47, manual=2

Risk

fuzzy_v2 rows (0.80–0.85) — higher false-positive risk, все с is_reviewed=false для review queue
fuzzy_trgm rows (0.85+) — безопасны для auto-use в velocity calculations
Idempotent: ON CONFLICT (objective_complex_name, objective_group) DO NOTHING
NB: после merge — НЕ забыть trigger 2 curl выше (data write — не делал в PR)

Related: task #44 Part A · prerequisite for Part D (SF#17 rosreestr fallback)

## Summary Task #44 Part A — re-trigger Objective↔DOM.РФ mapping pipeline + добавить v2 pruned-threshold mode. **Discovery**: existing `objective_backfill.py` (pg_trgm in-DB similarity, не rapidfuzz) **никогда не запускался на проде** — zero `fuzzy_trgm` rows. DB audit 2026-05-17: 1068 EKB obj unmapped. ## Audit (DB, EKB region_cd=66) | Score band | Count | Action | |---|---|---| | ≥ 0.90 | 109 | v1 auto-accept | | 0.85–0.90 | 24 | v1 auto-accept | | 0.80–0.85 | 47 | **v2** auto-accept (`is_reviewed=false`) | | 0.70–0.80 | 88 | below v2 threshold, skip | Expected after both runs: **129 (existing) + 133 (v1) + 47 (v2) ≈ 309 / 1516 ≈ 20% coverage** (вместо текущих 8.5%). ## Changes - `backend/app/services/etl/objective_backfill.py`: - `AUTO_ACCEPT_THRESHOLD_V2 = 0.80` constant - `min_threshold` param on `find_match_candidates()` (default 0.6) - `match_method` param on `auto_apply_matches()` (default `'fuzzy_trgm'`) - `backend/app/api/v1/admin_etl.py`: - `?v2=true` query param на `POST /api/v1/admin/etl/objective-backfill` — threshold=0.80, method=`fuzzy_v2`, search range 0.80–1.0 - Response now includes `threshold_used` + `match_method_used` для audit trail - Vault: `fixes/Fix_Mapping_Coverage_Fuzzy_v2_May17.md` ## How to trigger after merge (sequential) ```bash # Step 1: v1 run (~133 new rows, score ≥ 0.85, match_method='fuzzy_trgm') curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?refresh_mv=false" \ -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN" # Step 2: v2 run (~47 more rows, score ≥ 0.80, match_method='fuzzy_v2') curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?v2=true&refresh_mv=true" \ -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN" ``` ## Verify after run ```sql SELECT match_method, COUNT(*), ROUND(AVG(match_score)::numeric, 2) FROM objective_complex_mapping GROUP BY match_method; -- Expected: fuzzy=127, fuzzy_trgm~133, fuzzy_v2~47, manual=2 ``` ## Risk - `fuzzy_v2` rows (0.80–0.85) — higher false-positive risk, все с `is_reviewed=false` для review queue - `fuzzy_trgm` rows (0.85+) — безопасны для auto-use в velocity calculations - Idempotent: `ON CONFLICT (objective_complex_name, objective_group) DO NOTHING` - **NB**: после merge — НЕ забыть trigger 2 curl выше (data write — не делал в PR) Related: task #44 Part A · prerequisite for Part D (SF#17 rosreestr fallback)

lekss361 added 1 commit 2026-05-17 13:33:27 +00:00

feat(etl): fuzzy matcher v2 — pruned threshold 0.85→0.80 for objective mapping coverage 2694e3180b

- Add AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant to objective_backfill.py
- Add min_threshold param to find_match_candidates() (default REVIEW_THRESHOLD=0.6)
- Add match_method param to auto_apply_matches() (default 'fuzzy_trgm')
- Add ?v2=true query param to POST /api/v1/admin/etl/objective-backfill
  - v2 mode: threshold=0.80, method='fuzzy_v2', search from 0.80 (not 0.60)
- Return type widened to dict[str,object] to include threshold_used + match_method_used

DB audit: 1068 unmapped EKB objs; v1 adds ~133 rows, v2 adds ~47 more.
Expected coverage: 8.5% → ~20% after sequential v1+v2 runs on prod.
Run: POST /objective-backfill (v1) then POST /objective-backfill?v2=true (v2).
Part of task #44 Part A, epic #271.

lekss361 merged commit fcfc492ab5 into main

2026-05-17 13:39:29 +00:00

lekss361 referenced this pull request from a commit

2026-05-17 13:39:29 +00:00

Merge pull request 'feat: fuzzy matcher v2 — Objective mapping coverage 8.5%→20%' (#286) from feat/fuzzy-matcher-v2-coverage into main

lekss361 commented

2026-05-17 13:39:52 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Проверено:

psycopg v3 + CAST(:x AS type) clean (нет :: traps)
SAVEPOINT pattern сохранён в loop (with db.begin_nested() per-row)
Backward compat: оба новых param с defaults, existing callers не сломаются
Return type dict[str, object] корректен (новые float/str поля)
match_method column — TEXT (не ENUM) → значение 'fuzzy_v2' ОК без миграции
Anti-regression vs PR #273/#277: ON CONFLICT skip silently, idempotent
Vault: fixes/Fix_Mapping_Coverage_Fuzzy_v2_May17.md exists

Required manual steps (post-deploy):

# Step 1: v1 (~133 rows, threshold=0.85, match_method='fuzzy_trgm')
curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?refresh_mv=false" \
  -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN"

# Step 2: v2 (~47 rows, threshold=0.80, match_method='fuzzy_v2', is_reviewed=false)
curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?v2=true&refresh_mv=true" \
  -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN"

Verify через SQL tunnel (localhost:15432):

SELECT match_method, COUNT(*), ROUND(AVG(match_score)::numeric, 2) AS avg_score
FROM objective_complex_mapping
GROUP BY match_method
ORDER BY 2 DESC;
-- Expected: fuzzy=127, fuzzy_trgm~133, fuzzy_v2~47, manual=2

Followups (отдельными задачами):

Review UI / CLI для fuzzy_v2 rows с is_reviewed=false — 47 строк без процесса review накопятся как technical debt. Опционально: фильтр в velocity query на is_reviewed=true OR match_score >= 0.85 (consistent с PR #288 data_confidence pattern).
Проверить mv_layout_velocity definition — учитывает ли он is_reviewed? Если no — v2 rows влияют на velocity сразу после REFRESH.
Vault docs gap (minor): code/modules/Objective_Backfill_ETL.md отсутствует — добавить при следующем касании модуля.

Merged via deep-code-reviewer — verdict APPROVE. **Проверено:** - psycopg v3 + `CAST(:x AS type)` clean (нет `::` traps) - SAVEPOINT pattern сохранён в loop (`with db.begin_nested()` per-row) - Backward compat: оба новых param с defaults, existing callers не сломаются - Return type `dict[str, object]` корректен (новые float/str поля) - `match_method` column — TEXT (не ENUM) → значение `'fuzzy_v2'` ОК без миграции - Anti-regression vs PR #273/#277: ON CONFLICT skip silently, idempotent - Vault: `fixes/Fix_Mapping_Coverage_Fuzzy_v2_May17.md` exists **Required manual steps (post-deploy):** ```bash # Step 1: v1 (~133 rows, threshold=0.85, match_method='fuzzy_trgm') curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?refresh_mv=false" \ -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN" # Step 2: v2 (~47 rows, threshold=0.80, match_method='fuzzy_v2', is_reviewed=false) curl -X POST "https://gendsgn.ru/api/v1/admin/etl/objective-backfill?v2=true&refresh_mv=true" \ -H "X-Admin-Token: $SCRAPE_ADMIN_TOKEN" ``` **Verify через SQL tunnel (`localhost:15432`):** ```sql SELECT match_method, COUNT(*), ROUND(AVG(match_score)::numeric, 2) AS avg_score FROM objective_complex_mapping GROUP BY match_method ORDER BY 2 DESC; -- Expected: fuzzy=127, fuzzy_trgm~133, fuzzy_v2~47, manual=2 ``` **Followups (отдельными задачами):** 1. **Review UI / CLI** для `fuzzy_v2` rows с `is_reviewed=false` — 47 строк без процесса review накопятся как technical debt. Опционально: фильтр в velocity query на `is_reviewed=true OR match_score >= 0.85` (consistent с PR #288 `data_confidence` pattern). 2. **Проверить `mv_layout_velocity` definition** — учитывает ли он `is_reviewed`? Если no — v2 rows влияют на velocity сразу после REFRESH. 3. **Vault docs gap (minor):** `code/modules/Objective_Backfill_ETL.md` отсутствует — добавить при следующем касании модуля.

lekss361 referenced this pull request

2026-05-17 13:52:17 +00:00

fix(sf-17): rosreestr velocity fallback via cad_quarter (~100% EKB coverage) #289