gendesign/backend
lekss361 7863a99394 fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297)
Audit (2026-05-17): 496 153 total rows, 229 261 unique by
(obj_id, poi_category, poi_name, poi_address) = ~267k duplicates (~54%).
obj_id=65136 sample: 374 rows vs 192 on site = ×2 confirmed.

Root cause: old ON CONFLICT key included snapshot_date — each scrape run
inserted fresh rows for same POI. Accumulated ×N copies = N scrape runs.

Changes:
- data/sql/110_22j_dedup_infrastructure.sql: ONE-TIME cleanup keeping min(ctid)
  + DROP old constraint + ADD UNIQUE NULLS NOT DISTINCT (PG16) for nullable
  poi_category/poi_address.
- backend/app/services/scrapers/domrf_kn.py UPSERT_INFRA_SQL: ON CONFLICT
  (obj_id, poi_category, poi_name, poi_address) DO NOTHING — without snapshot_date.

Closes issue #297 sub-task 22j.
2026-05-17 18:12:14 +03:00
..
alembic ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
app fix(22j): DEDUP domrf_kn_infrastructure + ON CONFLICT (issue #297) 2026-05-17 18:12:14 +03:00
db/init ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
output feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
tests fix(sf-09): capacity-aware redistribute (round 2) — bounded + correct invariant 2026-05-17 15:30:50 +03:00
.dockerignore fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
.env.example feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00
.env.runtime.example sprint1: nspd scraper industrialization, per-bucket elasticity, cadastre cross-check, sentry releases 2026-04-30 21:51:19 +03:00
alembic.ini ops: alembic baseline, pre-commit, TIGER cleanup, pg_dump scripts 2026-04-26 13:08:51 +03:00
debug.log feat(geo): NSPD bulk-fetcher без Playwright + resume-friendly UI 2026-05-11 08:53:28 +03:00
Dockerfile fix(scraper): диагностируем «кнопка не работает» — Redis lock + task_received log 2026-04-28 23:17:42 +03:00
pyproject.toml feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00
uv.lock feat(backend): sentry-sdk init для FastAPI + Celery (#204 backend) (#207) 2026-05-16 15:09:16 +00:00