Audit (2026-05-17): 496 153 total rows, 229 261 unique by (obj_id, poi_category, poi_name, poi_address) = ~267k duplicates (~54%). obj_id=65136 sample: 374 rows vs 192 on site = ×2 confirmed. Root cause: old ON CONFLICT key included snapshot_date — each scrape run inserted fresh rows for same POI. Accumulated ×N copies = N scrape runs. Changes: - data/sql/110_22j_dedup_infrastructure.sql: ONE-TIME cleanup keeping min(ctid) + DROP old constraint + ADD UNIQUE NULLS NOT DISTINCT (PG16) for nullable poi_category/poi_address. - backend/app/services/scrapers/domrf_kn.py UPSERT_INFRA_SQL: ON CONFLICT (obj_id, poi_category, poi_name, poi_address) DO NOTHING — without snapshot_date. Closes issue #297 sub-task 22j. |
||
|---|---|---|
| .. | ||
| api | ||
| core | ||
| models | ||
| observability | ||
| schemas | ||
| scrapers | ||
| services | ||
| workers | ||
| __init__.py | ||
| main.py | ||