fix(site-finder): Overpass retry-backoff + mirror fallback for utility loader (#1746) #1931

Merged

bot-backend merged 1 commit from fix/1746-overpass-retry-backoff into main

2026-06-26 21:13:07 +00:00

bot-backend commented

2026-06-26 20:38:36 +00:00

Collaborator

Публичный overpass-api.de зарейтлимитил тяжёлые query-типы (429/504) → грузилось только ~1099/3000 utility-элементов (power+water прошли, остальное throttled).

Fix: per-query retry-with-backoff на 429/5xx (honors Retry-After, 3 попытки/endpoint, 10/30/60с + jitter) + mirror-fallback (OVERPASS_ENDPOINTS: overpass-api.de → kumi.systems → private.coffee) + inter-query delay 1с→3с. Query, упавший на всех retry+mirror, логируется и скипается (partial load, не abort); per-element SAVEPOINT/UPSERT без изменений.

Tests: retry-success, mirror-fallback, 504, all-fail-skip, one-failing-query-isolation (всё мокнуто). Full suite 3565 passed, 0 failed. ruff clean, без schema/codegen.

После мержа+деплоя — re-populate (sync_utility_infrastructure_to_db()) доберёт полное покрытие (towers/gas/heat/sewage), counts → ~3000.

Refs #1746

Публичный `overpass-api.de` зарейтлимитил тяжёлые query-типы (429/504) → грузилось только ~1099/3000 utility-элементов (power+water прошли, остальное throttled). **Fix:** per-query retry-with-backoff на 429/5xx (honors `Retry-After`, 3 попытки/endpoint, 10/30/60с + jitter) + mirror-fallback (`OVERPASS_ENDPOINTS`: overpass-api.de → kumi.systems → private.coffee) + inter-query delay 1с→3с. Query, упавший на всех retry+mirror, логируется и скипается (partial load, не abort); per-element SAVEPOINT/UPSERT без изменений. Tests: retry-success, mirror-fallback, 504, all-fail-skip, one-failing-query-isolation (всё мокнуто). **Full suite 3565 passed, 0 failed.** ruff clean, без schema/codegen. После мержа+деплоя — re-populate (`sync_utility_infrastructure_to_db()`) доберёт полное покрытие (towers/gas/heat/sewage), counts → ~3000. Refs #1746

bot-backend added 1 commit 2026-06-26 20:38:37 +00:00

fix(site-finder): Overpass retry-backoff + mirror fallback for utility loader (#1746 )

CI / changes (pull_request) Successful in 6s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Successful in 1m50s

Details

CI / backend-tests (pull_request) Successful in 11m7s

Details

c70154ed61

Public overpass-api.de throttled later heavy query-types (429/504) →
only ~1099/3000 utility elements loaded. Harden the loader:

- per-query retry-with-backoff on 429/5xx (esp. 504): up to 3 attempts
  per endpoint, exponential backoff (10s/30s/60s) + jitter, honoring the
  Retry-After header when present.
- mirror fallback: OVERPASS_ENDPOINTS list (overpass-api.de →
  kumi.systems → private.coffee); exhausting one endpoint rotates to the
  next mirror.
- inter-query delay bumped 1s → 3s (gentler on the public endpoint).
- a query that fails all retries+mirrors is logged and skipped
  (partial-load, not abort) — existing per-element SAVEPOINT/UPSERT kept.

Tests: 429-then-200 retry, persistent-429 primary → mirror fallback,
504 → mirror, all-fail skip-without-abort, one-failing-query-does-not-
abort-others, plus Retry-After/backoff helper units. All mocked, no
real Overpass.