gendesign/tradein-mvp/backend/scripts/README.md

tradein-mvp/backend/scripts/

Ops scripts that touch the production database directly. Run via python -m scripts.<name> from the backend/ working directory after uv sync.

All scripts are idempotent / resumable where they write — re-running the same --batch label skips already-processed rows (UNIQUE constraints in target tables). Failures inside a per-row loop never roll back the outer transaction; each row is wrapped in a SAVEPOINT (db.begin_nested()) per .claude/rules/backend.md.

---

Production usage (canonical)

Scripts ship inside the tradein-backend image (PR F — COPY scripts ./scripts в backend/Dockerfile). На VPS они уже в /app/scripts/ — никаких manual docker cp не нужно.

YANDEX_GEOCODER_API_KEY подтягивается из /opt/gendesign/tradein-mvp/backend/ .env.runtime через env_file: в docker-compose.prod.yml — никакого -e в docker exec не нужно.

# Backfill (forward geocode 4170 houses без coords)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'

# Audit-only (reverse geocode проверка для уже geocoded houses)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'

# Canary first
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'

После изменения backend/.env.runtime нужен --force-recreate контейнера (см. .claude/rules/deploy.md):

ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'

---

Address audit + backfill (issue #582)

End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.

Локальные примеры ниже — для dev-машины с uv run и переменными в shell. На prod используй canonical docker exec команды из секции выше — там YANDEX_GEOCODER_API_KEY уже подгружен из backend/.env.runtime.

audit_address_mismatch.py — Phase 1 baseline (PR #583)

Stratified-sample audit (200 EKB houses) comparing houses.address vs Yandex Geocoder reverse lookup. Writes one row per house into address_mismatch_audit with the snapped point + canonical address + distance.

DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.audit_address_mismatch \
    --batch 2026-05-25_run1 \
    --limit-per-district 25

Mode auto picks API if the key is set, otherwise Playwright (CAPTCHA-aware, 4-7s sleep between calls). API tier free is 25k req/day → 200-row sample takes ~10s with no quota concern.

Report:

psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
    -f scripts/address_audit_report.sql

backfill_house_coords.py — Phase 2-3 (PR for #582)

Two modes (--audit-only flag switches between them):

Backfill (default) — forward-geocode houses.address for the ~4141 rows WHERE lat IS NULL OR lon IS NULL. Only writes back if Yandex returns precision='exact' or 'number' (skips street-only / locality matches). Each processed row gets an address_mismatch_audit entry with status backfill / imprecise / no_match / error.

DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_backfill

Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request): 20-25 min. Expected output split (rough baseline from Phase 1 numbers):

Status	Approx rows	What it means
backfill	~3.3k–3.7k	UPDATE landed, lat/lon now populated
imprecise	~300–500	Match returned but precision too low — needs review
no_match	~100–300	Yandex couldn't resolve; address probably mangled
error	<50	HTTP errors / timeouts — re-run picks them up

Audit-only — reverse-geocode the ~4452 houses WITH coords, write audit rows with status ok (≤50m) / mismatch (>50m) / no_match / error. Does NOT modify the houses table.

uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_audit --audit-only

Combined budget for both phases (~8.6k requests) is well under the 25k/day Geocoder free tier.

Common ops

Canary first — run with --limit 100 and inspect the audit table before letting the full job loose:

uv run python -m scripts.backfill_house_coords \
    --batch canary_$(date +%F) --limit 100
psql "$DATABASE_URL" -c "
  SELECT audit_status, COUNT(*)
    FROM address_mismatch_audit
   WHERE audit_batch = 'canary_$(date +%F)'
   GROUP BY audit_status;
"

Resume after crash / quota hit — same --batch label, the UNIQUE (house_id, audit_batch) index skips finished rows:

uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# ... interruption ...
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"

Helpers (not entry points)

• _yandex_reverse.py — forward_via_api(), reverse_via_api(), reverse_via_playwright(), YandexReverseResult dataclass. Both API paths share _parse_api_payload because Yandex's forward/reverse envelopes have the same shape.
• audit_address_sample.sql — random sample for the Phase 1 audit (used by audit_address_mismatch.py).
• address_audit_report.sql — psql-driven post-run summary (p50/p75/p95 distance, top-20 outliers, per-district breakdown).