gendesign/tradein-mvp/backend/scripts/README.md

# tradein-mvp/backend/scripts/

Ops scripts that touch the production database directly. Run via `python -m
scripts.<name>` from the `backend/` working directory after `uv sync`.

All scripts are idempotent / resumable where they write — re-running the same
`--batch` label skips already-processed rows (UNIQUE constraints in target
tables). Failures inside a per-row loop never roll back the outer transaction;
each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/backend.md`.

---

## Production usage (canonical)

Scripts ship inside the `tradein-backend` image (PR F — `COPY scripts ./scripts`
в `backend/Dockerfile`). На VPS они уже в `/app/scripts/` — никаких manual
`docker cp` не нужно.

`YANDEX_GEOCODER_API_KEY` подтягивается из `/opt/gendesign/tradein-mvp/backend/
.env.runtime` через `env_file:` в `docker-compose.prod.yml` — никакого `-e` в
`docker exec` не нужно.

```bash
# Backfill (forward geocode 4170 houses без coords)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'

# Audit-only (reverse geocode проверка для уже geocoded houses)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'

# Canary first
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'
```

После изменения `backend/.env.runtime` нужен `--force-recreate` контейнера
(см. `.claude/rules/deploy.md`):

```bash
ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'
```

---

## Address audit + backfill (issue #582)

End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.

> Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell.
> На prod используй canonical `docker exec` команды из секции выше — там
> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`.

### `audit_address_mismatch.py` — Phase 1 baseline (PR #583)

Stratified-sample audit (200 EKB houses) comparing `houses.address` vs
Yandex Geocoder reverse lookup. Writes one row per house into
`address_mismatch_audit` with the snapped point + canonical address + distance.

```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.audit_address_mismatch \
    --batch 2026-05-25_run1 \
    --limit-per-district 25
```

Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware,
4-7s sleep between calls). API tier free is 25k req/day → 200-row sample
takes ~10s with no quota concern.

Report:

```bash
psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
    -f scripts/address_audit_report.sql
```

### `backfill_house_coords.py` — Phase 2-3 (PR for #582)

Two modes (`--audit-only` flag switches between them):

**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows
WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns
`precision='exact'` or `'number'` (skips street-only / locality matches).
Each processed row gets an `address_mismatch_audit` entry with status
`backfill` / `imprecise` / `no_match` / `error`.

```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_backfill
```

Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request):
20-25 min. Expected output split (rough baseline from Phase 1 numbers):

| Status      | Approx rows | What it means                                       |
|-------------|-------------|-----------------------------------------------------|
| `backfill`  | ~3.3k–3.7k  | UPDATE landed, lat/lon now populated                |
| `imprecise` | ~300–500    | Match returned but precision too low — needs review |
| `no_match`  | ~100–300    | Yandex couldn't resolve; address probably mangled   |
| `error`     | <50         | HTTP errors / timeouts — re-run picks them up       |

**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write
audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`.
Does NOT modify the `houses` table.

```bash
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_audit --audit-only
```

Combined budget for both phases (~8.6k requests) is well under the 25k/day
Geocoder free tier.

### Common ops

Canary first — run with `--limit 100` and inspect the audit table before
letting the full job loose:

```bash
uv run python -m scripts.backfill_house_coords \
    --batch canary_$(date +%F) --limit 100
psql "$DATABASE_URL" -c "
  SELECT audit_status, COUNT(*)
    FROM address_mismatch_audit
   WHERE audit_batch = 'canary_$(date +%F)'
   GROUP BY audit_status;
"
```

Resume after crash / quota hit — same `--batch` label, the UNIQUE
`(house_id, audit_batch)` index skips finished rows:

```bash
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# ... interruption ...
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"
```

### Helpers (not entry points)

- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`,
  `reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API
  paths share `_parse_api_payload` because Yandex's forward/reverse
  envelopes have the same shape.
- `audit_address_sample.sql` — random sample for the Phase 1 audit (used
  by `audit_address_mismatch.py`).
- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95
  distance, top-20 outliers, per-district breakdown).