# tradein-mvp/backend/scripts/

Ops scripts that touch the production database directly. Run via `python -m
scripts.<name>` from the `backend/` working directory after `uv sync`.

All scripts are idempotent / resumable where they write — re-running the same
`--batch` label skips already-processed rows (UNIQUE constraints in target
tables). Failures inside a per-row loop never roll back the outer transaction;
each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/backend.md`.

---

## Production usage (canonical)

Scripts ship inside the `tradein-backend` image (PR F — `COPY scripts ./scripts`
в `backend/Dockerfile`). На VPS они уже в `/app/scripts/` — никаких manual
`docker cp` не нужно.

`YANDEX_GEOCODER_API_KEY` подтягивается из `/opt/gendesign/tradein-mvp/backend/
.env.runtime` через `env_file:` в `docker-compose.prod.yml` — никакого `-e` в
`docker exec` не нужно.

```bash
# Backfill (forward geocode 4170 houses без coords)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'

# Audit-only (reverse geocode проверка для уже geocoded houses)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'

# Canary first
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'
```

После изменения `backend/.env.runtime` нужен `--force-recreate` контейнера
(см. `.claude/rules/deploy.md`):

```bash
ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'
```

---

## Address audit + backfill (issue #582)

End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.

> Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell.
> На prod используй canonical `docker exec` команды из секции выше — там
> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`.

### `audit_address_mismatch.py` — Phase 1 baseline (PR #583)

Stratified-sample audit (200 EKB houses) comparing `houses.address` vs
Yandex Geocoder reverse lookup. Writes one row per house into
`address_mismatch_audit` with the snapped point + canonical address + distance.

```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.audit_address_mismatch \
    --batch 2026-05-25_run1 \
    --limit-per-district 25
```

Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware,
4-7s sleep between calls). API tier free is 25k req/day → 200-row sample
takes ~10s with no quota concern.

Report:

```bash
psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
    -f scripts/address_audit_report.sql
```

### `backfill_house_coords.py` — Phase 2-3 (PR for #582)

Two modes (`--audit-only` flag switches between them):

**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows
WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns
`precision='exact'` or `'number'` (skips street-only / locality matches).
Each processed row gets an `address_mismatch_audit` entry with status
`backfill` / `imprecise` / `no_match` / `error`.

```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_backfill
```

Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request):
20-25 min. Expected output split (rough baseline from Phase 1 numbers):

| Status      | Approx rows | What it means                                       |
|-------------|-------------|-----------------------------------------------------|
| `backfill`  | ~3.3k–3.7k  | UPDATE landed, lat/lon now populated                |
| `imprecise` | ~300–500    | Match returned but precision too low — needs review |
| `no_match`  | ~100–300    | Yandex couldn't resolve; address probably mangled   |
| `error`     | <50         | HTTP errors / timeouts — re-run picks them up       |

**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write
audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`.
Does NOT modify the `houses` table.

```bash
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_audit --audit-only
```

Combined budget for both phases (~8.6k requests) is well under the 25k/day
Geocoder free tier.

### Common ops

Canary first — run with `--limit 100` and inspect the audit table before
letting the full job loose:

```bash
uv run python -m scripts.backfill_house_coords \
    --batch canary_$(date +%F) --limit 100
psql "$DATABASE_URL" -c "
  SELECT audit_status, COUNT(*)
    FROM address_mismatch_audit
   WHERE audit_batch = 'canary_$(date +%F)'
   GROUP BY audit_status;
"
```

Resume after crash / quota hit — same `--batch` label, the UNIQUE
`(house_id, audit_batch)` index skips finished rows:

```bash
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# ... interruption ...
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"
```

### Helpers (not entry points)

- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`,
  `reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API
  paths share `_parse_api_payload` because Yandex's forward/reverse
  envelopes have the same shape.
- `audit_address_sample.sql` — random sample for the Phase 1 audit (used
  by `audit_address_mismatch.py`).
- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95
  distance, top-20 outliers, per-district breakdown).

---

## Matching backfill (PR J)

### `backfill_listing_sources.py` — retroactive matching for ~18k listings

PR I (commit 7e24ccb) hooked the matching service into `save_listings()` so
every **new** scrape now writes a `listing_sources` row + resolves a canonical
`houses` row. This script does the same work retroactively for all
**existing** listings — `listing_sources` only had rows from new scrapes
post-PR I.

What it does per row:

1. `match_or_create_house()` (Tier 0-3) — uses `listings.house_source` /
   `house_ext_id` when present (Avito Houses Catalog, Cian newbuilding),
   else falls back to address/lat/lon/cadastrals.
2. `upsert_listing_source()` with `method='backfill'`, `confidence=0.9`
   (vs real-time `source_link` 1.0 — distinguishes the two in audits).
3. `UPDATE listings.house_id_fk` when the row didn't already have one.

```bash
# Canary
DATABASE_URL=postgresql+psycopg://... \
    uv run python -m scripts.backfill_listing_sources \
        --limit 100 --dry-run

# Real run, one source at a time (staged rollout)
uv run python -m scripts.backfill_listing_sources --source avito

# Full run
uv run python -m scripts.backfill_listing_sources --batch-size 500
```

**Idempotent / resumable** — the source query is
`WHERE NOT EXISTS (SELECT 1 FROM listing_sources ls WHERE ls.ext_source =
listings.source AND ls.ext_id = COALESCE(listings.source_id,
listings.dedup_hash))`. Re-runs only pick up rows still missing from
`listing_sources`. `upsert_listing_source` adds a second layer of safety via
`ON CONFLICT (ext_source, ext_id) DO UPDATE`.

**No network calls** — pure in-DB matching (Yandex Geocoder is blocked on
prod, and `match_or_create_house` does not call it anyway).

**Per-row SAVEPOINT** (`db.begin_nested()`) per `.claude/rules/backend.md` —
one bad row never aborts the surrounding batch.

**Expected output (PR J initial run):**

| Source | Rows  | Expected matched | Notes                                          |
|--------|-------|-------------------|------------------------------------------------|
| avito  | 9302  | 9000+             | Many carry `house_source`/`house_ext_id`       |
| cian   | 5158  | 5000+             | Most carry `house_source`/`house_ext_id`       |
| yandex | 3704  | 3700+             | No `source_id` → uses `dedup_hash` as `ext_id` |
| n1     | 264   | 264               | All have address/coords                        |

**Expected duration**: rough estimate ~5-15 minutes on prod for ~18k rows
(advisory-lock + 1-3 DB roundtrips per listing for Tier 0-3, ~500 commit
checkpoints at default batch size). Run with `--limit 100` first to
calibrate, then let the full job loose.

Final summary in the log includes per-source coverage % so you can verify
the run landed:

```
backfill done (dry_run=False): processed=18428 matched=18428
  house_resolved=18200 house_failed=228 skipped=0 errors=0
  avito      processed=9302 matched=9302 house_resolved=9290 ...
  cian       processed=5158 matched=5158 house_resolved=5100 ...
  yandex     processed=3704 matched=3704 house_resolved=3540 ...
  n1         processed=264  matched=264  house_resolved=270  ...
final listing_sources coverage:
  avito      9302 / 9302 (100.0%)
  cian       5158 / 5158 (100.0%)
  ...
```

---

## Estimator backtest (issue #648)

### `backtest_estimator.py` — asking→sold accuracy harness

**STRICTLY READ-ONLY** (SELECT-only; no INSERT/UPDATE/DDL/commit). Measures the
estimator's asking-median + Tukey-IQR core against rosreestr ДКП **sold** prices.
For a sample of ДКП deals it predicts the asking median from nearby active
listings (reusing the estimator's own `_filter_outliers` / `_percentile`), then
reports per-deal signed/abs error % aggregated overall + per-rooms (студия / 1к /
2к / 3к / 4+), plus a city-wide deal-vs-asking headline spread.

```bash
DATABASE_URL=postgresql+psycopg://... \
    python -m scripts.backtest_estimator --sample 300 --since 2025-06-01

# machine-readable:
python -m scripts.backtest_estimator --json
```

**Stage 1 correction block.** On top of the raw `[ASKING]` metrics the harness
emits a second `[CORRECTED]` block: from the SAME matched sample it derives a
per-rooms asking→sold ratio `ratio[bucket] = median(sold_ppm2) /
median(pred_ask_ppm2)` (global fallback for buckets with `< MIN_BUCKET = 20`
matched deals), then re-scores `pred_sold = pred_ask * ratio[bucket]` through the
same metric math. This DEMONSTRATES that a per-rooms factor removes the
systematic +29.6% asking→sold bias — it changes nothing in prod.

> **Honesty:** by default the ratio is IN-SAMPLE (derived AND evaluated on the
> same deals), so the corrected bias is near-zero *by construction*. That proves
> the MECHANISM, not out-of-sample accuracy. Pass `--holdout-split` to fit on
> even-id deals and evaluate on the odd-id half (deterministic, no RNG) for an
> honest number. The production ratio (Stage 2) is fit over a SEPARATE window
> and A/B'd on held-out data.

```bash
# honest out-of-sample corrected number (even-id fit / odd-id eval):
python -m scripts.backtest_estimator --sample 600 --holdout-split
```

Caveats (also printed): CURRENT listings vs PAST deals (not point-in-time —
needs `listing_source_snapshots` #570); asking-median + IQR core only; ДКП =
registered price. Pure metric/ratio helpers are unit-tested in
`tests/test_backtest_estimator.py` (no DB).

---

## `domclick_local_runner.py` — DomClick EKB вторичка с домашнего IP (offline)

**SELF-CONTAINED** (без `app.*` импортов, stdlib + `playwright.async_api`).
Запускается ОПЕРАТОРОМ на его машине с **домашнего/резидентного IP** — DomClick
фронтит QRATOR, который банит datacenter/mobile-proxy, но пропускает домашний.
Пишет JSONL (НЕ в БД); заливка в trade-in БД — отдельным шагом `ingest_domclick_jsonl.py`.

**Транспорт — оба слоя через браузер** (`page.goto`, verified live 2026-06-27):
- **enumerate** — фронтовый SERP `ekaterinburg.domclick.ru/search?...` (SSR). Прямой
  BFF `bff-search-web/api/offers/v1` режет QRATOR-403 уже с домашнего IP, а фронтовый
  SERP грузится чисто. id офферов — из DOM (`a[href*="/card/"]`, ждём через
  `wait_for_selector` — карточки догидрируются ~6-12с), total — из `<title>`. Пагинация
  `offset` (page-size 20); при total>2000 — price-bisection бакета (границы из title).
- **detail** — `page.goto(card)` → сырой HTML (`__SSR_STATE__`: renovation, living/kitchen,
  priceHistory, egrnData owners/collateral, sale_type, views, wall/floor) **+ offer-card v3
  XHR** `price_prediction` (AVM заполненный) и `sold_similar` (`--with-sold-similar`) —
  они с домашнего IP отдают 200 (в отличие от прод-прокси). Layer-A
  (price/area/rooms/floor/total_floors/lat/lon/address) тоже из card-SSR.

### Режим окна

QRATOR ловит СТАРЫЙ headless (`launch(headless=True)`) → `403 | Домклик`. NEW-headless
И headed → `200` + полный `__SSR_STATE__`. Дефолт = new-headless (окно скрыто);
`--headed` = видимое окно (debug).

### Resume / enum-cache (рестарт не теряет позицию)

- **detail-resume**: при старте читает JSONL и пропускает id с терминальным статусом
  (ok/blocked/parse_fail).
- **`--enum-cache PATH`**: enumerate (~6400 id, ~1.5-2ч) пишет полный список id в файл с
  sentinel завершённости; **следующий запуск грузит кэш и ПРОПУСКАЕТ enumerate** → сразу
  detail. Композится с detail-resume → рестарт продолжает строго с текущей позиции, без
  пересбора и без перекачки готовых.

### Темп (НЕ спалить домашний IP)

- пауза между карточками `--min-delay` (12с) / `--max-delay` (30с);
- render-wait `--render-min` (6с) / `--render-max` (10с) — settle поверх `wait_for_selector`;
- «человеческий перерыв» каждые 25-40 карточек; на 403 — ретрай, потом `status="blocked"`.

Дефолт ≈ 20-34с/карта; для быстрее — `--min-delay 5 --max-delay 12 --render-min 3
--render-max 5` (~16-20с/карта; card-path чистый, запас есть). Полный свод ЕКБ (~6300) —
несколько ночей; держи ПК от сна (иначе wall-clock простаивает, данные не теряются).

```bash
cd tradein-mvp/backend
# боевой прогон с кэшем id + sold_similar (рекомендуется):
python scripts/domclick_local_runner.py --out domclick_ekb.jsonl \
    --enum-cache enum_cache.jsonl --with-sold-similar
# быстрый smoke (маленькие задержки ТОЛЬКО для теста):
python scripts/domclick_local_runner.py --limit 2 --min-delay 3 --max-delay 5 --out dc_test.jsonl
# только перечислить (без detail):
python scripts/domclick_local_runner.py --enumerate-only --out enum.jsonl
```

Нужен python с playwright (`python -c "import playwright"`).

## `ingest_domclick_jsonl.py` — JSONL раннера → trade-in БД

Запускается ВНУТРИ tradein-контейнера (импортит `app`). Читает JSONL раннера, на каждую
ok-запись: `ScrapedLot` → `save_listings()` (upsert + house-match хук в SAVEPOINT,
fault-tolerant — нематчнутый листинг всё равно вставляется) → `save_detail_enrichment()`
(detail-колонки COALESCE + `offer_price_history`). Идемпотентно (ON CONFLICT), `--limit`,
`--dry-run`. Запуск: `python -m scripts.ingest_domclick_jsonl --jsonl <path>` (PYTHONPATH=/app).
