gendesign/tradein-mvp/backend/scripts

tradein-mvp/backend/scripts/

Ops scripts that touch the production database directly. Run via python -m scripts.<name> from the backend/ working directory after uv sync.

All scripts are idempotent / resumable where they write — re-running the same --batch label skips already-processed rows (UNIQUE constraints in target tables). Failures inside a per-row loop never roll back the outer transaction; each row is wrapped in a SAVEPOINT (db.begin_nested()) per .claude/rules/backend.md.

---

Production usage (canonical)

Scripts ship inside the tradein-backend image (PR F — COPY scripts ./scripts в backend/Dockerfile). На VPS они уже в /app/scripts/ — никаких manual docker cp не нужно.

YANDEX_GEOCODER_API_KEY подтягивается из /opt/gendesign/tradein-mvp/backend/ .env.runtime через env_file: в docker-compose.prod.yml — никакого -e в docker exec не нужно.

# Backfill (forward geocode 4170 houses без coords)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'

# Audit-only (reverse geocode проверка для уже geocoded houses)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'

# Canary first
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'

После изменения backend/.env.runtime нужен --force-recreate контейнера (см. .claude/rules/deploy.md):

ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'

---

Address audit + backfill (issue #582)

End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.

Локальные примеры ниже — для dev-машины с uv run и переменными в shell. На prod используй canonical docker exec команды из секции выше — там YANDEX_GEOCODER_API_KEY уже подгружен из backend/.env.runtime.

audit_address_mismatch.py — Phase 1 baseline (PR #583)

Stratified-sample audit (200 EKB houses) comparing houses.address vs Yandex Geocoder reverse lookup. Writes one row per house into address_mismatch_audit with the snapped point + canonical address + distance.

DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.audit_address_mismatch \
    --batch 2026-05-25_run1 \
    --limit-per-district 25

Mode auto picks API if the key is set, otherwise Playwright (CAPTCHA-aware, 4-7s sleep between calls). API tier free is 25k req/day → 200-row sample takes ~10s with no quota concern.

Report:

psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
    -f scripts/address_audit_report.sql

backfill_house_coords.py — Phase 2-3 (PR for #582)

Two modes (--audit-only flag switches between them):

Backfill (default) — forward-geocode houses.address for the ~4141 rows WHERE lat IS NULL OR lon IS NULL. Only writes back if Yandex returns precision='exact' or 'number' (skips street-only / locality matches). Each processed row gets an address_mismatch_audit entry with status backfill / imprecise / no_match / error.

DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_backfill

Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request): 20-25 min. Expected output split (rough baseline from Phase 1 numbers):

Status	Approx rows	What it means
backfill	~3.3k–3.7k	UPDATE landed, lat/lon now populated
imprecise	~300–500	Match returned but precision too low — needs review
no_match	~100–300	Yandex couldn't resolve; address probably mangled
error	<50	HTTP errors / timeouts — re-run picks them up

Audit-only — reverse-geocode the ~4452 houses WITH coords, write audit rows with status ok (≤50m) / mismatch (>50m) / no_match / error. Does NOT modify the houses table.

uv run python -m scripts.backfill_house_coords \
    --batch 2026-05-27_audit --audit-only

Combined budget for both phases (~8.6k requests) is well under the 25k/day Geocoder free tier.

Common ops

Canary first — run with --limit 100 and inspect the audit table before letting the full job loose:

uv run python -m scripts.backfill_house_coords \
    --batch canary_$(date +%F) --limit 100
psql "$DATABASE_URL" -c "
  SELECT audit_status, COUNT(*)
    FROM address_mismatch_audit
   WHERE audit_batch = 'canary_$(date +%F)'
   GROUP BY audit_status;
"

Resume after crash / quota hit — same --batch label, the UNIQUE (house_id, audit_batch) index skips finished rows:

uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# ... interruption ...
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"

Helpers (not entry points)

• _yandex_reverse.py — forward_via_api(), reverse_via_api(), reverse_via_playwright(), YandexReverseResult dataclass. Both API paths share _parse_api_payload because Yandex's forward/reverse envelopes have the same shape.
• audit_address_sample.sql — random sample for the Phase 1 audit (used by audit_address_mismatch.py).
• address_audit_report.sql — psql-driven post-run summary (p50/p75/p95 distance, top-20 outliers, per-district breakdown).

---

Matching backfill (PR J)

backfill_listing_sources.py — retroactive matching for ~18k listings

PR I (commit 7e24ccb) hooked the matching service into save_listings() so every new scrape now writes a listing_sources row + resolves a canonical houses row. This script does the same work retroactively for all existing listings — listing_sources only had rows from new scrapes post-PR I.

What it does per row:

1. match_or_create_house() (Tier 0-3) — uses listings.house_source / house_ext_id when present (Avito Houses Catalog, Cian newbuilding), else falls back to address/lat/lon/cadastrals.
2. upsert_listing_source() with method='backfill', confidence=0.9 (vs real-time source_link 1.0 — distinguishes the two in audits).
3. UPDATE listings.house_id_fk when the row didn't already have one.

# Canary
DATABASE_URL=postgresql+psycopg://... \
    uv run python -m scripts.backfill_listing_sources \
        --limit 100 --dry-run

# Real run, one source at a time (staged rollout)
uv run python -m scripts.backfill_listing_sources --source avito

# Full run
uv run python -m scripts.backfill_listing_sources --batch-size 500

Idempotent / resumable — the source query is WHERE NOT EXISTS (SELECT 1 FROM listing_sources ls WHERE ls.ext_source = listings.source AND ls.ext_id = COALESCE(listings.source_id, listings.dedup_hash)). Re-runs only pick up rows still missing from listing_sources. upsert_listing_source adds a second layer of safety via ON CONFLICT (ext_source, ext_id) DO UPDATE.

No network calls — pure in-DB matching (Yandex Geocoder is blocked on prod, and match_or_create_house does not call it anyway).

Per-row SAVEPOINT (db.begin_nested()) per .claude/rules/backend.md — one bad row never aborts the surrounding batch.

Expected output (PR J initial run):

Source	Rows	Expected matched	Notes
avito	9302	9000+	Many carry house_source/house_ext_id
cian	5158	5000+	Most carry house_source/house_ext_id
yandex	3704	3700+	No source_id → uses dedup_hash as ext_id
n1	264	264	All have address/coords

Expected duration: rough estimate ~5-15 minutes on prod for ~18k rows (advisory-lock + 1-3 DB roundtrips per listing for Tier 0-3, ~500 commit checkpoints at default batch size). Run with --limit 100 first to calibrate, then let the full job loose.

Final summary in the log includes per-source coverage % so you can verify the run landed:

backfill done (dry_run=False): processed=18428 matched=18428
  house_resolved=18200 house_failed=228 skipped=0 errors=0
  avito      processed=9302 matched=9302 house_resolved=9290 ...
  cian       processed=5158 matched=5158 house_resolved=5100 ...
  yandex     processed=3704 matched=3704 house_resolved=3540 ...
  n1         processed=264  matched=264  house_resolved=270  ...
final listing_sources coverage:
  avito      9302 / 9302 (100.0%)
  cian       5158 / 5158 (100.0%)
  ...

---

Estimator backtest (issue #648)

backtest_estimator.py — asking→sold accuracy harness

STRICTLY READ-ONLY (SELECT-only; no INSERT/UPDATE/DDL/commit). Measures the estimator's asking-median + Tukey-IQR core against rosreestr ДКП sold prices. For a sample of ДКП deals it predicts the asking median from nearby active listings (reusing the estimator's own _filter_outliers / _percentile), then reports per-deal signed/abs error % aggregated overall + per-rooms (студия / 1к / 2к / 3к / 4+), plus a city-wide deal-vs-asking headline spread.

DATABASE_URL=postgresql+psycopg://... \
    python -m scripts.backtest_estimator --sample 300 --since 2025-06-01

# machine-readable:
python -m scripts.backtest_estimator --json

Stage 1 correction block. On top of the raw [ASKING] metrics the harness emits a second [CORRECTED] block: from the SAME matched sample it derives a per-rooms asking→sold ratio ratio[bucket] = median(sold_ppm2) / median(pred_ask_ppm2) (global fallback for buckets with < MIN_BUCKET = 20 matched deals), then re-scores pred_sold = pred_ask * ratio[bucket] through the same metric math. This DEMONSTRATES that a per-rooms factor removes the systematic +29.6% asking→sold bias — it changes nothing in prod.

Honesty: by default the ratio is IN-SAMPLE (derived AND evaluated on the same deals), so the corrected bias is near-zero by construction. That proves the MECHANISM, not out-of-sample accuracy. Pass --holdout-split to fit on even-id deals and evaluate on the odd-id half (deterministic, no RNG) for an honest number. The production ratio (Stage 2) is fit over a SEPARATE window and A/B'd on held-out data.

# honest out-of-sample corrected number (even-id fit / odd-id eval):
python -m scripts.backtest_estimator --sample 600 --holdout-split

Caveats (also printed): CURRENT listings vs PAST deals (not point-in-time — needs listing_source_snapshots #570); asking-median + IQR core only; ДКП = registered price. Pure metric/ratio helpers are unit-tested in tests/test_backtest_estimator.py (no DB).

---

domclick_local_runner.py — DomClick EKB вторичка с домашнего IP (offline)

SELF-CONTAINED (без app.* импортов, stdlib + playwright.async_api). Запускается ОПЕРАТОРОМ на его машине с домашнего/резидентного IP — DomClick фронтит QRATOR, который банит datacenter/mobile-proxy, но пропускает домашний. Пишет JSONL (НЕ в БД); заливка в trade-in БД — отдельным шагом ingest_domclick_jsonl.py.

Транспорт — оба слоя через браузер (page.goto, verified live 2026-06-27):

• enumerate — фронтовый SERP ekaterinburg.domclick.ru/search?... (SSR). Прямой BFF bff-search-web/api/offers/v1 режет QRATOR-403 уже с домашнего IP, а фронтовый SERP грузится чисто. id офферов — из DOM (a[href*="/card/"], ждём через wait_for_selector — карточки догидрируются ~6-12с), total — из <title>. Пагинация offset (page-size 20); при total>2000 — price-bisection бакета (границы из title).
• detail — page.goto(card) → сырой HTML (__SSR_STATE__: renovation, living/kitchen, priceHistory, egrnData owners/collateral, sale_type, views, wall/floor) + offer-card v3 XHR price_prediction (AVM заполненный) и sold_similar (--with-sold-similar) — они с домашнего IP отдают 200 (в отличие от прод-прокси). Layer-A (price/area/rooms/floor/total_floors/lat/lon/address) тоже из card-SSR.

Режим окна

QRATOR ловит СТАРЫЙ headless (launch(headless=True)) → 403 | Домклик. NEW-headless И headed → 200 + полный __SSR_STATE__. Дефолт = new-headless (окно скрыто); --headed = видимое окно (debug).

Resume / enum-cache (рестарт не теряет позицию)

• detail-resume: при старте читает JSONL и пропускает id с терминальным статусом (ok/blocked/parse_fail).
• --enum-cache PATH: enumerate (~6400 id, ~1.5-2ч) пишет полный список id в файл с sentinel завершённости; следующий запуск грузит кэш и ПРОПУСКАЕТ enumerate → сразу detail. Композится с detail-resume → рестарт продолжает строго с текущей позиции, без пересбора и без перекачки готовых.

Темп (НЕ спалить домашний IP)

• пауза между карточками --min-delay (12с) / --max-delay (30с);
• render-wait --render-min (6с) / --render-max (10с) — settle поверх wait_for_selector;
• «человеческий перерыв» каждые 25-40 карточек; на 403 — ретрай, потом status="blocked".

Дефолт ≈ 20-34с/карта; для быстрее — --min-delay 5 --max-delay 12 --render-min 3 --render-max 5 (~16-20с/карта; card-path чистый, запас есть). Полный свод ЕКБ (~6300) — несколько ночей; держи ПК от сна (иначе wall-clock простаивает, данные не теряются).

cd tradein-mvp/backend
# боевой прогон с кэшем id + sold_similar (рекомендуется):
python scripts/domclick_local_runner.py --out domclick_ekb.jsonl \
    --enum-cache enum_cache.jsonl --with-sold-similar
# быстрый smoke (маленькие задержки ТОЛЬКО для теста):
python scripts/domclick_local_runner.py --limit 2 --min-delay 3 --max-delay 5 --out dc_test.jsonl
# только перечислить (без detail):
python scripts/domclick_local_runner.py --enumerate-only --out enum.jsonl

Нужен python с playwright (python -c "import playwright").

ingest_domclick_jsonl.py — JSONL раннера → trade-in БД

Запускается ВНУТРИ tradein-контейнера (импортит app). Читает JSONL раннера, на каждую ok-запись: ScrapedLot → save_listings() (upsert + house-match хук в SAVEPOINT, fault-tolerant — нематчнутый листинг всё равно вставляется) → save_detail_enrichment() (detail-колонки COALESCE + offer_price_history). Идемпотентно (ON CONFLICT), --limit, --dry-run. Запуск: python -m scripts.ingest_domclick_jsonl --jsonl <path> (PYTHONPATH=/app).