bot-backend
480ecc90e5
Self-contained script (playwright chromium, NO app imports) для сбора DomClick EKB detail с домашнего residential IP (обходит QRATOR — verified live: 2 карточки ok, repair/areas/priceHistory/egrn/owners собраны). Enumerate via bff (Layer A) + card __SSR_STATE__ parse (Layer B) → JSONL (resume/idempotent). KEY: дифференциатор — РЕЖИМ HEADLESS, не только IP. С того же IP старый headless=True → 403, new-headless → 200 + SSR. Раннер дефолтит на new-headless. Пейсинг: рандом 12-30с между карточками, render-jitter 6-10с, human-break каждые 25-40 (60-180с), 1 retry на 403. ~100-150 карточек/час, full ~6400 ≈ 45-65ч. JSONL ингестится в tradein DB отдельным шагом. НЕ deploy — локальный ops-инструмент.
318 lines
14 KiB
Markdown
318 lines
14 KiB
Markdown
# tradein-mvp/backend/scripts/
|
||
|
||
Ops scripts that touch the production database directly. Run via `python -m
|
||
scripts.<name>` from the `backend/` working directory after `uv sync`.
|
||
|
||
All scripts are idempotent / resumable where they write — re-running the same
|
||
`--batch` label skips already-processed rows (UNIQUE constraints in target
|
||
tables). Failures inside a per-row loop never roll back the outer transaction;
|
||
each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/backend.md`.
|
||
|
||
---
|
||
|
||
## Production usage (canonical)
|
||
|
||
Scripts ship inside the `tradein-backend` image (PR F — `COPY scripts ./scripts`
|
||
в `backend/Dockerfile`). На VPS они уже в `/app/scripts/` — никаких manual
|
||
`docker cp` не нужно.
|
||
|
||
`YANDEX_GEOCODER_API_KEY` подтягивается из `/opt/gendesign/tradein-mvp/backend/
|
||
.env.runtime` через `env_file:` в `docker-compose.prod.yml` — никакого `-e` в
|
||
`docker exec` не нужно.
|
||
|
||
```bash
|
||
# Backfill (forward geocode 4170 houses без coords)
|
||
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'
|
||
|
||
# Audit-only (reverse geocode проверка для уже geocoded houses)
|
||
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'
|
||
|
||
# Canary first
|
||
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'
|
||
```
|
||
|
||
После изменения `backend/.env.runtime` нужен `--force-recreate` контейнера
|
||
(см. `.claude/rules/deploy.md`):
|
||
|
||
```bash
|
||
ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'
|
||
```
|
||
|
||
---
|
||
|
||
## Address audit + backfill (issue #582)
|
||
|
||
End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.
|
||
|
||
> Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell.
|
||
> На prod используй canonical `docker exec` команды из секции выше — там
|
||
> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`.
|
||
|
||
### `audit_address_mismatch.py` — Phase 1 baseline (PR #583)
|
||
|
||
Stratified-sample audit (200 EKB houses) comparing `houses.address` vs
|
||
Yandex Geocoder reverse lookup. Writes one row per house into
|
||
`address_mismatch_audit` with the snapped point + canonical address + distance.
|
||
|
||
```bash
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
YANDEX_GEOCODER_API_KEY=... \
|
||
uv run python -m scripts.audit_address_mismatch \
|
||
--batch 2026-05-25_run1 \
|
||
--limit-per-district 25
|
||
```
|
||
|
||
Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware,
|
||
4-7s sleep between calls). API tier free is 25k req/day → 200-row sample
|
||
takes ~10s with no quota concern.
|
||
|
||
Report:
|
||
|
||
```bash
|
||
psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
|
||
-f scripts/address_audit_report.sql
|
||
```
|
||
|
||
### `backfill_house_coords.py` — Phase 2-3 (PR for #582)
|
||
|
||
Two modes (`--audit-only` flag switches between them):
|
||
|
||
**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows
|
||
WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns
|
||
`precision='exact'` or `'number'` (skips street-only / locality matches).
|
||
Each processed row gets an `address_mismatch_audit` entry with status
|
||
`backfill` / `imprecise` / `no_match` / `error`.
|
||
|
||
```bash
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
YANDEX_GEOCODER_API_KEY=... \
|
||
uv run python -m scripts.backfill_house_coords \
|
||
--batch 2026-05-27_backfill
|
||
```
|
||
|
||
Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request):
|
||
20-25 min. Expected output split (rough baseline from Phase 1 numbers):
|
||
|
||
| Status | Approx rows | What it means |
|
||
|-------------|-------------|-----------------------------------------------------|
|
||
| `backfill` | ~3.3k–3.7k | UPDATE landed, lat/lon now populated |
|
||
| `imprecise` | ~300–500 | Match returned but precision too low — needs review |
|
||
| `no_match` | ~100–300 | Yandex couldn't resolve; address probably mangled |
|
||
| `error` | <50 | HTTP errors / timeouts — re-run picks them up |
|
||
|
||
**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write
|
||
audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`.
|
||
Does NOT modify the `houses` table.
|
||
|
||
```bash
|
||
uv run python -m scripts.backfill_house_coords \
|
||
--batch 2026-05-27_audit --audit-only
|
||
```
|
||
|
||
Combined budget for both phases (~8.6k requests) is well under the 25k/day
|
||
Geocoder free tier.
|
||
|
||
### Common ops
|
||
|
||
Canary first — run with `--limit 100` and inspect the audit table before
|
||
letting the full job loose:
|
||
|
||
```bash
|
||
uv run python -m scripts.backfill_house_coords \
|
||
--batch canary_$(date +%F) --limit 100
|
||
psql "$DATABASE_URL" -c "
|
||
SELECT audit_status, COUNT(*)
|
||
FROM address_mismatch_audit
|
||
WHERE audit_batch = 'canary_$(date +%F)'
|
||
GROUP BY audit_status;
|
||
"
|
||
```
|
||
|
||
Resume after crash / quota hit — same `--batch` label, the UNIQUE
|
||
`(house_id, audit_batch)` index skips finished rows:
|
||
|
||
```bash
|
||
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
|
||
# ... interruption ...
|
||
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
|
||
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"
|
||
```
|
||
|
||
### Helpers (not entry points)
|
||
|
||
- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`,
|
||
`reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API
|
||
paths share `_parse_api_payload` because Yandex's forward/reverse
|
||
envelopes have the same shape.
|
||
- `audit_address_sample.sql` — random sample for the Phase 1 audit (used
|
||
by `audit_address_mismatch.py`).
|
||
- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95
|
||
distance, top-20 outliers, per-district breakdown).
|
||
|
||
---
|
||
|
||
## Matching backfill (PR J)
|
||
|
||
### `backfill_listing_sources.py` — retroactive matching for ~18k listings
|
||
|
||
PR I (commit 7e24ccb) hooked the matching service into `save_listings()` so
|
||
every **new** scrape now writes a `listing_sources` row + resolves a canonical
|
||
`houses` row. This script does the same work retroactively for all
|
||
**existing** listings — `listing_sources` only had rows from new scrapes
|
||
post-PR I.
|
||
|
||
What it does per row:
|
||
|
||
1. `match_or_create_house()` (Tier 0-3) — uses `listings.house_source` /
|
||
`house_ext_id` when present (Avito Houses Catalog, Cian newbuilding),
|
||
else falls back to address/lat/lon/cadastrals.
|
||
2. `upsert_listing_source()` with `method='backfill'`, `confidence=0.9`
|
||
(vs real-time `source_link` 1.0 — distinguishes the two in audits).
|
||
3. `UPDATE listings.house_id_fk` when the row didn't already have one.
|
||
|
||
```bash
|
||
# Canary
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
uv run python -m scripts.backfill_listing_sources \
|
||
--limit 100 --dry-run
|
||
|
||
# Real run, one source at a time (staged rollout)
|
||
uv run python -m scripts.backfill_listing_sources --source avito
|
||
|
||
# Full run
|
||
uv run python -m scripts.backfill_listing_sources --batch-size 500
|
||
```
|
||
|
||
**Idempotent / resumable** — the source query is
|
||
`WHERE NOT EXISTS (SELECT 1 FROM listing_sources ls WHERE ls.ext_source =
|
||
listings.source AND ls.ext_id = COALESCE(listings.source_id,
|
||
listings.dedup_hash))`. Re-runs only pick up rows still missing from
|
||
`listing_sources`. `upsert_listing_source` adds a second layer of safety via
|
||
`ON CONFLICT (ext_source, ext_id) DO UPDATE`.
|
||
|
||
**No network calls** — pure in-DB matching (Yandex Geocoder is blocked on
|
||
prod, and `match_or_create_house` does not call it anyway).
|
||
|
||
**Per-row SAVEPOINT** (`db.begin_nested()`) per `.claude/rules/backend.md` —
|
||
one bad row never aborts the surrounding batch.
|
||
|
||
**Expected output (PR J initial run):**
|
||
|
||
| Source | Rows | Expected matched | Notes |
|
||
|--------|-------|-------------------|------------------------------------------------|
|
||
| avito | 9302 | 9000+ | Many carry `house_source`/`house_ext_id` |
|
||
| cian | 5158 | 5000+ | Most carry `house_source`/`house_ext_id` |
|
||
| yandex | 3704 | 3700+ | No `source_id` → uses `dedup_hash` as `ext_id` |
|
||
| n1 | 264 | 264 | All have address/coords |
|
||
|
||
**Expected duration**: rough estimate ~5-15 minutes on prod for ~18k rows
|
||
(advisory-lock + 1-3 DB roundtrips per listing for Tier 0-3, ~500 commit
|
||
checkpoints at default batch size). Run with `--limit 100` first to
|
||
calibrate, then let the full job loose.
|
||
|
||
Final summary in the log includes per-source coverage % so you can verify
|
||
the run landed:
|
||
|
||
```
|
||
backfill done (dry_run=False): processed=18428 matched=18428
|
||
house_resolved=18200 house_failed=228 skipped=0 errors=0
|
||
avito processed=9302 matched=9302 house_resolved=9290 ...
|
||
cian processed=5158 matched=5158 house_resolved=5100 ...
|
||
yandex processed=3704 matched=3704 house_resolved=3540 ...
|
||
n1 processed=264 matched=264 house_resolved=270 ...
|
||
final listing_sources coverage:
|
||
avito 9302 / 9302 (100.0%)
|
||
cian 5158 / 5158 (100.0%)
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
## Estimator backtest (issue #648)
|
||
|
||
### `backtest_estimator.py` — asking→sold accuracy harness
|
||
|
||
**STRICTLY READ-ONLY** (SELECT-only; no INSERT/UPDATE/DDL/commit). Measures the
|
||
estimator's asking-median + Tukey-IQR core against rosreestr ДКП **sold** prices.
|
||
For a sample of ДКП deals it predicts the asking median from nearby active
|
||
listings (reusing the estimator's own `_filter_outliers` / `_percentile`), then
|
||
reports per-deal signed/abs error % aggregated overall + per-rooms (студия / 1к /
|
||
2к / 3к / 4+), plus a city-wide deal-vs-asking headline spread.
|
||
|
||
```bash
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
python -m scripts.backtest_estimator --sample 300 --since 2025-06-01
|
||
|
||
# machine-readable:
|
||
python -m scripts.backtest_estimator --json
|
||
```
|
||
|
||
**Stage 1 correction block.** On top of the raw `[ASKING]` metrics the harness
|
||
emits a second `[CORRECTED]` block: from the SAME matched sample it derives a
|
||
per-rooms asking→sold ratio `ratio[bucket] = median(sold_ppm2) /
|
||
median(pred_ask_ppm2)` (global fallback for buckets with `< MIN_BUCKET = 20`
|
||
matched deals), then re-scores `pred_sold = pred_ask * ratio[bucket]` through the
|
||
same metric math. This DEMONSTRATES that a per-rooms factor removes the
|
||
systematic +29.6% asking→sold bias — it changes nothing in prod.
|
||
|
||
> **Honesty:** by default the ratio is IN-SAMPLE (derived AND evaluated on the
|
||
> same deals), so the corrected bias is near-zero *by construction*. That proves
|
||
> the MECHANISM, not out-of-sample accuracy. Pass `--holdout-split` to fit on
|
||
> even-id deals and evaluate on the odd-id half (deterministic, no RNG) for an
|
||
> honest number. The production ratio (Stage 2) is fit over a SEPARATE window
|
||
> and A/B'd on held-out data.
|
||
|
||
```bash
|
||
# honest out-of-sample corrected number (even-id fit / odd-id eval):
|
||
python -m scripts.backtest_estimator --sample 600 --holdout-split
|
||
```
|
||
|
||
Caveats (also printed): CURRENT listings vs PAST deals (not point-in-time —
|
||
needs `listing_source_snapshots` #570); asking-median + IQR core only; ДКП =
|
||
registered price. Pure metric/ratio helpers are unit-tested in
|
||
`tests/test_backtest_estimator.py` (no DB).
|
||
|
||
---
|
||
|
||
## `domclick_local_runner.py` — DomClick EKB вторичка с домашнего IP (offline)
|
||
|
||
**SELF-CONTAINED** (без `app.*` импортов, stdlib + `playwright.async_api`).
|
||
Запускается ОПЕРАТОРОМ на его машине с домашнего/резидентного IP — DomClick
|
||
фронтит QRATOR, который банит datacenter/mobile-proxy, но пропускает домашний.
|
||
Пишет JSONL (НЕ в БД); заливка в trade-in БД — отдельным ingest-шагом.
|
||
|
||
Собирает Layer A (BFF search: `/api/offers/v1` + `/api/offers/count/v1`,
|
||
room×price-бакетинг под cap offset<2000) + Layer B (detail-карточка,
|
||
`window.__SSR_STATE__`: renovation, living/kitchen, priceHistory, egrnData
|
||
owners/collateral, sale_type, views/calls, wall/floor type, AVM). Логика
|
||
скопирована инлайн из `app/services/scrapers/domclick.py` + `domclick_detail.py`.
|
||
|
||
### Режим окна (verified live 2026-06-27 на домашнем IP)
|
||
|
||
QRATOR ловит СТАРЫЙ headless (`launch(headless=True)`) → `403 | Домклик` даже
|
||
с домашнего IP. NEW-headless (`--headless=new`) И headed → `200` + полный
|
||
`__SSR_STATE__`. Поэтому дефолт = new-headless (окно скрыто, детект проходит);
|
||
`--headed` = видимое окно (debug). BFF-enumerate проходит в любом режиме.
|
||
|
||
### Темп (явное требование — НЕ спалить домашний IP)
|
||
|
||
- пауза между карточками `--min-delay` 12с / `--max-delay` 30с (`random.uniform`);
|
||
- render-wait после goto `--render-min` 6с / `--render-max` 10с;
|
||
- «человеческий перерыв» каждые 25-40 карточек → 60-180с;
|
||
- на 403 — один ретрай через 45-90с, потом `status="blocked"` (не долбит).
|
||
|
||
Дефолтный темп ≈ 100-150 карточек/час → полный свод ЕКБ (~6400) ≈ 45-65 ч
|
||
(несколько ночей). Скрипт РЕЗЮМИРУЕМ: при старте читает JSONL и пропускает
|
||
id с терминальным статусом (ok/blocked/parse_fail).
|
||
|
||
```bash
|
||
cd tradein-mvp/backend
|
||
# боевой прогон (дефолтный медленный темп):
|
||
python scripts/domclick_local_runner.py --out domclick_ekb.jsonl
|
||
# быстрый smoke (маленькие задержки ТОЛЬКО для теста):
|
||
python scripts/domclick_local_runner.py --limit 2 --min-delay 3 --max-delay 5 --out dc_test.jsonl
|
||
# только перечислить (без detail):
|
||
python scripts/domclick_local_runner.py --enumerate-only --out enum.jsonl
|
||
```
|
||
|
||
Запускать python'ом, где установлен playwright (`python -c "import playwright"`).
|