feff55bea9
All checks were successful
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 11s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 2m33s
Удалены мёртвые ops-скрипты Yandex Geocoder (уже недостижимы после #2593 частей 1-2): - scripts/_yandex_reverse.py, scripts/audit_address_mismatch.py, scripts/backfill_house_coords.py - их тесты + осиротевшая фикстура tests/fixtures/yandex_geocode_sample.json - осиротевшие SQL-хелперы scripts/audit_address_sample.sql, scripts/address_audit_report.sql (использовались только audit_address_mismatch.py) Обновлена документация (осиротевшие упоминания YANDEX_GEOCODER_API_KEY / удалённых скриптов): scripts/README.md, tradein-mvp/DEPLOY.md, docs/Secrets_Rotation_Policy.md. Добавлен tests/test_geocoder_nominatim_lookup.py — покрывает _nominatim_lookup (единственный живой внешний геокодер) на предмет реальной передачи city_hint в исходящий HTTP-запрос к Nominatim; закрывает дыру в coverage, оставленную удалёнными yandex-тестами. Refs #2593
216 lines
12 KiB
Markdown
216 lines
12 KiB
Markdown
# tradein-mvp/backend/scripts/
|
||
|
||
Ops scripts that touch the production database directly. Run via `python -m
|
||
scripts.<name>` from the `backend/` working directory after `uv sync`.
|
||
|
||
All scripts are idempotent / resumable where they write — re-running the same
|
||
`--batch` label skips already-processed rows (UNIQUE constraints in target
|
||
tables). Failures inside a per-row loop never roll back the outer transaction;
|
||
each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/backend.md`.
|
||
|
||
---
|
||
|
||
## Address audit + backfill (issue #582) — REMOVED (#2593)
|
||
|
||
`audit_address_mismatch.py`, `backfill_house_coords.py`, `_yandex_reverse.py`
|
||
и их SQL-хелперы (`audit_address_sample.sql`, `address_audit_report.sql`)
|
||
удалены — весь pipeline опирался на Yandex Geocoder API, который выпилен
|
||
из проекта (#2593, части 1-3). `houses.address`→lat/lon geocoding теперь
|
||
идёт через `app/services/geocoder.py` (кадастр/геопортал ЕКБ-тиры + Nominatim
|
||
fallback, единственный живой внешний провайдер) на обычном write-path
|
||
(`/api/v1/trade-in/estimate`, listing ingest). Разовый forward-backfill
|
||
недостающих `houses` координат — `scripts/geocode_deals_nominatim.py`
|
||
(живой, работает с `rosreestr_deals`, не с `houses` — читай его docstring
|
||
перед использованием на других таблицах). Таблица `address_mismatch_audit`
|
||
осталась в схеме (используется `house_dedup_merge.py` при слиянии дублей
|
||
домов, независимо от Yandex-аудита).
|
||
|
||
---
|
||
|
||
## Matching backfill (PR J)
|
||
|
||
### `backfill_listing_sources.py` — retroactive matching for ~18k listings
|
||
|
||
PR I (commit 7e24ccb) hooked the matching service into `save_listings()` so
|
||
every **new** scrape now writes a `listing_sources` row + resolves a canonical
|
||
`houses` row. This script does the same work retroactively for all
|
||
**existing** listings — `listing_sources` only had rows from new scrapes
|
||
post-PR I.
|
||
|
||
What it does per row:
|
||
|
||
1. `match_or_create_house()` (Tier 0-3) — uses `listings.house_source` /
|
||
`house_ext_id` when present (Avito Houses Catalog, Cian newbuilding),
|
||
else falls back to address/lat/lon/cadastrals.
|
||
2. `upsert_listing_source()` with `method='backfill'`, `confidence=0.9`
|
||
(vs real-time `source_link` 1.0 — distinguishes the two in audits).
|
||
3. `UPDATE listings.house_id_fk` when the row didn't already have one.
|
||
|
||
```bash
|
||
# Canary
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
uv run python -m scripts.backfill_listing_sources \
|
||
--limit 100 --dry-run
|
||
|
||
# Real run, one source at a time (staged rollout)
|
||
uv run python -m scripts.backfill_listing_sources --source avito
|
||
|
||
# Full run
|
||
uv run python -m scripts.backfill_listing_sources --batch-size 500
|
||
```
|
||
|
||
**Idempotent / resumable** — the source query is
|
||
`WHERE NOT EXISTS (SELECT 1 FROM listing_sources ls WHERE ls.ext_source =
|
||
listings.source AND ls.ext_id = COALESCE(listings.source_id,
|
||
listings.dedup_hash))`. Re-runs only pick up rows still missing from
|
||
`listing_sources`. `upsert_listing_source` adds a second layer of safety via
|
||
`ON CONFLICT (ext_source, ext_id) DO UPDATE`.
|
||
|
||
**No network calls** — pure in-DB matching (Yandex Geocoder is blocked on
|
||
prod, and `match_or_create_house` does not call it anyway).
|
||
|
||
**Per-row SAVEPOINT** (`db.begin_nested()`) per `.claude/rules/backend.md` —
|
||
one bad row never aborts the surrounding batch.
|
||
|
||
**Expected output (PR J initial run):**
|
||
|
||
| Source | Rows | Expected matched | Notes |
|
||
|--------|-------|-------------------|------------------------------------------------|
|
||
| avito | 9302 | 9000+ | Many carry `house_source`/`house_ext_id` |
|
||
| cian | 5158 | 5000+ | Most carry `house_source`/`house_ext_id` |
|
||
| yandex | 3704 | 3700+ | No `source_id` → uses `dedup_hash` as `ext_id` |
|
||
| n1 | 264 | 264 | All have address/coords |
|
||
|
||
**Expected duration**: rough estimate ~5-15 minutes on prod for ~18k rows
|
||
(advisory-lock + 1-3 DB roundtrips per listing for Tier 0-3, ~500 commit
|
||
checkpoints at default batch size). Run with `--limit 100` first to
|
||
calibrate, then let the full job loose.
|
||
|
||
Final summary in the log includes per-source coverage % so you can verify
|
||
the run landed:
|
||
|
||
```
|
||
backfill done (dry_run=False): processed=18428 matched=18428
|
||
house_resolved=18200 house_failed=228 skipped=0 errors=0
|
||
avito processed=9302 matched=9302 house_resolved=9290 ...
|
||
cian processed=5158 matched=5158 house_resolved=5100 ...
|
||
yandex processed=3704 matched=3704 house_resolved=3540 ...
|
||
n1 processed=264 matched=264 house_resolved=270 ...
|
||
final listing_sources coverage:
|
||
avito 9302 / 9302 (100.0%)
|
||
cian 5158 / 5158 (100.0%)
|
||
...
|
||
```
|
||
|
||
---
|
||
|
||
## Estimator backtest (issue #648)
|
||
|
||
### `backtest_estimator.py` — asking→sold accuracy harness
|
||
|
||
**STRICTLY READ-ONLY** (SELECT-only; no INSERT/UPDATE/DDL/commit). Measures the
|
||
estimator's asking-median + Tukey-IQR core against rosreestr ДКП **sold** prices.
|
||
For a sample of ДКП deals it predicts the asking median from nearby active
|
||
listings (reusing the estimator's own `_filter_outliers` / `_percentile`), then
|
||
reports per-deal signed/abs error % aggregated overall + per-rooms (студия / 1к /
|
||
2к / 3к / 4+), plus a city-wide deal-vs-asking headline spread.
|
||
|
||
```bash
|
||
DATABASE_URL=postgresql+psycopg://... \
|
||
python -m scripts.backtest_estimator --sample 300 --since 2025-06-01
|
||
|
||
# machine-readable:
|
||
python -m scripts.backtest_estimator --json
|
||
```
|
||
|
||
**Stage 1 correction block.** On top of the raw `[ASKING]` metrics the harness
|
||
emits a second `[CORRECTED]` block: from the SAME matched sample it derives a
|
||
per-rooms asking→sold ratio `ratio[bucket] = median(sold_ppm2) /
|
||
median(pred_ask_ppm2)` (global fallback for buckets with `< MIN_BUCKET = 20`
|
||
matched deals), then re-scores `pred_sold = pred_ask * ratio[bucket]` through the
|
||
same metric math. This DEMONSTRATES that a per-rooms factor removes the
|
||
systematic +29.6% asking→sold bias — it changes nothing in prod.
|
||
|
||
> **Honesty:** by default the ratio is IN-SAMPLE (derived AND evaluated on the
|
||
> same deals), so the corrected bias is near-zero *by construction*. That proves
|
||
> the MECHANISM, not out-of-sample accuracy. Pass `--holdout-split` to fit on
|
||
> even-id deals and evaluate on the odd-id half (deterministic, no RNG) for an
|
||
> honest number. The production ratio (Stage 2) is fit over a SEPARATE window
|
||
> and A/B'd on held-out data.
|
||
|
||
```bash
|
||
# honest out-of-sample corrected number (even-id fit / odd-id eval):
|
||
python -m scripts.backtest_estimator --sample 600 --holdout-split
|
||
```
|
||
|
||
Caveats (also printed): CURRENT listings vs PAST deals (not point-in-time —
|
||
needs `listing_source_snapshots` #570); asking-median + IQR core only; ДКП =
|
||
registered price. Pure metric/ratio helpers are unit-tested in
|
||
`tests/test_backtest_estimator.py` (no DB).
|
||
|
||
---
|
||
|
||
## `domclick_local_runner.py` — DomClick EKB вторичка с домашнего IP (offline)
|
||
|
||
**SELF-CONTAINED** (без `app.*` импортов, stdlib + `playwright.async_api`).
|
||
Запускается ОПЕРАТОРОМ на его машине с **домашнего/резидентного IP** — DomClick
|
||
фронтит QRATOR, который банит datacenter/mobile-proxy, но пропускает домашний.
|
||
Пишет JSONL (НЕ в БД); заливка в trade-in БД — отдельным шагом `ingest_domclick_jsonl.py`.
|
||
|
||
**Транспорт — оба слоя через браузер** (`page.goto`, verified live 2026-06-27):
|
||
- **enumerate** — фронтовый SERP `ekaterinburg.domclick.ru/search?...` (SSR). Прямой
|
||
BFF `bff-search-web/api/offers/v1` режет QRATOR-403 уже с домашнего IP, а фронтовый
|
||
SERP грузится чисто. id офферов — из DOM (`a[href*="/card/"]`, ждём через
|
||
`wait_for_selector` — карточки догидрируются ~6-12с), total — из `<title>`. Пагинация
|
||
`offset` (page-size 20); при total>2000 — price-bisection бакета (границы из title).
|
||
- **detail** — `page.goto(card)` → сырой HTML (`__SSR_STATE__`: renovation, living/kitchen,
|
||
priceHistory, egrnData owners/collateral, sale_type, views, wall/floor) **+ offer-card v3
|
||
XHR** `price_prediction` (AVM заполненный) и `sold_similar` (`--with-sold-similar`) —
|
||
они с домашнего IP отдают 200 (в отличие от прод-прокси). Layer-A
|
||
(price/area/rooms/floor/total_floors/lat/lon/address) тоже из card-SSR.
|
||
|
||
### Режим окна
|
||
|
||
QRATOR ловит СТАРЫЙ headless (`launch(headless=True)`) → `403 | Домклик`. NEW-headless
|
||
И headed → `200` + полный `__SSR_STATE__`. Дефолт = new-headless (окно скрыто);
|
||
`--headed` = видимое окно (debug).
|
||
|
||
### Resume / enum-cache (рестарт не теряет позицию)
|
||
|
||
- **detail-resume**: при старте читает JSONL и пропускает id с терминальным статусом
|
||
(ok/blocked/parse_fail).
|
||
- **`--enum-cache PATH`**: enumerate (~6400 id, ~1.5-2ч) пишет полный список id в файл с
|
||
sentinel завершённости; **следующий запуск грузит кэш и ПРОПУСКАЕТ enumerate** → сразу
|
||
detail. Композится с detail-resume → рестарт продолжает строго с текущей позиции, без
|
||
пересбора и без перекачки готовых.
|
||
|
||
### Темп (НЕ спалить домашний IP)
|
||
|
||
- пауза между карточками `--min-delay` (12с) / `--max-delay` (30с);
|
||
- render-wait `--render-min` (6с) / `--render-max` (10с) — settle поверх `wait_for_selector`;
|
||
- «человеческий перерыв» каждые 25-40 карточек; на 403 — ретрай, потом `status="blocked"`.
|
||
|
||
Дефолт ≈ 20-34с/карта; для быстрее — `--min-delay 5 --max-delay 12 --render-min 3
|
||
--render-max 5` (~16-20с/карта; card-path чистый, запас есть). Полный свод ЕКБ (~6300) —
|
||
несколько ночей; держи ПК от сна (иначе wall-clock простаивает, данные не теряются).
|
||
|
||
```bash
|
||
cd tradein-mvp/backend
|
||
# боевой прогон с кэшем id + sold_similar (рекомендуется):
|
||
python scripts/domclick_local_runner.py --out domclick_ekb.jsonl \
|
||
--enum-cache enum_cache.jsonl --with-sold-similar
|
||
# быстрый smoke (маленькие задержки ТОЛЬКО для теста):
|
||
python scripts/domclick_local_runner.py --limit 2 --min-delay 3 --max-delay 5 --out dc_test.jsonl
|
||
# только перечислить (без detail):
|
||
python scripts/domclick_local_runner.py --enumerate-only --out enum.jsonl
|
||
```
|
||
|
||
Нужен python с playwright (`python -c "import playwright"`).
|
||
|
||
## `ingest_domclick_jsonl.py` — JSONL раннера → trade-in БД
|
||
|
||
Запускается ВНУТРИ tradein-контейнера (импортит `app`). Читает JSONL раннера, на каждую
|
||
ok-запись: `ScrapedLot` → `save_listings()` (upsert + house-match хук в SAVEPOINT,
|
||
fault-tolerant — нематчнутый листинг всё равно вставляется) → `save_detail_enrichment()`
|
||
(detail-колонки COALESCE + `offer_price_history`). Идемпотентно (ON CONFLICT), `--limit`,
|
||
`--dry-run`. Запуск: `python -m scripts.ingest_domclick_jsonl --jsonl <path>` (PYTHONPATH=/app).
|