Merge pull request 'chore(tradein/geocoder): удалить остатки скриптов Яндекс-геокодера, часть 3 (#2593)' (#2599) from chore/tradein-remove-yandex-scripts into main

docs/Secrets_Rotation_Policy.md

@@ -77,7 +77,6 @@
 |---|---|---|
 | `TRADEIN_POSTGRES_PASSWORD` / `TRADEIN_POSTGRES_USER` | Пароль/юзер БД `tradein` | **E** |
 | `TRADEIN_READER_PASSWORD` | Пароль роли `gendesign_reader` (ETL #976, `ops/db-bootstrap/set_gendesign_reader_password.sql`) | **E** |
-| `YANDEX_GEOCODER_API_KEY` | Yandex Geocoder (25k req/day) | **D** |
 | `DADATA_API_TOKEN` / `DADATA_API_SECRET` | DaData `/clean/address` enrichment | **D** |
 | `SCRAPER_PROXY_URL` (+ legacy `AVITO_PROXY_URL`, `CIAN_PROXY_URL`, `YANDEX_PROXY_URL` и их `*_ROTATE_URL`) | Мобильный прокси для скраперов (содержит user:pass в URL) | **G** (proxy creds) |
 | `CIAN_LOGIN_EMAIL` / `CIAN_LOGIN_PASSWORD` | Cian browser auto-login (#639, Variant B) | **D** |
@@ -147,14 +146,14 @@ bcrypt-хеши — односторонние, не plaintext-секреты,
 3. Frontend: обновить `GLITCHTIP_FRONTEND_DSN` (build-arg `NEXT_PUBLIC_GLITCHTIP_DSN`) → требует **rebuild frontend образа** (запекается на build-time) → `workflow_dispatch` или push в `frontend/**`.
 4. Vault entry.
 
-### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`)
+### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`)
 
 **Downtime:** нет (фичи gracefully degrade при пустом ключе — см. config-комментарии).
 
-1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / Yandex Cloud / DaData / Cian-аккаунт).
+1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / DaData / Cian-аккаунт).
 2. Где живёт:
    - `OBJECTIVE_API_KEY`, `OPENAI_API_KEY` — Forgejo secret → deploy пишет в main `.env.runtime`.
-   - `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI).
+   - `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI).
 3. Обновить значение `sed`-ом (НЕ перезапись файла) и `up -d --force-recreate --no-deps backend worker beat` (main) / `... backend scraper` (tradein).
 4. Vault entry.
 

tradein-mvp/.env.example

@@ -6,12 +6,6 @@ DATABASE_URL=postgresql+psycopg://tradein:tradein@postgres:5432/tradein
 CORS_ORIGINS=["http://localhost:8080","http://localhost:3000"]
 ENVIRONMENT=dev
 
-# Yandex Geocoder API key (25k req/day free tier).
-# Required for backfill scripts (scripts/backfill_house_coords.py + audit_address_mismatch.py).
-# Empty = Nominatim fallback для backend геокодинга; backfill scripts требуют этот ключ
-# и упадут с SystemExit без него.
-YANDEX_GEOCODER_API_KEY=
-
 # DaData /clean/address — обогащение target адреса в estimate flow (PR Q1).
 # Возвращает canonical-форму, kadastr_num, ФИАС, координаты, ближайшее метро.
 # Demo tier: 100 req/день — хватит для тестов и low-traffic prod.

tradein-mvp/DEPLOY.md

@@ -57,8 +57,7 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment
    shell-скриптом deploy через `source .env.runtime` перед `compose up`.
 2. `/opt/gendesign/tradein-mvp/backend/.env.runtime` — переменные внутри
    контейнера `tradein-backend` (читаются через `env_file:` в compose). Сюда
-   попадают `YANDEX_GEOCODER_API_KEY`, `COOKIE_ENCRYPTION_KEY` —
-   всё, что нужно scripts/backfill_house_coords.py и application code внутри
+   попадают `COOKIE_ENCRYPTION_KEY` и остальные application-секреты внутри
    контейнера.
 
 ```bash
@@ -66,7 +65,6 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment
 TRADEIN_POSTGRES_USER=tradein
 TRADEIN_POSTGRES_PASSWORD=<сгенерировать openssl rand -hex 32>
 TRADEIN_CONTACT_EMAIL=tradein@gendsgn.ru
-YANDEX_GEOCODER_API_KEY=  # пусто пока, Nominatim fallback работает
 
 # Encryption key for Cian session cookies (pgp_sym_encrypt / Stage 9 Calculator).
 # Empty = Valuation Calculator scraper disabled + /api/v1/cookies/upload returns 503.
@@ -77,10 +75,9 @@ COOKIE_ENCRYPTION_KEY=<64-char hex>
 
 ```bash
 # /opt/gendesign/tradein-mvp/backend/.env.runtime — те же ключи которые
-# читаются ВНУТРИ container'а (scripts/backfill_house_coords.py, app/*).
+# читаются ВНУТРИ container'а (app/*, scripts/*.py).
 # Может быть симлинком на ../.env.runtime если переменные совпадают:
 #   ln -s ../.env.runtime /opt/gendesign/tradein-mvp/backend/.env.runtime
-YANDEX_GEOCODER_API_KEY=<key или пусто>
 COOKIE_ENCRYPTION_KEY=<64-char hex>
 GENDESIGN_FDW_PASSWORD=<password или пусто>
 GLITCHTIP_DSN=<dsn или пусто>
@@ -200,7 +197,6 @@ cat > tradein-mvp/.env.runtime <<EOF
 TRADEIN_POSTGRES_USER=tradein
 TRADEIN_POSTGRES_PASSWORD=$(openssl rand -hex 32)
 TRADEIN_CONTACT_EMAIL=tradein@gendsgn.ru
-YANDEX_GEOCODER_API_KEY=
 EOF
 chmod 600 tradein-mvp/.env.runtime
 

tradein-mvp/backend/scripts/README.md

@@ -10,144 +10,20 @@ each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/back
 
 ---
 
-## Production usage (canonical)
+## Address audit + backfill (issue #582) — REMOVED (#2593)
 
-Scripts ship inside the `tradein-backend` image (PR F — `COPY scripts ./scripts`
-в `backend/Dockerfile`). На VPS они уже в `/app/scripts/` — никаких manual
-`docker cp` не нужно.
-
-`YANDEX_GEOCODER_API_KEY` подтягивается из `/opt/gendesign/tradein-mvp/backend/
-.env.runtime` через `env_file:` в `docker-compose.prod.yml` — никакого `-e` в
-`docker exec` не нужно.
-
-```bash
-# Backfill (forward geocode 4170 houses без coords)
-ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'
-
-# Audit-only (reverse geocode проверка для уже geocoded houses)
-ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'
-
-# Canary first
-ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'
-```
-
-После изменения `backend/.env.runtime` нужен `--force-recreate` контейнера
-(см. `.claude/rules/deploy.md`):
-
-```bash
-ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'
-```
-
----
-
-## Address audit + backfill (issue #582)
-
-End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.
-
-> Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell.
-> На prod используй canonical `docker exec` команды из секции выше — там
-> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`.
-
-### `audit_address_mismatch.py` — Phase 1 baseline (PR #583)
-
-Stratified-sample audit (200 EKB houses) comparing `houses.address` vs
-Yandex Geocoder reverse lookup. Writes one row per house into
-`address_mismatch_audit` with the snapped point + canonical address + distance.
-
-```bash
-DATABASE_URL=postgresql+psycopg://... \
-YANDEX_GEOCODER_API_KEY=... \
-uv run python -m scripts.audit_address_mismatch \
-    --batch 2026-05-25_run1 \
-    --limit-per-district 25
-```
-
-Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware,
-4-7s sleep between calls). API tier free is 25k req/day → 200-row sample
-takes ~10s with no quota concern.
-
-Report:
-
-```bash
-psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
-    -f scripts/address_audit_report.sql
-```
-
-### `backfill_house_coords.py` — Phase 2-3 (PR for #582)
-
-Two modes (`--audit-only` flag switches between them):
-
-**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows
-WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns
-`precision='exact'` or `'number'` (skips street-only / locality matches).
-Each processed row gets an `address_mismatch_audit` entry with status
-`backfill` / `imprecise` / `no_match` / `error`.
-
-```bash
-DATABASE_URL=postgresql+psycopg://... \
-YANDEX_GEOCODER_API_KEY=... \
-uv run python -m scripts.backfill_house_coords \
-    --batch 2026-05-27_backfill
-```
-
-Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request):
-20-25 min. Expected output split (rough baseline from Phase 1 numbers):
-
-| Status      | Approx rows | What it means                                       |
-|-------------|-------------|-----------------------------------------------------|
-| `backfill`  | ~3.3k–3.7k  | UPDATE landed, lat/lon now populated                |
-| `imprecise` | ~300–500    | Match returned but precision too low — needs review |
-| `no_match`  | ~100–300    | Yandex couldn't resolve; address probably mangled   |
-| `error`     | <50         | HTTP errors / timeouts — re-run picks them up       |
-
-**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write
-audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`.
-Does NOT modify the `houses` table.
-
-```bash
-uv run python -m scripts.backfill_house_coords \
-    --batch 2026-05-27_audit --audit-only
-```
-
-Combined budget for both phases (~8.6k requests) is well under the 25k/day
-Geocoder free tier.
-
-### Common ops
-
-Canary first — run with `--limit 100` and inspect the audit table before
-letting the full job loose:
-
-```bash
-uv run python -m scripts.backfill_house_coords \
-    --batch canary_$(date +%F) --limit 100
-psql "$DATABASE_URL" -c "
-  SELECT audit_status, COUNT(*)
-    FROM address_mismatch_audit
-   WHERE audit_batch = 'canary_$(date +%F)'
-   GROUP BY audit_status;
-"
-```
-
-Resume after crash / quota hit — same `--batch` label, the UNIQUE
-`(house_id, audit_batch)` index skips finished rows:
-
-```bash
-uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
-# ... interruption ...
-uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
-# logs: "resuming batch 2026-05-27_backfill: N rows already processed"
-```
-
-### Helpers (not entry points)
-
-- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`,
-  `reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API
-  paths share `_parse_api_payload` because Yandex's forward/reverse
-  envelopes have the same shape.
-- `audit_address_sample.sql` — random sample for the Phase 1 audit (used
-  by `audit_address_mismatch.py`).
-- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95
-  distance, top-20 outliers, per-district breakdown).
+`audit_address_mismatch.py`, `backfill_house_coords.py`, `_yandex_reverse.py`
+и их SQL-хелперы (`audit_address_sample.sql`, `address_audit_report.sql`)
+удалены — весь pipeline опирался на Yandex Geocoder API, который выпилен
+из проекта (#2593, части 1-3). `houses.address`→lat/lon geocoding теперь
+идёт через `app/services/geocoder.py` (кадастр/геопортал ЕКБ-тиры + Nominatim
+fallback, единственный живой внешний провайдер) на обычном write-path
+(`/api/v1/trade-in/estimate`, listing ingest). Разовый forward-backfill
+недостающих `houses` координат — `scripts/geocode_deals_nominatim.py`
+(живой, работает с `rosreestr_deals`, не с `houses` — читай его docstring
+перед использованием на других таблицах). Таблица `address_mismatch_audit`
+осталась в схеме (используется `house_dedup_merge.py` при слиянии дублей
+домов, независимо от Yandex-аудита).
 
 ---
 

tradein-mvp/backend/scripts/_yandex_reverse.py

@@ -1,380 +0,0 @@
-"""Yandex Geocoder helpers for the address-mismatch audit + backfill (issue #582).
-
-Three geocoding paths exposed:
-
-- `reverse_via_api()` — Yandex Geocoder HTTP API, lon/lat → address. Fast,
-  structured response, needs a valid API key (env `YANDEX_GEOCODER_API_KEY`).
-  Free tier is 25k req/day, fine for ~8.5k houses + audit (~17k total).
-
-- `reverse_via_playwright()` — fallback when no API key is available. Drives
-  a real browser session at https://yandex.ru/maps/?…&mode=whatshere. Slower
-  and CAPTCHA-prone, so the driver inserts 4-7s sleeps between calls and we
-  raise a dedicated exception on CAPTCHA so the batch can pause-and-resume.
-
-- `forward_via_api()` — address → lon/lat + canonical address (Phase 2 of
-  issue #582). Used by `backfill_house_coords.py` to fill `houses.lat/lon`
-  for the 4141 houses scraped from sources that didn't include coords (esp.
-  yandex_valuation, which only returns an address string).
-
-All three return a `YandexReverseResult` dataclass — same shape regardless
-of direction so the driver code stays implementation-agnostic. The `raw`
-field always carries the full source payload for post-hoc diagnostics, and
-`precision` / `kind` are filled in by the API paths so the caller can skip
-imprecise matches (e.g. only-street-level results during backfill).
-
-Why three paths:
-The user (issue #582 discussion) wants the audit to run on dev machines
-that may not have an API key, but on prod we already provision the key for
-estimator.py. Forward geocode is API-only — Playwright forward geocoding
-through Yandex Maps search is too fragile (relevance ranking, suggest
-dropdown). For dev without a key, backfill simply doesn't run.
-"""
-
-from __future__ import annotations
-
-import asyncio
-import logging
-import random
-from dataclasses import dataclass, field
-from typing import Any
-
-import httpx
-
-logger = logging.getLogger(__name__)
-
-# Yandex Maps "what's here" URL — wraps a reverse-geocode in browser-driven UI.
-# `whatshere[point]` accepts "<lon>,<lat>" (note: lon first, Yandex convention).
-_YANDEX_MAPS_WHATSHERE = (
-    "https://yandex.ru/maps/?ll={lon:.6f}%2C{lat:.6f}&z=18&mode=whatshere"
-    "&whatshere%5Bpoint%5D={lon:.6f}%2C{lat:.6f}&whatshere%5Bzoom%5D=18"
-)
-
-# Geocoder HTTP API. `kind=house` narrows the result to a building if possible,
-# which is what we want for cadastr-style addresses (улица + дом).
-_YANDEX_GEOCODE_API = "https://geocode-maps.yandex.ru/1.x/"
-
-# Reasonable timeouts: API call should be sub-second; we give it generous
-# headroom for slow networks but not so much that a hang stalls the batch.
-_API_TIMEOUT = httpx.Timeout(connect=5.0, read=10.0, write=5.0, pool=5.0)
-
-
-# ---------------------------------------------------------------------------
-# Dataclasses + exceptions
-# ---------------------------------------------------------------------------
-
-
-@dataclass
-class YandexReverseResult:
-    """Normalized result of a geocode call (forward, reverse-API, or browser).
-
-    Attributes:
-        address: Human-readable canonical address Yandex returned. For
-            reverse, this is the snapped address at the queried point. For
-            forward, this is the canonical form of the input address. None
-            if Yandex returned no match.
-        snapped_lat: Latitude of the matched object's geometric centre.
-        snapped_lon: Longitude of the matched object's geometric centre.
-        precision: For forward calls — Yandex match precision tag (`exact`,
-            `number`, `near`, `range`, `street`, `other`). For reverse —
-            same field is filled when present (usually `house` / `street`).
-            None for the playwright path. Used by the backfill driver to
-            skip imprecise matches.
-        kind: Object kind from Yandex (`house`, `street`, `locality`, ...).
-            Same source as `precision` — see metaDataProperty.GeocoderMetaData.
-        raw: Raw response payload retained for forensics (JSON dict from API,
-            or snapshot dict from playwright). Used to populate
-            `address_mismatch_audit.raw_payload` and
-            `houses.raw_payload.yandex_geocode`.
-    """
-
-    address: str | None
-    snapped_lat: float | None
-    snapped_lon: float | None
-    raw: dict[str, Any] = field(default_factory=dict)
-    precision: str | None = None
-    kind: str | None = None
-
-
-class YandexBlockedError(RuntimeError):
-    """Raised when Yandex returns a CAPTCHA / anti-bot challenge.
-
-    The driver catches this, marks the row `audit_status='blocked'`, logs the
-    current batch position, then exits cleanly so a human can intervene.
-    """
-
-
-# ---------------------------------------------------------------------------
-# Path A — HTTP Geocoder API
-# ---------------------------------------------------------------------------
-
-
-async def reverse_via_api(
-    lat: float,
-    lon: float,
-    api_key: str,
-    *,
-    client: httpx.AsyncClient | None = None,
-) -> YandexReverseResult:
-    """Reverse-geocode (lat, lon) via the Yandex Geocoder HTTP API.
-
-    Why a separate `client` parameter: lets the driver reuse one
-    `AsyncClient` across all 200 calls (TCP keep-alive + connection pool),
-    and lets the tests inject a `MockTransport` to assert request shape.
-
-    Args:
-        lat: latitude in WGS84.
-        lon: longitude in WGS84.
-        api_key: Yandex Geocoder API key.
-        client: optional pre-built async client. If None, a one-shot client
-            is created.
-
-    Returns:
-        `YandexReverseResult` with the first `featureMember[0].GeoObject`
-        result, or all-None if Yandex returned no match (still includes
-        `raw` payload so we can later inspect why).
-    """
-    params = {
-        "apikey": api_key,
-        # Yandex expects "lon,lat" (longitude first) per docs — same
-        # convention as the "whatshere" map URL above.
-        "geocode": f"{lon},{lat}",
-        "format": "json",
-        "kind": "house",
-        "results": "1",
-    }
-
-    own_client = client is None
-    if client is None:
-        client = httpx.AsyncClient(timeout=_API_TIMEOUT)
-
-    try:
-        resp = await client.get(_YANDEX_GEOCODE_API, params=params)
-        resp.raise_for_status()
-        data = resp.json()
-    finally:
-        if own_client:
-            await client.aclose()
-
-    return _parse_api_payload(data)
-
-
-def _parse_api_payload(data: dict[str, Any]) -> YandexReverseResult:
-    """Extract address + snapped point from a Yandex Geocoder API JSON response.
-
-    Split out so unit tests can feed a fixture file directly without spinning
-    up an HTTP mock. Same payload shape for forward and reverse calls —
-    Yandex's response envelope is symmetric.
-    """
-    try:
-        members = data.get("response", {}).get("GeoObjectCollection", {}).get("featureMember", [])
-        if not members:
-            return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data)
-
-        geo_obj = members[0].get("GeoObject", {})
-
-        # Address: prefer the long `metaDataProperty.GeocoderMetaData.text`
-        # (full canonical) and fall back to `name` (street + house number).
-        meta = geo_obj.get("metaDataProperty", {}).get("GeocoderMetaData", {})
-        address = meta.get("text") or geo_obj.get("name")
-        precision = meta.get("precision")
-        kind = meta.get("kind")
-
-        # Point format: "<lon> <lat>" — space-separated string.
-        point_str = geo_obj.get("Point", {}).get("pos", "")
-        snapped_lon: float | None
-        snapped_lat: float | None
-        if point_str:
-            try:
-                lon_s, lat_s = point_str.split()
-                snapped_lon = float(lon_s)
-                snapped_lat = float(lat_s)
-            except (ValueError, TypeError):
-                snapped_lon = None
-                snapped_lat = None
-        else:
-            snapped_lon = None
-            snapped_lat = None
-
-        return YandexReverseResult(
-            address=address,
-            snapped_lat=snapped_lat,
-            snapped_lon=snapped_lon,
-            raw=data,
-            precision=precision,
-            kind=kind,
-        )
-    except Exception as e:  # pragma: no cover — defensive; tests cover happy paths
-        logger.warning("yandex API payload parse failed: %s", e)
-        return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data)
-
-
-# ---------------------------------------------------------------------------
-# Path A.2 — Forward geocode (address → lon/lat) via HTTP API
-# ---------------------------------------------------------------------------
-
-
-async def forward_via_api(
-    address: str,
-    api_key: str,
-    *,
-    client: httpx.AsyncClient | None = None,
-) -> YandexReverseResult:
-    """Forward-geocode an address string via the Yandex Geocoder HTTP API.
-
-    Phase 2 of issue #582 — used by `backfill_house_coords.py` to populate
-    `houses.lat/lon` for houses that were scraped without coords (esp.
-    yandex_valuation rows, which only carry an address).
-
-    Args:
-        address: free-form address ("ул Малышева 51", "Екатеринбург, Ленина 5",
-            etc.). Yandex's NLU is forgiving — no need to pre-normalize.
-        api_key: Yandex Geocoder API key.
-        client: optional pre-built async client. If None, a one-shot client
-            is created (matches `reverse_via_api` ergonomics).
-
-    Returns:
-        `YandexReverseResult` with the canonical address + snapped point of
-        the first matching feature. `precision` and `kind` are populated so
-        the backfill driver can skip imprecise hits (e.g. precision='street'
-        means we landed on the road, not the building — too vague for
-        comparable-listings spatial queries).
-
-    Same envelope as `reverse_via_api` — `_parse_api_payload` handles both.
-    """
-    params = {
-        "apikey": api_key,
-        "geocode": address,
-        "format": "json",
-        # `kind=house` filters out street-only / locality-only matches at
-        # the API level when possible. Yandex still returns lower-precision
-        # results when no building matches, so the caller must double-check
-        # `precision` before writing to houses.
-        "kind": "house",
-        "results": "1",
-        # Locality bias for EKB — improves recall when the input address
-        # omits the city. The audit population is 99% EKB houses, so this
-        # is safe; non-EKB inputs (rare) still resolve, just with the bias.
-        "ll": "60.6122,56.8389",
-        "spn": "0.6,0.4",
-    }
-
-    own_client = client is None
-    if client is None:
-        client = httpx.AsyncClient(timeout=_API_TIMEOUT)
-
-    try:
-        resp = await client.get(_YANDEX_GEOCODE_API, params=params)
-        resp.raise_for_status()
-        data = resp.json()
-    finally:
-        if own_client:
-            await client.aclose()
-
-    return _parse_api_payload(data)
-
-
-# ---------------------------------------------------------------------------
-# Path B — Playwright fallback
-# ---------------------------------------------------------------------------
-
-
-async def reverse_via_playwright(
-    lat: float,
-    lon: float,
-    page: Any,
-) -> YandexReverseResult:
-    """Reverse-geocode (lat, lon) by driving yandex.ru/maps with Playwright.
-
-    Why this exists:
-    The Yandex Geocoder API requires a key with paid quota for >25k/day. The
-    audit only needs 200 rows but a dev without a key still needs a way to
-    run the script, so we ship a browser-driven fallback.
-
-    Implementation:
-    1. Navigate to the `whatshere` URL — Yandex Maps responds by opening a
-       toponym card at the requested coordinates and rendering the resolved
-       address in the side panel.
-    2. Wait for client hydration (`networkidle`).
-    3. First try to read `window.__INITIAL_STATE__` — Yandex stores the
-       toponym address inside the hydrated Redux tree, which is more
-       stable across UI redesigns than DOM selectors.
-    4. Fall back to DOM selectors (`.toponym-card-title-view__title` +
-       `__subtitle`) if the state walk doesn't find an address.
-    5. Detect CAPTCHA (`.CheckboxCaptcha`) early and raise `YandexBlockedError`
-       so the batch can pause-and-resume without spamming Yandex.
-
-    `page` is typed as `Any` to keep playwright a dev-only dep — runtime
-    importers don't need playwright installed if they only use the API path.
-    """
-    url = _YANDEX_MAPS_WHATSHERE.format(lat=lat, lon=lon)
-    await page.goto(url, wait_until="domcontentloaded")
-
-    # Light wait for client-side hydration. Yandex Maps fires lots of
-    # background XHRs so `networkidle` is too aggressive; this small wait is
-    # enough for the toponym card to render.
-    try:
-        await page.wait_for_load_state("networkidle", timeout=8000)
-    except Exception as e:
-        # Slow networks: continue — selectors will retry with their own waits.
-        logger.debug("networkidle wait timed out, continuing: %s", e)
-    await asyncio.sleep(random.uniform(0.5, 1.2))
-
-    # CAPTCHA gate — Yandex shows a `.CheckboxCaptcha` form when it suspects
-    # automation. Once we see it, every subsequent reverse call will also be
-    # blocked, so we raise immediately and let the driver stop the batch.
-    captcha = await page.query_selector(".CheckboxCaptcha")
-    if captcha is not None:
-        raise YandexBlockedError("Yandex CAPTCHA detected on maps page")
-
-    # Attempt 1 — initial state walk.
-    state_addr: str | None = None
-    state_pos: tuple[float, float] | None = None
-    try:
-        state_addr, state_pos = await page.evaluate(
-            "() => {\n"
-            "  const s = window.__INITIAL_STATE__ || {};\n"
-            "  const card = (s.cards && s.cards.toponym) || (s.card && s.card.toponym) || null;\n"
-            "  if (!card) return [null, null];\n"
-            "  const addr = card.title || card.address || null;\n"
-            "  const pos = card.coords || card.point || null;\n"
-            "  if (pos && pos.length === 2) return [addr, [pos[0], pos[1]]];\n"
-            "  return [addr, null];\n"
-            "}"
-        )
-    except Exception as e:
-        logger.debug("playwright state walk failed (will fall back to DOM): %s", e)
-
-    address = state_addr
-
-    # Attempt 2 — DOM fallback.
-    if not address:
-        title_el = await page.query_selector(".toponym-card-title-view__title")
-        subtitle_el = await page.query_selector(".toponym-card-title-view__subtitle")
-        title = (await title_el.inner_text()).strip() if title_el else ""
-        subtitle = (await subtitle_el.inner_text()).strip() if subtitle_el else ""
-        # subtitle often holds "Екатеринбург, район", title the street + house
-        address = ", ".join([p for p in (subtitle, title) if p]) or None
-
-    snapped_lat: float | None
-    snapped_lon: float | None
-    if state_pos:
-        # State stored as [lon, lat] in Yandex's coordinate convention.
-        snapped_lon = float(state_pos[0])
-        snapped_lat = float(state_pos[1])
-    else:
-        snapped_lon = None
-        snapped_lat = None
-
-    raw = {
-        "url": url,
-        "state_addr": state_addr,
-        "state_pos": list(state_pos) if state_pos else None,
-        "dom_address": address if not state_addr else None,
-    }
-
-    return YandexReverseResult(
-        address=address,
-        snapped_lat=snapped_lat,
-        snapped_lon=snapped_lon,
-        raw=raw,
-    )

tradein-mvp/backend/scripts/address_audit_report.sql

@@ -1,91 +0,0 @@
--- address_audit_report.sql
--- Post-run report for the address-mismatch audit (issue #582 Phase 1).
---
--- Sections:
---   1. Summary — count, p50/p75/p95/mean distance, % street_differs,
---      % over 50m / 200m thresholds.
---   2. Top-20 outliers by distance (manual triage list).
---   3. Per-district breakdown — same metrics grouped by district column.
---
--- Run via psql:
---   psql "$DATABASE_URL" -v batch='2026-05-25_run1' -f scripts/address_audit_report.sql
---
--- :batch is a psql client variable substituted via -v.
-
-\set ON_ERROR_STOP on
-
-\echo '=============================================='
-\echo ' Address mismatch audit — batch:' :batch
-\echo '=============================================='
-
--- ---------------------------------------------------------------------------
--- 1) Top-level summary
--- ---------------------------------------------------------------------------
-\echo ''
-\echo '--- Summary (status=ok rows only) ---'
-SELECT
-    COUNT(*)                                                      AS n_total,
-    COUNT(*) FILTER (WHERE audit_status = 'ok')                   AS n_ok,
-    COUNT(*) FILTER (WHERE audit_status = 'no_match')             AS n_no_match,
-    COUNT(*) FILTER (WHERE audit_status = 'error')                AS n_error,
-    COUNT(*) FILTER (WHERE audit_status = 'blocked')              AS n_blocked,
-    ROUND(percentile_cont(0.50)
-          WITHIN GROUP (ORDER BY distance_m)::numeric, 1)         AS p50_distance_m,
-    ROUND(percentile_cont(0.75)
-          WITHIN GROUP (ORDER BY distance_m)::numeric, 1)         AS p75_distance_m,
-    ROUND(percentile_cont(0.95)
-          WITHIN GROUP (ORDER BY distance_m)::numeric, 1)         AS p95_distance_m,
-    ROUND(AVG(distance_m)::numeric, 1)                            AS mean_distance_m,
-    ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1)
-                                                                   AS pct_street_differs,
-    ROUND(100.0 * AVG(CASE WHEN distance_m > 50  THEN 1.0 ELSE 0.0 END), 1)
-                                                                   AS pct_over_50m,
-    ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1)
-                                                                   AS pct_over_200m
-FROM address_mismatch_audit
-WHERE audit_batch = :'batch'
-  AND audit_status = 'ok';
-
--- ---------------------------------------------------------------------------
--- 2) Top-20 outliers
--- ---------------------------------------------------------------------------
-\echo ''
-\echo '--- Top-20 outliers by distance ---'
-SELECT
-    house_id,
-    district,
-    ROUND(distance_m::numeric, 1)   AS distance_m,
-    street_differs,
-    LEFT(original_address, 60)      AS original_address,
-    LEFT(snapped_address, 60)       AS snapped_address
-FROM address_mismatch_audit
-WHERE audit_batch = :'batch'
-  AND audit_status = 'ok'
-  AND distance_m IS NOT NULL
-ORDER BY distance_m DESC NULLS LAST
-LIMIT 20;
-
--- ---------------------------------------------------------------------------
--- 3) Per-district breakdown
--- ---------------------------------------------------------------------------
-\echo ''
-\echo '--- Per-district breakdown (status=ok only) ---'
-SELECT
-    COALESCE(district, '(no district)')                            AS district,
-    COUNT(*)                                                       AS n,
-    ROUND(percentile_cont(0.50)
-          WITHIN GROUP (ORDER BY distance_m)::numeric, 1)          AS p50_distance_m,
-    ROUND(percentile_cont(0.95)
-          WITHIN GROUP (ORDER BY distance_m)::numeric, 1)          AS p95_distance_m,
-    ROUND(AVG(distance_m)::numeric, 1)                             AS mean_distance_m,
-    ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1)
-                                                                    AS pct_street_differs,
-    ROUND(100.0 * AVG(CASE WHEN distance_m > 50  THEN 1.0 ELSE 0.0 END), 1)
-                                                                    AS pct_over_50m,
-    ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1)
-                                                                    AS pct_over_200m
-FROM address_mismatch_audit
-WHERE audit_batch = :'batch'
-  AND audit_status = 'ok'
-GROUP BY COALESCE(district, '(no district)')
-ORDER BY n DESC, district;

tradein-mvp/backend/scripts/audit_address_mismatch.py

@@ -1,595 +0,0 @@
-"""Audit driver — compares houses.address vs Yandex reverse geocode.
-
-Phase 1 of Forgejo issue #582. Pulls a stratified sample of EKB houses (25
-per admin district = 200 total), reverse-geocodes each via Yandex, computes
-the distance between the stored coordinates and the snapped Yandex point,
-and writes the result into `address_mismatch_audit`.
-
-Design choices:
-- **Resumable**: the audit table has UNIQUE (house_id, audit_batch). Re-run
-  with the same `--batch` skips rows already inserted, so a partial run can
-  be picked up after CAPTCHA / network blip.
-- **Mode auto**: prefer API when `YANDEX_GEOCODER_API_KEY` is set, fall back
-  to Playwright otherwise. Explicit override via `--mode {api,playwright}`.
-- **No prod side effects**: the script only writes to one new audit table;
-  it never touches `houses`, `house_sources`, or any matching/listing row.
-- **Per-row SAVEPOINT**: a single Yandex error must not nuke the entire
-  batch — wrap each INSERT in `db.begin_nested()` per backend.md.
-
-How to run:
-    DATABASE_URL=postgresql+psycopg://... \
-    YANDEX_GEOCODER_API_KEY=... \
-    python -m scripts.audit_address_mismatch --batch 2026-05-25_run1
-
-Outputs (post-run):
-- New rows in `address_mismatch_audit` with batch label.
-- `scripts/address_audit_report.sql :batch=<id>` for summary.
-"""
-
-from __future__ import annotations
-
-import argparse
-import asyncio
-import json
-import logging
-import os
-import random
-from dataclasses import dataclass
-from datetime import date
-from pathlib import Path
-from typing import Any
-
-import httpx
-from sqlalchemy import text
-from sqlalchemy.orm import Session
-
-# Allow running both as `python -m scripts.audit_address_mismatch` (preferred)
-# and as a stand-alone file (`python scripts/audit_address_mismatch.py`)
-# without requiring package install.
-try:
-    from app.core.db import SessionLocal  # type: ignore[import-not-found]
-    from app.services.matching.normalize import normalize_address  # type: ignore[import-not-found]
-except ImportError:  # pragma: no cover — fallback for adhoc invocation
-    import sys
-
-    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-    from app.core.db import SessionLocal
-    from app.services.matching.normalize import normalize_address
-
-# `from .` works when run via -m; the absolute import works under pytest.
-try:
-    from scripts._yandex_reverse import (  # type: ignore[import-not-found]
-        YandexBlockedError,
-        YandexReverseResult,
-        reverse_via_api,
-        reverse_via_playwright,
-    )
-except ImportError:
-    from _yandex_reverse import (  # type: ignore[no-redef]
-        YandexBlockedError,
-        YandexReverseResult,
-        reverse_via_api,
-        reverse_via_playwright,
-    )
-
-logging.basicConfig(
-    level=logging.INFO,
-    format="%(asctime)s %(levelname)s %(name)s %(message)s",
-)
-logger = logging.getLogger("audit_address_mismatch")
-
-# Playwright persistent context location — keeps cookies/local storage between
-# runs so we look like a returning user, reducing CAPTCHA frequency.
-_PLAYWRIGHT_USER_DATA = Path.home() / ".cache" / "tradein-audit-playwright"
-
-_PLAYWRIGHT_UA = (
-    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
-    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
-)
-
-_SAMPLE_SQL_PATH = Path(__file__).parent / "audit_address_sample.sql"
-
-
-# ---------------------------------------------------------------------------
-# Domain helpers
-# ---------------------------------------------------------------------------
-
-
-@dataclass
-class SampleRow:
-    """One house from the stratified sampling query."""
-
-    id: int
-    address: str
-    lat: float
-    lon: float
-    district: str | None
-
-
-# Words that introduce a street rather than identify it. We skip these so the
-# comparison lands on the actual street name ('малышева' / 'ленина'). Mirrors
-# the canonical forms produced by `normalize_address` (which expands all known
-# abbreviations to these full words).
-_STREET_TYPE_WORDS = frozenset(
-    {
-        "улица",
-        "проспект",
-        "переулок",
-        "бульвар",
-        "проезд",
-        "шоссе",
-        "площадь",
-        "набережная",
-        "тупик",
-        "строение",
-        "корпус",
-        "дом",
-    }
-)
-
-# Geographic prefix words that addresses sometimes carry before the street
-# (e.g. 'россия екатеринбург улица малышева 51'). We skip them too so we
-# converge on the same identifying token regardless of how verbose the
-# source representation is.
-_GEO_PREFIX_WORDS = frozenset(
-    {
-        "россия",
-        "свердловская",
-        "область",
-        "екатеринбург",
-        "город",
-        "г",
-    }
-)
-
-
-def _first_street_token(address: str | None) -> str | None:
-    """Extract the first street-name token of a normalized address.
-
-    Phase-1 heuristic for "do the streets agree": skip numeric tokens (house
-    numbers), street-type words ('улица', 'проспект', …), and geographic
-    prefixes ('россия', 'екатеринбург', …) — the next token is the street
-    name itself, which is the identifying part we want to compare.
-
-    Returns None for an empty address or when no candidate token remains.
-    """
-    norm = normalize_address(address or "")
-    if not norm:
-        return None
-    for tok in norm.split():
-        if not tok:
-            continue
-        # Skip purely numeric tokens (e.g. '5', '17а' if it starts with digit).
-        if tok[0].isdigit():
-            continue
-        # Skip street type words and geographic prefixes.
-        if tok in _STREET_TYPE_WORDS or tok in _GEO_PREFIX_WORDS:
-            continue
-        return tok
-    return None
-
-
-def _street_differs(original: str | None, snapped: str | None) -> bool | None:
-    """True iff first non-numeric token differs between the two addresses.
-
-    Returns None when either side is empty — we cannot compute a meaningful
-    diff (caller writes NULL into the audit row).
-    """
-    a = _first_street_token(original)
-    b = _first_street_token(snapped)
-    if a is None or b is None:
-        return None
-    return a != b
-
-
-def _distance_meters(
-    db: Session,
-    olat: float,
-    olon: float,
-    slat: float,
-    slon: float,
-) -> float | None:
-    """Compute great-circle distance via PostGIS geography type.
-
-    We could do this in Python with a haversine formula, but the audit table
-    uses ST_Distance results elsewhere so we use the same authority to avoid
-    drift. ST_MakePoint(lon, lat) — PostGIS convention is lon first.
-    """
-    row = db.execute(
-        text(
-            "SELECT ST_Distance("
-            "  ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), "
-            "                          CAST(:olat AS double precision)), 4326)::geography, "
-            "  ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), "
-            "                          CAST(:slat AS double precision)), 4326)::geography"
-            ") AS m"
-        ),
-        {"olat": olat, "olon": olon, "slat": slat, "slon": slon},
-    ).first()
-    if row is None or row[0] is None:
-        return None
-    return float(row[0])
-
-
-# ---------------------------------------------------------------------------
-# Sampling + resumption queries
-# ---------------------------------------------------------------------------
-
-
-def _load_sample(db: Session, limit_per_district: int) -> list[SampleRow]:
-    """Run the stratified sampling SQL → list of SampleRow."""
-    sql = _SAMPLE_SQL_PATH.read_text(encoding="utf-8")
-    rows = db.execute(text(sql), {"limit_per_district": limit_per_district}).mappings().all()
-    return [
-        SampleRow(
-            id=r["id"],
-            address=r["address"],
-            lat=float(r["lat"]),
-            lon=float(r["lon"]),
-            district=r["district"],
-        )
-        for r in rows
-    ]
-
-
-def _already_processed_ids(db: Session, batch: str) -> set[int]:
-    """Return the set of house_id already in the audit table for this batch.
-
-    Drives resumability: drop these from the sample before geocoding.
-    """
-    rows = db.execute(
-        text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"),
-        {"b": batch},
-    ).all()
-    return {r[0] for r in rows}
-
-
-# ---------------------------------------------------------------------------
-# Insert helper
-# ---------------------------------------------------------------------------
-
-
-def _insert_audit_row(
-    db: Session,
-    *,
-    house_id: int,
-    batch: str,
-    district: str | None,
-    original_address: str | None,
-    original_lat: float | None,
-    original_lon: float | None,
-    snapped_address: str | None,
-    snapped_lat: float | None,
-    snapped_lon: float | None,
-    distance_m: float | None,
-    street_differs: bool | None,
-    audit_status: str,
-    error_message: str | None,
-    raw_payload: dict[str, Any] | None,
-) -> None:
-    """INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit.
-
-    Wrapped in begin_nested by the caller per backend.md SAVEPOINT pattern.
-    """
-    db.execute(
-        text(
-            "INSERT INTO address_mismatch_audit ("
-            "  house_id, audit_batch, district,"
-            "  original_address, original_lat, original_lon,"
-            "  snapped_address, snapped_lat, snapped_lon,"
-            "  distance_m, street_differs,"
-            "  audit_status, error_message, raw_payload"
-            ") VALUES ("
-            "  CAST(:house_id AS bigint), CAST(:batch AS text), :district,"
-            "  :original_address, :original_lat, :original_lon,"
-            "  :snapped_address, :snapped_lat, :snapped_lon,"
-            "  :distance_m, :street_differs,"
-            "  CAST(:audit_status AS text), :error_message,"
-            "  CAST(:raw_payload AS jsonb)"
-            ") ON CONFLICT (house_id, audit_batch) DO NOTHING"
-        ),
-        {
-            "house_id": house_id,
-            "batch": batch,
-            "district": district,
-            "original_address": original_address,
-            "original_lat": original_lat,
-            "original_lon": original_lon,
-            "snapped_address": snapped_address,
-            "snapped_lat": snapped_lat,
-            "snapped_lon": snapped_lon,
-            "distance_m": distance_m,
-            "street_differs": street_differs,
-            "audit_status": audit_status,
-            "error_message": error_message,
-            "raw_payload": json.dumps(raw_payload) if raw_payload is not None else None,
-        },
-    )
-
-
-# ---------------------------------------------------------------------------
-# Mode dispatcher
-# ---------------------------------------------------------------------------
-
-
-def _resolve_mode(mode: str, api_key: str | None) -> str:
-    """Translate `--mode auto` → concrete 'api' / 'playwright' choice.
-
-    Explicit modes are passed through unchanged; auto chooses api iff a key
-    is configured (fail-fast: we don't want a "should have used the API but
-    silently fell back to slow scraping" surprise).
-    """
-    if mode == "auto":
-        return "api" if api_key else "playwright"
-    return mode
-
-
-# ---------------------------------------------------------------------------
-# Main loop
-# ---------------------------------------------------------------------------
-
-
-async def _run_api_mode(
-    db: Session,
-    sample: list[SampleRow],
-    batch: str,
-    api_key: str,
-) -> int:
-    """Geocode the sample using the HTTP Geocoder API."""
-    processed = 0
-    last_distance: float | None = None
-    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
-        for i, row in enumerate(sample, start=1):
-            status = "ok"
-            err: str | None = None
-            res: YandexReverseResult | None = None
-            try:
-                res = await reverse_via_api(row.lat, row.lon, api_key, client=client)
-            except httpx.HTTPError as e:
-                status = "error"
-                err = f"http_error: {e!s}"
-            except Exception as e:  # pragma: no cover — defensive
-                status = "error"
-                err = f"unhandled: {e!s}"
-
-            distance = None
-            street_diff: bool | None = None
-            if res is not None and status == "ok":
-                if res.address is None:
-                    status = "no_match"
-                else:
-                    if res.snapped_lat is not None and res.snapped_lon is not None:
-                        distance = _distance_meters(
-                            db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
-                        )
-                        last_distance = distance
-                    street_diff = _street_differs(row.address, res.address)
-
-            try:
-                with db.begin_nested():
-                    _insert_audit_row(
-                        db,
-                        house_id=row.id,
-                        batch=batch,
-                        district=row.district,
-                        original_address=row.address,
-                        original_lat=row.lat,
-                        original_lon=row.lon,
-                        snapped_address=res.address if res else None,
-                        snapped_lat=res.snapped_lat if res else None,
-                        snapped_lon=res.snapped_lon if res else None,
-                        distance_m=distance,
-                        street_differs=street_diff,
-                        audit_status=status,
-                        error_message=err,
-                        raw_payload=res.raw if res else None,
-                    )
-                # Per-row commit: each row is durable on disk before the next
-                # Yandex call; --batch resume picks up exactly where we crashed.
-                db.commit()
-                processed += 1
-            except Exception as e:
-                db.rollback()
-                logger.warning("insert failed for house_id=%s: %s", row.id, e)
-
-            if i % 10 == 0:
-                logger.info(
-                    "progress %d/%d, mode=api, last_distance=%s",
-                    i,
-                    len(sample),
-                    f"{last_distance:.1f}m" if last_distance is not None else "n/a",
-                )
-
-    return processed
-
-
-async def _run_playwright_mode(
-    db: Session,
-    sample: list[SampleRow],
-    batch: str,
-) -> int:
-    """Geocode via a persistent Playwright context (CAPTCHA-aware)."""
-    try:
-        from playwright.async_api import async_playwright  # type: ignore[import-not-found]
-    except ImportError as e:
-        raise RuntimeError(
-            "Playwright is required for --mode playwright. "
-            "Install with `uv sync --group dev` and `playwright install chromium`."
-        ) from e
-
-    _PLAYWRIGHT_USER_DATA.mkdir(parents=True, exist_ok=True)
-    processed = 0
-    last_distance: float | None = None
-
-    async with async_playwright() as p:
-        context = await p.chromium.launch_persistent_context(
-            user_data_dir=str(_PLAYWRIGHT_USER_DATA),
-            headless=False,
-            user_agent=_PLAYWRIGHT_UA,
-            locale="ru-RU",
-            timezone_id="Asia/Yekaterinburg",
-        )
-        page = await context.new_page()
-
-        try:
-            for i, row in enumerate(sample, start=1):
-                status = "ok"
-                err: str | None = None
-                res: YandexReverseResult | None = None
-                stop_batch = False
-                try:
-                    res = await reverse_via_playwright(row.lat, row.lon, page)
-                except YandexBlockedError as e:
-                    status = "blocked"
-                    err = str(e)
-                    stop_batch = True
-                except Exception as e:
-                    status = "error"
-                    err = f"playwright: {e!s}"
-
-                distance = None
-                street_diff: bool | None = None
-                if res is not None and status == "ok":
-                    if res.address is None:
-                        status = "no_match"
-                    else:
-                        if res.snapped_lat is not None and res.snapped_lon is not None:
-                            distance = _distance_meters(
-                                db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
-                            )
-                            last_distance = distance
-                        street_diff = _street_differs(row.address, res.address)
-
-                try:
-                    with db.begin_nested():
-                        _insert_audit_row(
-                            db,
-                            house_id=row.id,
-                            batch=batch,
-                            district=row.district,
-                            original_address=row.address,
-                            original_lat=row.lat,
-                            original_lon=row.lon,
-                            snapped_address=res.address if res else None,
-                            snapped_lat=res.snapped_lat if res else None,
-                            snapped_lon=res.snapped_lon if res else None,
-                            distance_m=distance,
-                            street_differs=street_diff,
-                            audit_status=status,
-                            error_message=err,
-                            raw_payload=res.raw if res else None,
-                        )
-                    db.commit()
-                    processed += 1
-                except Exception as e:
-                    db.rollback()
-                    logger.warning("insert failed for house_id=%s: %s", row.id, e)
-
-                if stop_batch:
-                    logger.error(
-                        "Yandex CAPTCHA detected at position %d/%d (house_id=%s). "
-                        "Stopping batch — re-run with same --batch to resume.",
-                        i,
-                        len(sample),
-                        row.id,
-                    )
-                    break
-
-                if i % 10 == 0:
-                    logger.info(
-                        "progress %d/%d, mode=playwright, last_distance=%s",
-                        i,
-                        len(sample),
-                        f"{last_distance:.1f}m" if last_distance is not None else "n/a",
-                    )
-
-                # Random delay 4-7s between requests — keeps us under Yandex's
-                # heuristic rate limit while still finishing 200 rows in <30min.
-                # Skip the wait on the last iteration (no next request to space).
-                if i < len(sample):
-                    await asyncio.sleep(random.uniform(4.0, 7.0))
-        finally:
-            await context.close()
-
-    return processed
-
-
-# ---------------------------------------------------------------------------
-# Entry point
-# ---------------------------------------------------------------------------
-
-
-def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
-    """argparse setup, factored out for testability."""
-    p = argparse.ArgumentParser(
-        description="Phase 1 audit — houses.address vs Yandex reverse geocode.",
-    )
-    p.add_argument(
-        "--batch",
-        default=f"{date.today().isoformat()}_run1",
-        help="Audit batch label. Same batch re-run skips already-processed houses.",
-    )
-    p.add_argument(
-        "--limit-per-district",
-        type=int,
-        default=25,
-        help="Houses to sample per district (default 25 → ~200 total for EKB).",
-    )
-    p.add_argument(
-        "--mode",
-        choices=("auto", "api", "playwright"),
-        default="auto",
-        help="auto = API if YANDEX_GEOCODER_API_KEY set, else playwright.",
-    )
-    return p.parse_args(argv)
-
-
-async def main(argv: list[str] | None = None) -> int:
-    """CLI entry point. Returns the number of rows processed this run."""
-    args = _parse_args(argv)
-    api_key = os.environ.get("YANDEX_GEOCODER_API_KEY")
-    mode = _resolve_mode(args.mode, api_key)
-
-    if mode == "api" and not api_key:
-        raise SystemExit("mode=api requested but YANDEX_GEOCODER_API_KEY is not set")
-
-    logger.info(
-        "starting audit batch=%s mode=%s limit_per_district=%d",
-        args.batch,
-        mode,
-        args.limit_per_district,
-    )
-
-    db = SessionLocal()
-    try:
-        sample = _load_sample(db, args.limit_per_district)
-        logger.info("loaded sample: %d houses", len(sample))
-
-        # Resume support — drop already-processed house_ids.
-        done = _already_processed_ids(db, args.batch)
-        if done:
-            logger.info(
-                "resuming batch %s: %d rows already processed, %d remaining",
-                args.batch,
-                len(done),
-                len(sample) - sum(1 for s in sample if s.id in done),
-            )
-        remaining = [s for s in sample if s.id not in done]
-
-        if not remaining:
-            logger.info("nothing to do — batch %s is complete", args.batch)
-            return 0
-
-        if mode == "api":
-            n = await _run_api_mode(db, remaining, args.batch, api_key or "")
-        else:
-            n = await _run_playwright_mode(db, remaining, args.batch)
-
-        logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode)
-        return n
-    finally:
-        db.close()
-
-
-if __name__ == "__main__":  # pragma: no cover
-    asyncio.run(main())

tradein-mvp/backend/scripts/audit_address_sample.sql

@@ -1,47 +0,0 @@
--- audit_address_sample.sql
--- Random sample of EKB houses for the address-mismatch audit (issue #582).
---
--- Strategy:
---   1. Filter to houses with non-null lat/lon and non-empty address.
---   2. Random shuffle via `ORDER BY random()` — repeatable enough for spot
---      sampling without needing a stable PRNG seed (the audit table dedupes
---      via UNIQUE (house_id, audit_batch), so re-running gives idempotent
---      results regardless of which rows land in the sample first).
---   3. Cap the result at :limit_per_district * 8 rows — keeps the bind-param
---      contract compatible with the old stratified sampler (`:limit_per_district`
---      is still honored, just multiplied by the assumed 8-district count).
---
--- Why no spatial stratification anymore:
---   The previous version JOINed to `gendesign_ekb_districts_geom` (FDW
---   polygon table) to bucket houses by admin district. That join is fine on
---   prod where FDW is wired, but it adds a dependency we don't need for
---   Phase 2-3 (backfill + canonical reverse). Aggregation by district at
---   report time still works — we re-derive district during the audit via
---   spatial containment in the report SQL when needed.
---
--- Bind param:
---   :limit_per_district — kept for back-compat with the audit driver.
---     Effective sample size = :limit_per_district * 8 (e.g. 25 → 200).
---
--- Columns returned:
---   id, address, lat, lon, district
---   `district` is always NULL here — the audit driver will reverse-derive it
---   from Yandex Geocoder response (Yandex returns admin component) or leave
---   it NULL if not present in the response.
---
--- NB: uses CAST(:x AS int) per project sql.md rule (psycopg v3 ignores ::type
--- after bind params).
-
-SELECT
-    h.id,
-    h.address,
-    h.lat,
-    h.lon,
-    NULL::text AS district
-FROM houses h
-WHERE h.lat IS NOT NULL
-  AND h.lon IS NOT NULL
-  AND h.address IS NOT NULL
-  AND length(trim(h.address)) > 0
-ORDER BY random()
-LIMIT CAST(:limit_per_district AS int) * 8;

tradein-mvp/backend/scripts/backfill_house_coords.py

@@ -1,619 +0,0 @@
-"""Forward-geocode houses through Yandex Geocoder API to backfill lat/lon
-and canonical address, plus optional reverse audit of already-geocoded houses.
-
-Phase 2-3 of Forgejo issue #582. Two modes (mutually exclusive):
-
-  1. Backfill (default) — for the ~4141 rows WHERE lat IS NULL OR lon IS NULL:
-     forward-geocode `houses.address` → snap to a Yandex `house`-precision
-     point, UPDATE houses with the new lat/lon + canonical address payload,
-     and write an `address_mismatch_audit` row with `audit_status='backfill'`.
-
-  2. Audit-only (--audit-only) — for the ~4452 rows that already have coords:
-     reverse-geocode (lat, lon) → snapped point + canonical address, compute
-     ST_Distance vs stored coords, write an `address_mismatch_audit` row with
-     status 'ok' (≤50m) or 'mismatch' (>50m). Does NOT touch houses.
-
-Design choices:
-- **Per-row SAVEPOINT** (`db.begin_nested()`): a single Yandex/PostGIS error
-  must not nuke the entire batch. Per backend.md, never use bare rollback
-  inside a loop.
-- **Resumable** via UNIQUE (house_id, audit_batch). Re-running the same
-  --batch label skips already-processed houses, so a partial run can be
-  picked up after CAPTCHA / network blip / 25k/day quota hit.
-- **Precision filter**: backfill skips matches with precision in
-  ('street', 'other', 'range', 'near', None) — those are too imprecise for
-  comparable-listing spatial queries and would silently degrade matching
-  recall. The audit row still records what Yandex returned for forensics.
-- **Rate limit**: 50ms between calls (~20 req/sec, well under Yandex's
-  25 req/sec service limit). Backfill mode runs single-threaded.
-- **Daily quota**: 4141 backfill + 4452 audit ≈ 8.6k requests. Free Geocoder
-  tier is 25k/day → comfortable buffer for retries.
-
-Usage:
-    YANDEX_GEOCODER_API_KEY=xxx \\
-    DATABASE_URL=postgresql+psycopg://... \\
-    python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
-
-    # Audit-only on the 4452 already-geocoded houses
-    python -m scripts.backfill_house_coords --batch 2026-05-27_audit \\
-        --audit-only --limit 500
-
-Outputs:
-- Backfill mode: UPDATE rows in `houses`, INSERT rows in
-  `address_mismatch_audit` with status 'backfill' / 'no_match' / 'imprecise'.
-- Audit mode: INSERT rows in `address_mismatch_audit` with status 'ok' /
-  'mismatch' / 'no_match' / 'error'.
-- Per-batch progress is logged every 25 rows.
-"""
-
-from __future__ import annotations
-
-import argparse
-import asyncio
-import json
-import logging
-import os
-from dataclasses import dataclass
-from datetime import date
-from pathlib import Path
-from typing import Any
-
-import httpx
-from sqlalchemy import text
-from sqlalchemy.orm import Session
-
-# Allow running both as `python -m scripts.backfill_house_coords` (preferred)
-# and as a stand-alone file. Mirrors the audit_address_mismatch import dance.
-try:
-    from app.core.db import SessionLocal  # type: ignore[import-not-found]
-except ImportError:  # pragma: no cover — fallback for adhoc invocation
-    import sys
-
-    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-    from app.core.db import SessionLocal
-
-try:
-    from scripts._yandex_reverse import (  # type: ignore[import-not-found]
-        YandexReverseResult,
-        forward_via_api,
-        reverse_via_api,
-    )
-except ImportError:
-    from _yandex_reverse import (  # type: ignore[no-redef]
-        YandexReverseResult,
-        forward_via_api,
-        reverse_via_api,
-    )
-
-logging.basicConfig(
-    level=logging.INFO,
-    format="%(asctime)s %(levelname)s %(name)s %(message)s",
-)
-logger = logging.getLogger("backfill_house_coords")
-
-# Yandex Geocoder service limits per docs (as of 2026-05):
-#   - 25k requests/day free tier
-#   - 25 requests/sec sustained
-# 50ms between calls = ~20 req/sec, leaving headroom for connection ramp-up.
-_REQUEST_DELAY_S = 0.05
-
-# Precision values we ACCEPT for backfill — anything else means Yandex didn't
-# resolve to a specific building, and writing the result back into houses
-# would degrade matching recall.
-#   `exact`  → match found at the exact address (best case)
-#   `number` → house number matched, but unit/entrance unspecified (acceptable)
-# `near` / `range` / `street` / `other` / None → skipped (logged for analysis).
-_BACKFILL_OK_PRECISION = frozenset({"exact", "number"})
-
-# Audit threshold per issue #582 — distances above this flag a "mismatch"
-# (the row still goes in the audit table, just with status='mismatch' for
-# the report SQL to bucket separately).
-_MISMATCH_DISTANCE_M = 50.0
-
-
-# ---------------------------------------------------------------------------
-# Domain types
-# ---------------------------------------------------------------------------
-
-
-@dataclass
-class HouseRow:
-    """One house from the source query — minimal fields needed for geocode."""
-
-    id: int
-    address: str
-    lat: float | None
-    lon: float | None
-
-
-# ---------------------------------------------------------------------------
-# Source-row queries
-# ---------------------------------------------------------------------------
-
-
-def _select_houses_without_coords(db: Session, limit: int | None) -> list[HouseRow]:
-    """Pull houses needing forward geocode (lat IS NULL OR lon IS NULL).
-
-    Skips rows with empty address — there's nothing to geocode there, they
-    need a separate cleanup pass.
-    """
-    sql = (
-        "SELECT id, address, lat, lon "
-        "FROM houses "
-        "WHERE (lat IS NULL OR lon IS NULL) "
-        "  AND address IS NOT NULL "
-        "  AND length(trim(address)) > 0 "
-        "ORDER BY id"
-    )
-    if limit is not None:
-        sql += " LIMIT CAST(:limit AS int)"
-        rows = db.execute(text(sql), {"limit": limit}).mappings().all()
-    else:
-        rows = db.execute(text(sql)).mappings().all()
-    return [
-        HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows
-    ]
-
-
-def _select_houses_with_coords(db: Session, limit: int | None) -> list[HouseRow]:
-    """Pull houses needing reverse audit (both lat AND lon present)."""
-    sql = (
-        "SELECT id, address, lat, lon "
-        "FROM houses "
-        "WHERE lat IS NOT NULL "
-        "  AND lon IS NOT NULL "
-        "  AND address IS NOT NULL "
-        "  AND length(trim(address)) > 0 "
-        "ORDER BY id"
-    )
-    if limit is not None:
-        sql += " LIMIT CAST(:limit AS int)"
-        rows = db.execute(text(sql), {"limit": limit}).mappings().all()
-    else:
-        rows = db.execute(text(sql)).mappings().all()
-    return [
-        HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows
-    ]
-
-
-def _already_processed_ids(db: Session, batch: str) -> set[int]:
-    """house_ids already in address_mismatch_audit for this batch → skip set."""
-    rows = db.execute(
-        text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"),
-        {"b": batch},
-    ).all()
-    return {r[0] for r in rows}
-
-
-# ---------------------------------------------------------------------------
-# Distance helper — PostGIS, lon/lat order
-# ---------------------------------------------------------------------------
-
-
-def _distance_meters(
-    db: Session, olat: float, olon: float, slat: float, slon: float
-) -> float | None:
-    """Great-circle distance (meters) via PostGIS geography type.
-
-    Lifted from `audit_address_mismatch.py` to keep the two scripts using
-    the same authority for distance computation. ST_MakePoint takes lon
-    first per PostGIS convention.
-    """
-    row = db.execute(
-        text(
-            "SELECT ST_Distance("
-            "  ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), "
-            "                          CAST(:olat AS double precision)), 4326)::geography, "
-            "  ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), "
-            "                          CAST(:slat AS double precision)), 4326)::geography"
-            ") AS m"
-        ),
-        {"olat": olat, "olon": olon, "slat": slat, "slon": slon},
-    ).first()
-    if row is None or row[0] is None:
-        return None
-    return float(row[0])
-
-
-# ---------------------------------------------------------------------------
-# DB writers
-# ---------------------------------------------------------------------------
-
-
-def _update_house_coords(
-    db: Session,
-    *,
-    house_id: int,
-    lat: float,
-    lon: float,
-    payload: dict[str, Any],
-) -> None:
-    """UPDATE houses SET lat/lon + merge yandex_geocode into raw_payload.
-
-    The `houses_set_geom_trg` BEFORE UPDATE trigger (009_houses.sql) maintains
-    `geom` automatically when lat/lon change, so we don't need to set geom
-    explicitly here. `raw_payload || jsonb_build_object(...)` is the idiomatic
-    psycopg-safe way to merge — single ALTER, no read-modify-write race.
-    """
-    db.execute(
-        text(
-            "UPDATE houses "
-            "   SET lat = CAST(:lat AS double precision), "
-            "       lon = CAST(:lon AS double precision), "
-            "       raw_payload = COALESCE(raw_payload, '{}'::jsonb) "
-            "                     || jsonb_build_object('yandex_geocode', "
-            "                                           CAST(:payload AS jsonb)) "
-            " WHERE id = CAST(:id AS bigint)"
-        ),
-        {"id": house_id, "lat": lat, "lon": lon, "payload": json.dumps(payload)},
-    )
-
-
-def _insert_audit_row(
-    db: Session,
-    *,
-    house_id: int,
-    batch: str,
-    original_address: str | None,
-    original_lat: float | None,
-    original_lon: float | None,
-    snapped_address: str | None,
-    snapped_lat: float | None,
-    snapped_lon: float | None,
-    distance_m: float | None,
-    audit_status: str,
-    error_message: str | None,
-    raw_payload: dict[str, Any] | None,
-) -> None:
-    """INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit.
-
-    Same column shape as `audit_address_mismatch._insert_audit_row` but the
-    `district` and `street_differs` fields are left NULL — backfill/audit
-    here doesn't have a stratification basis and we let the report SQL
-    derive district at query time if needed (via Yandex address parse).
-
-    Caller wraps in `begin_nested()` per backend.md SAVEPOINT pattern.
-    """
-    db.execute(
-        text(
-            "INSERT INTO address_mismatch_audit ("
-            "  house_id, audit_batch, district,"
-            "  original_address, original_lat, original_lon,"
-            "  snapped_address, snapped_lat, snapped_lon,"
-            "  distance_m, street_differs,"
-            "  audit_status, error_message, raw_payload"
-            ") VALUES ("
-            "  CAST(:house_id AS bigint), CAST(:batch AS text), NULL,"
-            "  :original_address, :original_lat, :original_lon,"
-            "  :snapped_address, :snapped_lat, :snapped_lon,"
-            "  :distance_m, NULL,"
-            "  CAST(:audit_status AS text), :error_message,"
-            "  CAST(:raw_payload AS jsonb)"
-            ") ON CONFLICT (house_id, audit_batch) DO NOTHING"
-        ),
-        {
-            "house_id": house_id,
-            "batch": batch,
-            "original_address": original_address,
-            "original_lat": original_lat,
-            "original_lon": original_lon,
-            "snapped_address": snapped_address,
-            "snapped_lat": snapped_lat,
-            "snapped_lon": snapped_lon,
-            "distance_m": distance_m,
-            "audit_status": audit_status,
-            "error_message": error_message,
-            "raw_payload": json.dumps(raw_payload) if raw_payload is not None else None,
-        },
-    )
-
-
-# ---------------------------------------------------------------------------
-# Backfill loop (forward geocode, lat IS NULL houses)
-# ---------------------------------------------------------------------------
-
-
-def _classify_backfill_status(res: YandexReverseResult | None) -> str:
-    """Translate a forward-geocode result into an audit_status value.
-
-    'backfill' — Yandex returned a precise hit, lat/lon will be written.
-    'imprecise' — match returned but precision is too low (street/other/...).
-    'no_match' — Yandex returned an empty featureMember.
-    'error' — handled by the caller's exception branch.
-    """
-    if res is None or res.address is None:
-        return "no_match"
-    if res.precision not in _BACKFILL_OK_PRECISION:
-        return "imprecise"
-    if res.snapped_lat is None or res.snapped_lon is None:
-        return "no_match"
-    return "backfill"
-
-
-async def _run_backfill_mode(
-    db: Session, sample: list[HouseRow], batch: str, api_key: str
-) -> int:
-    """Forward-geocode each house, UPDATE coords on precise hits, audit-log all."""
-    processed = 0
-    updated = 0
-    n_imprecise = 0
-    n_no_match = 0
-    n_error = 0
-
-    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
-        for i, row in enumerate(sample, start=1):
-            status = "backfill"
-            err: str | None = None
-            res: YandexReverseResult | None = None
-            try:
-                res = await forward_via_api(row.address, api_key, client=client)
-            except httpx.HTTPError as e:
-                status = "error"
-                err = f"http_error: {e!s}"
-                n_error += 1
-            except Exception as e:  # pragma: no cover — defensive
-                status = "error"
-                err = f"unhandled: {e!s}"
-                n_error += 1
-
-            if status != "error":
-                status = _classify_backfill_status(res)
-                if status == "imprecise":
-                    n_imprecise += 1
-                elif status == "no_match":
-                    n_no_match += 1
-
-            try:
-                with db.begin_nested():
-                    if status == "backfill" and res is not None and res.snapped_lat is not None:
-                        # safe: status='backfill' guarantees snapped_lat/lon non-None.
-                        assert res.snapped_lon is not None
-                        _update_house_coords(
-                            db,
-                            house_id=row.id,
-                            lat=res.snapped_lat,
-                            lon=res.snapped_lon,
-                            payload={
-                                "address": res.address,
-                                "precision": res.precision,
-                                "kind": res.kind,
-                                "batch": batch,
-                                "source": "yandex_geocoder_api",
-                            },
-                        )
-                        updated += 1
-                    _insert_audit_row(
-                        db,
-                        house_id=row.id,
-                        batch=batch,
-                        original_address=row.address,
-                        original_lat=row.lat,
-                        original_lon=row.lon,
-                        snapped_address=res.address if res else None,
-                        snapped_lat=res.snapped_lat if res else None,
-                        snapped_lon=res.snapped_lon if res else None,
-                        distance_m=None,
-                        audit_status=status,
-                        error_message=err,
-                        raw_payload=res.raw if res else None,
-                    )
-                # Per-row commit so resume picks up exactly where we crashed.
-                db.commit()
-                processed += 1
-            except Exception as e:
-                db.rollback()
-                logger.warning("backfill insert failed for house_id=%s: %s", row.id, e)
-
-            if i % 25 == 0:
-                logger.info(
-                    "backfill progress %d/%d updated=%d imprecise=%d no_match=%d error=%d",
-                    i,
-                    len(sample),
-                    updated,
-                    n_imprecise,
-                    n_no_match,
-                    n_error,
-                )
-
-            # Yandex 25 req/sec → 50ms between calls is plenty of headroom.
-            if i < len(sample):
-                await asyncio.sleep(_REQUEST_DELAY_S)
-
-    logger.info(
-        "backfill done: processed=%d updated=%d imprecise=%d no_match=%d error=%d",
-        processed,
-        updated,
-        n_imprecise,
-        n_no_match,
-        n_error,
-    )
-    return processed
-
-
-# ---------------------------------------------------------------------------
-# Audit-only loop (reverse geocode, lat IS NOT NULL houses)
-# ---------------------------------------------------------------------------
-
-
-async def _run_audit_mode(
-    db: Session, sample: list[HouseRow], batch: str, api_key: str
-) -> int:
-    """Reverse-geocode each house, compute distance, audit-log status/mismatch."""
-    processed = 0
-    n_ok = 0
-    n_mismatch = 0
-    n_no_match = 0
-    n_error = 0
-
-    async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
-        for i, row in enumerate(sample, start=1):
-            # Type-narrow: audit mode only feeds rows with non-null coords.
-            assert row.lat is not None and row.lon is not None
-            status = "ok"
-            err: str | None = None
-            res: YandexReverseResult | None = None
-            try:
-                res = await reverse_via_api(row.lat, row.lon, api_key, client=client)
-            except httpx.HTTPError as e:
-                status = "error"
-                err = f"http_error: {e!s}"
-                n_error += 1
-            except Exception as e:  # pragma: no cover — defensive
-                status = "error"
-                err = f"unhandled: {e!s}"
-                n_error += 1
-
-            distance = None
-            if res is not None and status == "ok":
-                if res.address is None:
-                    status = "no_match"
-                    n_no_match += 1
-                else:
-                    if res.snapped_lat is not None and res.snapped_lon is not None:
-                        distance = _distance_meters(
-                            db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
-                        )
-                        if distance is not None and distance > _MISMATCH_DISTANCE_M:
-                            status = "mismatch"
-                            n_mismatch += 1
-                        else:
-                            n_ok += 1
-                    else:
-                        n_ok += 1
-
-            try:
-                with db.begin_nested():
-                    _insert_audit_row(
-                        db,
-                        house_id=row.id,
-                        batch=batch,
-                        original_address=row.address,
-                        original_lat=row.lat,
-                        original_lon=row.lon,
-                        snapped_address=res.address if res else None,
-                        snapped_lat=res.snapped_lat if res else None,
-                        snapped_lon=res.snapped_lon if res else None,
-                        distance_m=distance,
-                        audit_status=status,
-                        error_message=err,
-                        raw_payload=res.raw if res else None,
-                    )
-                db.commit()
-                processed += 1
-            except Exception as e:
-                db.rollback()
-                logger.warning("audit insert failed for house_id=%s: %s", row.id, e)
-
-            if i % 25 == 0:
-                logger.info(
-                    "audit progress %d/%d ok=%d mismatch=%d no_match=%d error=%d",
-                    i,
-                    len(sample),
-                    n_ok,
-                    n_mismatch,
-                    n_no_match,
-                    n_error,
-                )
-
-            if i < len(sample):
-                await asyncio.sleep(_REQUEST_DELAY_S)
-
-    logger.info(
-        "audit done: processed=%d ok=%d mismatch=%d no_match=%d error=%d",
-        processed,
-        n_ok,
-        n_mismatch,
-        n_no_match,
-        n_error,
-    )
-    return processed
-
-
-# ---------------------------------------------------------------------------
-# CLI
-# ---------------------------------------------------------------------------
-
-
-def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
-    """argparse setup, factored out for testability."""
-    p = argparse.ArgumentParser(
-        description=(
-            "Phase 2-3 of issue #582 — backfill houses.lat/lon via Yandex forward "
-            "geocode, or audit already-geocoded houses via reverse geocode."
-        ),
-    )
-    p.add_argument(
-        "--batch",
-        default=f"{date.today().isoformat()}_backfill",
-        help="Audit batch label. Same batch re-run skips already-processed houses.",
-    )
-    p.add_argument(
-        "--audit-only",
-        action="store_true",
-        help=(
-            "Run reverse-geocode audit on houses WITH coords instead of forward "
-            "backfill on houses WITHOUT coords. Does not modify the houses table."
-        ),
-    )
-    p.add_argument(
-        "--limit",
-        type=int,
-        default=None,
-        help=(
-            "Optional cap on source-row count. Useful for canary runs "
-            "(e.g. --limit 100 before letting the full 4k loose)."
-        ),
-    )
-    return p.parse_args(argv)
-
-
-async def main(argv: list[str] | None = None) -> int:
-    """CLI entry point. Returns the number of rows processed this run."""
-    args = _parse_args(argv)
-    api_key = os.environ.get("YANDEX_GEOCODER_API_KEY")
-    if not api_key:
-        raise SystemExit(
-            "YANDEX_GEOCODER_API_KEY is required — forward geocode is API-only."
-        )
-
-    mode = "audit" if args.audit_only else "backfill"
-    logger.info(
-        "starting batch=%s mode=%s limit=%s",
-        args.batch,
-        mode,
-        args.limit if args.limit is not None else "all",
-    )
-
-    db = SessionLocal()
-    try:
-        if args.audit_only:
-            sample = _select_houses_with_coords(db, args.limit)
-        else:
-            sample = _select_houses_without_coords(db, args.limit)
-        logger.info("loaded source rows: %d", len(sample))
-
-        done = _already_processed_ids(db, args.batch)
-        if done:
-            logger.info(
-                "resuming batch %s: %d rows already processed",
-                args.batch,
-                len(done),
-            )
-        remaining = [s for s in sample if s.id not in done]
-        if not remaining:
-            logger.info("nothing to do — batch %s is complete for the loaded sample", args.batch)
-            return 0
-
-        if args.audit_only:
-            n = await _run_audit_mode(db, remaining, args.batch, api_key)
-        else:
-            n = await _run_backfill_mode(db, remaining, args.batch, api_key)
-
-        logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode)
-        return n
-    finally:
-        db.close()
-
-
-if __name__ == "__main__":  # pragma: no cover
-    asyncio.run(main())

tradein-mvp/backend/tests/fixtures/yandex_geocode_sample.json

@@ -1,74 +0,0 @@
-{
-  "response": {
-    "GeoObjectCollection": {
-      "metaDataProperty": {
-        "GeocoderResponseMetaData": {
-          "request": "60.586,56.838",
-          "results": "1",
-          "found": "1"
-        }
-      },
-      "featureMember": [
-        {
-          "GeoObject": {
-            "metaDataProperty": {
-              "GeocoderMetaData": {
-                "precision": "exact",
-                "text": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
-                "kind": "house",
-                "Address": {
-                  "country_code": "RU",
-                  "formatted": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
-                  "postal_code": "620075",
-                  "Components": [
-                    {"kind": "country", "name": "Россия"},
-                    {"kind": "province", "name": "Уральский федеральный округ"},
-                    {"kind": "province", "name": "Свердловская область"},
-                    {"kind": "area", "name": "городской округ Екатеринбург"},
-                    {"kind": "locality", "name": "Екатеринбург"},
-                    {"kind": "street", "name": "улица Малышева"},
-                    {"kind": "house", "name": "51"}
-                  ]
-                },
-                "AddressDetails": {
-                  "Country": {
-                    "AddressLine": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
-                    "CountryNameCode": "RU",
-                    "CountryName": "Россия",
-                    "AdministrativeArea": {
-                      "AdministrativeAreaName": "Свердловская область",
-                      "SubAdministrativeArea": {
-                        "SubAdministrativeAreaName": "городской округ Екатеринбург",
-                        "Locality": {
-                          "LocalityName": "Екатеринбург",
-                          "Thoroughfare": {
-                            "ThoroughfareName": "улица Малышева",
-                            "Premise": {
-                              "PremiseNumber": "51",
-                              "PostalCode": {"PostalCodeNumber": "620075"}
-                            }
-                          }
-                        }
-                      }
-                    }
-                  }
-                }
-              }
-            },
-            "name": "улица Малышева, 51",
-            "description": "Екатеринбург, Россия",
-            "boundedBy": {
-              "Envelope": {
-                "lowerCorner": "60.585217 56.837461",
-                "upperCorner": "60.587094 56.838547"
-              }
-            },
-            "Point": {
-              "pos": "60.586155 56.838004"
-            }
-          }
-        }
-      ]
-    }
-  }
-}

tradein-mvp/backend/tests/test_audit_address_mismatch.py

@@ -1,373 +0,0 @@
-"""Unit tests for the Phase-1 address-mismatch audit (issue #582).
-
-Coverage:
-  - `_first_street_token` / `_street_differs` — normalization-driven diff.
-  - `_distance_meters` — verified against a MagicMock'd DB that returns a
-    canned distance, plus a Haversine cross-check on the bind values to
-    catch lat/lon swaps.
-  - `reverse_via_api` via httpx MockTransport with the fixture file.
-  - `main()` resumability — call twice with the same batch, second call
-    inserts 0 (uses MagicMock DB session).
-
-Why no real Postgres in unit tests:
-The repo doesn't bundle pytest-postgresql / testcontainers and the existing
-tests all use `MagicMock` for the DB. We follow that convention here. The
-distance and SQL-level resumability are validated by:
-  - The Haversine cross-check (pure-Python expected ≈ PostGIS result for
-    same coords, see `test_distance_calc_matches_haversine`).
-  - Calling `main()` twice in `test_audit_script_resumable` — first run
-    inserts N rows, second run sees the same set of house_ids in the
-    "already processed" query and processes 0.
-"""
-
-from __future__ import annotations
-
-import json
-import math
-import os
-from pathlib import Path
-from unittest.mock import AsyncMock, MagicMock, patch
-
-# Settings requires DATABASE_URL at init time — set dummy DSN before any
-# `app.*` import (same pattern as test_cian_valuation.py).
-os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
-
-import httpx
-import pytest
-
-from scripts._yandex_reverse import (
-    YandexBlockedError,
-    YandexReverseResult,
-    _parse_api_payload,
-    reverse_via_api,
-)
-from scripts.audit_address_mismatch import (
-    SampleRow,
-    _distance_meters,
-    _first_street_token,
-    _resolve_mode,
-    _run_api_mode,
-    _street_differs,
-    main,
-)
-
-_FIXTURES = Path(__file__).parent / "fixtures"
-
-
-# ---------------------------------------------------------------------------
-# _first_street_token / _street_differs
-# ---------------------------------------------------------------------------
-
-
-def test_normalize_address_street_token_basic():
-    """First identifying token of a normalized address — skips street type."""
-    assert _first_street_token("ул Малышева 51") == "малышева"
-
-
-def test_normalize_address_street_token_skips_leading_numbers():
-    """Numeric tokens are skipped — the street name carries identity."""
-    # No type prefix → first non-numeric token is the street name itself.
-    assert _first_street_token("123 Постовского") == "постовского"
-
-
-def test_normalize_address_street_token_handles_none():
-    assert _first_street_token(None) is None
-    assert _first_street_token("") is None
-
-
-def test_street_differs_true_when_streets_differ():
-    assert _street_differs("ул Малышева 51", "ул Ленина 51") is True
-
-
-def test_street_differs_false_when_same_after_normalization():
-    # 'ул' expands to 'улица' on both sides → same first token.
-    assert _street_differs("ул Малышева 51", "улица Малышева, 51") is False
-
-
-def test_street_differs_none_on_empty_side():
-    assert _street_differs(None, "ул Малышева 51") is None
-    assert _street_differs("ул Малышева 51", "") is None
-
-
-# ---------------------------------------------------------------------------
-# _distance_meters — MagicMock DB + Haversine cross-check
-# ---------------------------------------------------------------------------
-
-
-def _haversine_m(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
-    """Reference implementation for sanity-checking the PostGIS call."""
-    r = 6_371_000.0
-    p1 = math.radians(lat1)
-    p2 = math.radians(lat2)
-    dp = math.radians(lat2 - lat1)
-    dl = math.radians(lon2 - lon1)
-    a = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2
-    return 2 * r * math.asin(math.sqrt(a))
-
-
-def test_distance_calc_passes_correct_bindings():
-    """Test the helper passes lat/lon in correct order to the SQL bind names."""
-    db = MagicMock()
-    # PostGIS would return one row, single column (distance in meters).
-    db.execute.return_value.first.return_value = (123.45,)
-
-    out = _distance_meters(db, 56.838, 60.586, 56.840, 60.590)
-
-    assert out == 123.45
-    # Verify the bind dict — guard against lat/lon swap regressions.
-    args, _kwargs = db.execute.call_args
-    bound = args[1]
-    assert bound == {
-        "olat": 56.838,
-        "olon": 60.586,
-        "slat": 56.840,
-        "slon": 60.590,
-    }
-
-
-def test_distance_calc_returns_none_when_postgis_null():
-    """ST_Distance can return NULL — caller must propagate None, not 0."""
-    db = MagicMock()
-    db.execute.return_value.first.return_value = (None,)
-    assert _distance_meters(db, 56.0, 60.0, 56.0, 60.0) is None
-
-
-def test_distance_calc_matches_haversine_within_tolerance():
-    """Sanity check: if PostGIS returned 555.7m for a known pair, that's
-    within ~1% of the Haversine reference (PostGIS uses Vincenty on
-    geography which is slightly more accurate)."""
-    expected = _haversine_m(56.838, 60.586, 56.843, 60.591)
-    # Just assert reference is in a sensible range — proves the test helper
-    # works; the actual call is mocked.
-    assert 500 < expected < 700
-
-
-# ---------------------------------------------------------------------------
-# Yandex API: payload parsing + reverse_via_api with MockTransport
-# ---------------------------------------------------------------------------
-
-
-def test_yandex_parse_api_fixture():
-    """Sanity check: parse the bundled fixture into a YandexReverseResult."""
-    data = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8"))
-    res = _parse_api_payload(data)
-    assert res.address is not None
-    assert "Малышева" in res.address
-    # Fixture Point.pos = "60.586155 56.838004" → lon then lat.
-    assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6)
-    assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6)
-    assert res.raw == data
-
-
-def test_yandex_parse_api_no_match():
-    """Empty featureMember → all-None result, raw still preserved."""
-    data = {"response": {"GeoObjectCollection": {"featureMember": []}}}
-    res = _parse_api_payload(data)
-    assert res.address is None
-    assert res.snapped_lat is None
-    assert res.snapped_lon is None
-    assert res.raw == data
-
-
-async def test_yandex_reverse_api_mock():
-    """End-to-end: reverse_via_api hits a MockTransport, returns parsed result."""
-    fixture = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8"))
-
-    captured: dict[str, httpx.Request] = {}
-
-    def handler(request: httpx.Request) -> httpx.Response:
-        captured["req"] = request
-        return httpx.Response(200, json=fixture)
-
-    transport = httpx.MockTransport(handler)
-    async with httpx.AsyncClient(transport=transport) as client:
-        res = await reverse_via_api(56.838004, 60.586155, "DUMMY_KEY", client=client)
-
-    assert res.address is not None and "Малышева" in res.address
-    # Verify the request shape — lon,lat order + apikey + kind=house.
-    req = captured["req"]
-    qs = dict(httpx.QueryParams(req.url.query))
-    assert qs["apikey"] == "DUMMY_KEY"
-    assert qs["geocode"] == "60.586155,56.838004"
-    assert qs["format"] == "json"
-    assert qs["kind"] == "house"
-
-
-async def test_yandex_blocked_error_raised_on_captcha():
-    """`reverse_via_playwright` must raise YandexBlockedError on captcha.
-
-    We mock the page object so we don't need an actual browser.
-    """
-    from scripts._yandex_reverse import reverse_via_playwright
-
-    page = MagicMock()
-    page.goto = AsyncMock()
-    page.wait_for_load_state = AsyncMock()
-    page.query_selector = AsyncMock(
-        side_effect=lambda sel: MagicMock() if sel == ".CheckboxCaptcha" else None
-    )
-    page.evaluate = AsyncMock(return_value=[None, None])
-
-    with pytest.raises(YandexBlockedError):
-        await reverse_via_playwright(56.838, 60.586, page)
-
-
-# ---------------------------------------------------------------------------
-# Mode resolver
-# ---------------------------------------------------------------------------
-
-
-def test_resolve_mode_auto_with_key():
-    assert _resolve_mode("auto", "abc") == "api"
-
-
-def test_resolve_mode_auto_without_key():
-    assert _resolve_mode("auto", None) == "playwright"
-    assert _resolve_mode("auto", "") == "playwright"
-
-
-def test_resolve_mode_explicit_passes_through():
-    assert _resolve_mode("api", None) == "api"
-    assert _resolve_mode("playwright", "abc") == "playwright"
-
-
-# ---------------------------------------------------------------------------
-# Resumability — main() twice with same batch
-# ---------------------------------------------------------------------------
-
-
-def _make_db_mock(initial_sample: list[dict], processed_ids: set[int]):
-    """Build a MagicMock SQLAlchemy session that:
-    - returns `initial_sample` for the sampling SQL (text() with limit_per_district)
-    - returns `processed_ids` for the resume SQL (text() with batch only)
-    - records INSERTs so the test can count them
-    """
-    inserted: list[dict] = []
-
-    db = MagicMock()
-    db.begin_nested.return_value.__enter__ = lambda self: self
-    db.begin_nested.return_value.__exit__ = lambda self, *a: False
-
-    def execute_side_effect(sql, params=None):
-        sql_str = str(sql)
-        result = MagicMock()
-        if "FROM houses h" in sql_str or "houses_in_districts" in sql_str:
-            result.mappings.return_value.all.return_value = initial_sample
-        elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str:
-            # Resume query — returns list of (house_id,) tuples.
-            result.all.return_value = [(hid,) for hid in processed_ids]
-        elif "INSERT INTO address_mismatch_audit" in sql_str:
-            inserted.append(dict(params))
-            # Simulate ON CONFLICT DO NOTHING — track id locally for re-run.
-            processed_ids.add(params["house_id"])
-            result = MagicMock()
-        elif "ST_Distance" in sql_str:
-            result.first.return_value = (42.0,)
-        else:
-            result = MagicMock()
-        return result
-
-    db.execute.side_effect = execute_side_effect
-    db.commit = MagicMock()
-    db.rollback = MagicMock()
-    db.close = MagicMock()
-    return db, inserted
-
-
-async def test_audit_script_resumable(monkeypatch):
-    """Run main() twice with the same batch — second pass inserts 0."""
-    sample = [
-        {
-            "id": 1,
-            "address": "ул Малышева 51",
-            "lat": 56.838,
-            "lon": 60.586,
-            "district": "Кировский",
-        },
-        {"id": 2, "address": "ул Ленина 5", "lat": 56.840, "lon": 60.600, "district": "Ленинский"},
-    ]
-    processed_ids: set[int] = set()
-    db, inserted = _make_db_mock(sample, processed_ids)
-
-    # Force API mode without needing a real key.
-    monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
-
-    fake_result = YandexReverseResult(
-        address="Россия, Екатеринбург, улица Малышева, 51",
-        snapped_lat=56.838004,
-        snapped_lon=60.586155,
-        raw={"ok": True},
-    )
-
-    with (
-        patch("scripts.audit_address_mismatch.SessionLocal", return_value=db),
-        patch(
-            "scripts.audit_address_mismatch.reverse_via_api",
-            new=AsyncMock(return_value=fake_result),
-        ),
-    ):
-        # First run — both rows processed.
-        n1 = await main(["--batch", "test_batch_1", "--mode", "api"])
-        assert n1 == 2
-        assert len(inserted) == 2
-
-        # Second run with same batch — nothing left to do.
-        inserted.clear()
-        n2 = await main(["--batch", "test_batch_1", "--mode", "api"])
-        assert n2 == 0
-        assert inserted == []
-
-
-async def test_audit_script_api_mode_marks_error(monkeypatch):
-    """When the reverse call raises, the row is still inserted with status=error."""
-    sample = [
-        {
-            "id": 99,
-            "address": "ул Малышева 51",
-            "lat": 56.838,
-            "lon": 60.586,
-            "district": "Кировский",
-        },
-    ]
-    processed_ids: set[int] = set()
-    db, inserted = _make_db_mock(sample, processed_ids)
-
-    monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
-
-    with (
-        patch("scripts.audit_address_mismatch.SessionLocal", return_value=db),
-        patch(
-            "scripts.audit_address_mismatch.reverse_via_api",
-            new=AsyncMock(side_effect=httpx.HTTPError("boom")),
-        ),
-    ):
-        n = await main(["--batch", "err_batch", "--mode", "api"])
-        assert n == 1
-
-    assert len(inserted) == 1
-    assert inserted[0]["audit_status"] == "error"
-    assert "boom" in (inserted[0]["error_message"] or "")
-
-
-# ---------------------------------------------------------------------------
-# Internal _run_api_mode no-match path
-# ---------------------------------------------------------------------------
-
-
-async def test_api_mode_no_match_path():
-    """If Yandex returns address=None, row goes in with status=no_match."""
-    sample = [SampleRow(id=7, address="ул X 1", lat=56.0, lon=60.0, district="Кировский")]
-    processed_ids: set[int] = set()
-    db, inserted = _make_db_mock([], processed_ids)
-
-    res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True})
-
-    with patch(
-        "scripts.audit_address_mismatch.reverse_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_api_mode(db, sample, "b1", "key")
-
-    assert n == 1
-    assert inserted[0]["audit_status"] == "no_match"
-    assert inserted[0]["snapped_address"] is None

tradein-mvp/backend/tests/test_backfill_house_coords.py

@@ -1,510 +0,0 @@
-"""Unit tests for the Phase 2-3 backfill/audit script (issue #582).
-
-Coverage:
-  - `forward_via_api` request shape — verifies geocode/format/locality bias.
-  - `_parse_api_payload` precision + kind extraction.
-  - `_classify_backfill_status` precision filter rules.
-  - `_update_house_coords` — UPDATE shape + raw_payload merge.
-  - `_run_backfill_mode` — happy path UPDATE + audit row, plus imprecise-skip.
-  - `_run_audit_mode` — ok / mismatch / no_match distinction.
-  - `main()` resumability — second pass on same batch inserts 0.
-
-No real Postgres in unit tests (same convention as test_audit_address_mismatch).
-DB is a MagicMock that records INSERT/UPDATE calls and routes SELECT side-effects.
-"""
-
-from __future__ import annotations
-
-import json
-import os
-from unittest.mock import AsyncMock, MagicMock, patch
-
-# Same dance as test_audit_address_mismatch — settings needs a DSN at import.
-os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
-
-import httpx
-import pytest
-
-from scripts._yandex_reverse import (
-    YandexReverseResult,
-    _parse_api_payload,
-    forward_via_api,
-)
-from scripts.backfill_house_coords import (
-    HouseRow,
-    _classify_backfill_status,
-    _run_audit_mode,
-    _run_backfill_mode,
-    _update_house_coords,
-    main,
-)
-
-# ---------------------------------------------------------------------------
-# forward_via_api — request shape
-# ---------------------------------------------------------------------------
-
-
-async def test_forward_api_request_shape():
-    """Verify the GET param dict — address as `geocode`, kind=house, EKB bias."""
-    fixture = {
-        "response": {
-            "GeoObjectCollection": {
-                "featureMember": [
-                    {
-                        "GeoObject": {
-                            "metaDataProperty": {
-                                "GeocoderMetaData": {
-                                    "text": "Россия, Свердловская область, Екатеринбург, "
-                                    "улица Малышева, 51",
-                                    "precision": "exact",
-                                    "kind": "house",
-                                }
-                            },
-                            "name": "улица Малышева, 51",
-                            "Point": {"pos": "60.586155 56.838004"},
-                        }
-                    }
-                ]
-            }
-        }
-    }
-    captured: dict[str, httpx.Request] = {}
-
-    def handler(request: httpx.Request) -> httpx.Response:
-        captured["req"] = request
-        return httpx.Response(200, json=fixture)
-
-    transport = httpx.MockTransport(handler)
-    async with httpx.AsyncClient(transport=transport) as client:
-        res = await forward_via_api("ул Малышева 51", "DUMMY_KEY", client=client)
-
-    assert res.address is not None and "Малышева" in res.address
-    assert res.precision == "exact"
-    assert res.kind == "house"
-    assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6)
-    assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6)
-
-    qs = dict(httpx.QueryParams(captured["req"].url.query))
-    assert qs["apikey"] == "DUMMY_KEY"
-    assert qs["geocode"] == "ул Малышева 51"
-    assert qs["format"] == "json"
-    assert qs["kind"] == "house"
-    # EKB locality bias for forward geocode — important so addresses without
-    # the city resolve to the correct Малышева (there's one in Moscow too).
-    assert "ll" in qs
-    assert "spn" in qs
-
-
-# ---------------------------------------------------------------------------
-# Precision / kind passthrough in _parse_api_payload
-# ---------------------------------------------------------------------------
-
-
-def test_parse_api_payload_propagates_precision_and_kind():
-    data = {
-        "response": {
-            "GeoObjectCollection": {
-                "featureMember": [
-                    {
-                        "GeoObject": {
-                            "metaDataProperty": {
-                                "GeocoderMetaData": {
-                                    "text": "ул Ленина 5",
-                                    "precision": "exact",
-                                    "kind": "house",
-                                }
-                            },
-                            "name": "ул Ленина 5",
-                            "Point": {"pos": "60.6 56.8"},
-                        }
-                    }
-                ]
-            }
-        }
-    }
-    res = _parse_api_payload(data)
-    assert res.precision == "exact"
-    assert res.kind == "house"
-
-
-# ---------------------------------------------------------------------------
-# _classify_backfill_status — precision filter rules
-# ---------------------------------------------------------------------------
-
-
-def test_classify_backfill_status_exact_match():
-    res = YandexReverseResult(
-        address="ул Малышева 51",
-        snapped_lat=56.838,
-        snapped_lon=60.586,
-        precision="exact",
-        kind="house",
-    )
-    assert _classify_backfill_status(res) == "backfill"
-
-
-def test_classify_backfill_status_number_match():
-    res = YandexReverseResult(
-        address="ул Ленина 5",
-        snapped_lat=56.840,
-        snapped_lon=60.600,
-        precision="number",
-        kind="house",
-    )
-    assert _classify_backfill_status(res) == "backfill"
-
-
-def test_classify_backfill_status_street_is_imprecise():
-    res = YandexReverseResult(
-        address="ул Ленина",
-        snapped_lat=56.840,
-        snapped_lon=60.600,
-        precision="street",
-        kind="street",
-    )
-    assert _classify_backfill_status(res) == "imprecise"
-
-
-def test_classify_backfill_status_other_is_imprecise():
-    res = YandexReverseResult(
-        address="Свердловская область",
-        snapped_lat=56.8,
-        snapped_lon=60.6,
-        precision="other",
-        kind="locality",
-    )
-    assert _classify_backfill_status(res) == "imprecise"
-
-
-def test_classify_backfill_status_no_match():
-    res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None)
-    assert _classify_backfill_status(res) == "no_match"
-
-
-def test_classify_backfill_status_none():
-    assert _classify_backfill_status(None) == "no_match"
-
-
-def test_classify_backfill_status_precision_ok_but_no_coords():
-    """Defensive: precision=exact but snapped point missing → no_match, not backfill."""
-    res = YandexReverseResult(
-        address="ул Малышева 51",
-        snapped_lat=None,
-        snapped_lon=None,
-        precision="exact",
-        kind="house",
-    )
-    assert _classify_backfill_status(res) == "no_match"
-
-
-# ---------------------------------------------------------------------------
-# _update_house_coords — UPDATE shape verification
-# ---------------------------------------------------------------------------
-
-
-def test_update_house_coords_passes_bindings():
-    db = MagicMock()
-    _update_house_coords(
-        db,
-        house_id=42,
-        lat=56.838,
-        lon=60.586,
-        payload={"address": "ул Малышева 51", "precision": "exact"},
-    )
-    args, _kw = db.execute.call_args
-    sql_str = str(args[0])
-    binds = args[1]
-    assert "UPDATE houses" in sql_str
-    assert "raw_payload" in sql_str
-    assert "yandex_geocode" in sql_str
-    assert binds["id"] == 42
-    assert binds["lat"] == 56.838
-    assert binds["lon"] == 60.586
-    # payload bound as JSON string for CAST(:payload AS jsonb)
-    decoded = json.loads(binds["payload"])
-    assert decoded["address"] == "ул Малышева 51"
-
-
-# ---------------------------------------------------------------------------
-# DB mock helper — same approach as test_audit_address_mismatch
-# ---------------------------------------------------------------------------
-
-
-def _make_db_mock(
-    backfill_sample: list[dict] | None = None,
-    audit_sample: list[dict] | None = None,
-    processed_ids: set[int] | None = None,
-    distance_value: float = 12.5,
-):
-    """MagicMock DB that:
-    - returns `backfill_sample` for `lat IS NULL OR lon IS NULL` SELECT
-    - returns `audit_sample` for `lat IS NOT NULL` SELECT
-    - returns `processed_ids` for the resume SELECT
-    - records INSERTs and UPDATEs
-    - returns `distance_value` for ST_Distance calls
-    """
-    backfill_sample = backfill_sample or []
-    audit_sample = audit_sample or []
-    processed_ids = processed_ids if processed_ids is not None else set()
-
-    inserted: list[dict] = []
-    updated: list[dict] = []
-
-    db = MagicMock()
-    db.begin_nested.return_value.__enter__ = lambda self: self
-    db.begin_nested.return_value.__exit__ = lambda self, *a: False
-
-    def execute_side_effect(sql, params=None):
-        sql_str = str(sql)
-        result = MagicMock()
-        if "FROM houses" in sql_str and "lat IS NULL OR lon IS NULL" in sql_str:
-            result.mappings.return_value.all.return_value = backfill_sample
-        elif "FROM houses" in sql_str and "lat IS NOT NULL" in sql_str:
-            result.mappings.return_value.all.return_value = audit_sample
-        elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str:
-            result.all.return_value = [(hid,) for hid in processed_ids]
-        elif "INSERT INTO address_mismatch_audit" in sql_str:
-            inserted.append(dict(params))
-            processed_ids.add(params["house_id"])
-        elif "UPDATE houses" in sql_str:
-            updated.append(dict(params))
-        elif "ST_Distance" in sql_str:
-            result.first.return_value = (distance_value,)
-        return result
-
-    db.execute.side_effect = execute_side_effect
-    db.commit = MagicMock()
-    db.rollback = MagicMock()
-    db.close = MagicMock()
-    return db, inserted, updated
-
-
-# ---------------------------------------------------------------------------
-# _run_backfill_mode — happy path + imprecise-skip
-# ---------------------------------------------------------------------------
-
-
-async def test_run_backfill_mode_writes_update_and_audit():
-    sample = [
-        HouseRow(id=1, address="ул Малышева 51", lat=None, lon=None),
-    ]
-    db, inserted, updated = _make_db_mock()
-    res = YandexReverseResult(
-        address="Россия, Екатеринбург, улица Малышева, 51",
-        snapped_lat=56.838,
-        snapped_lon=60.586,
-        precision="exact",
-        kind="house",
-        raw={"ok": True},
-    )
-    with patch(
-        "scripts.backfill_house_coords.forward_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_backfill_mode(db, sample, "b1", "KEY")
-    assert n == 1
-    assert len(updated) == 1
-    assert updated[0]["id"] == 1
-    assert updated[0]["lat"] == 56.838
-    assert updated[0]["lon"] == 60.586
-    assert len(inserted) == 1
-    assert inserted[0]["audit_status"] == "backfill"
-    assert inserted[0]["snapped_address"] == "Россия, Екатеринбург, улица Малышева, 51"
-
-
-async def test_run_backfill_mode_imprecise_skips_update():
-    """precision='street' → audit row written with status=imprecise, no UPDATE."""
-    sample = [HouseRow(id=2, address="ул Ленина", lat=None, lon=None)]
-    db, inserted, updated = _make_db_mock()
-    res = YandexReverseResult(
-        address="ул Ленина",
-        snapped_lat=56.840,
-        snapped_lon=60.600,
-        precision="street",
-        kind="street",
-        raw={"oh_well": True},
-    )
-    with patch(
-        "scripts.backfill_house_coords.forward_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_backfill_mode(db, sample, "b2", "KEY")
-    assert n == 1
-    assert updated == []
-    assert len(inserted) == 1
-    assert inserted[0]["audit_status"] == "imprecise"
-
-
-async def test_run_backfill_mode_no_match():
-    """Yandex returns empty result → status=no_match, no UPDATE."""
-    sample = [HouseRow(id=3, address="несуществующая улица 99", lat=None, lon=None)]
-    db, inserted, updated = _make_db_mock()
-    res = YandexReverseResult(
-        address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True}
-    )
-    with patch(
-        "scripts.backfill_house_coords.forward_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_backfill_mode(db, sample, "b3", "KEY")
-    assert n == 1
-    assert updated == []
-    assert inserted[0]["audit_status"] == "no_match"
-
-
-async def test_run_backfill_mode_http_error_marks_error():
-    sample = [HouseRow(id=4, address="ул X 1", lat=None, lon=None)]
-    db, inserted, updated = _make_db_mock()
-    with patch(
-        "scripts.backfill_house_coords.forward_via_api",
-        new=AsyncMock(side_effect=httpx.HTTPError("boom")),
-    ):
-        n = await _run_backfill_mode(db, sample, "b4", "KEY")
-    assert n == 1
-    assert updated == []
-    assert inserted[0]["audit_status"] == "error"
-    assert "boom" in (inserted[0]["error_message"] or "")
-
-
-# ---------------------------------------------------------------------------
-# _run_audit_mode — ok / mismatch / no_match
-# ---------------------------------------------------------------------------
-
-
-async def test_run_audit_mode_ok_within_50m():
-    sample = [HouseRow(id=10, address="ул Малышева 51", lat=56.838, lon=60.586)]
-    db, inserted, _updated = _make_db_mock(distance_value=12.5)
-    res = YandexReverseResult(
-        address="Россия, Екатеринбург, улица Малышева, 51",
-        snapped_lat=56.838004,
-        snapped_lon=60.586155,
-        precision="exact",
-        kind="house",
-        raw={"r": 1},
-    )
-    with patch(
-        "scripts.backfill_house_coords.reverse_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_audit_mode(db, sample, "ba1", "KEY")
-    assert n == 1
-    assert inserted[0]["audit_status"] == "ok"
-    assert inserted[0]["distance_m"] == 12.5
-
-
-async def test_run_audit_mode_mismatch_above_50m():
-    sample = [HouseRow(id=11, address="ул Ленина 5", lat=56.840, lon=60.600)]
-    db, inserted, _updated = _make_db_mock(distance_value=312.0)
-    res = YandexReverseResult(
-        address="Россия, Екатеринбург, улица Ленина, 7",
-        snapped_lat=56.841,
-        snapped_lon=60.601,
-        precision="exact",
-        kind="house",
-        raw={"r": 2},
-    )
-    with patch(
-        "scripts.backfill_house_coords.reverse_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_audit_mode(db, sample, "ba2", "KEY")
-    assert n == 1
-    assert inserted[0]["audit_status"] == "mismatch"
-    assert inserted[0]["distance_m"] == 312.0
-
-
-async def test_run_audit_mode_no_match():
-    sample = [HouseRow(id=12, address="ул X 99", lat=56.0, lon=60.0)]
-    db, inserted, _updated = _make_db_mock()
-    res = YandexReverseResult(
-        address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True}
-    )
-    with patch(
-        "scripts.backfill_house_coords.reverse_via_api",
-        new=AsyncMock(return_value=res),
-    ):
-        n = await _run_audit_mode(db, sample, "ba3", "KEY")
-    assert n == 1
-    assert inserted[0]["audit_status"] == "no_match"
-
-
-# ---------------------------------------------------------------------------
-# Resumability — second pass on same batch inserts 0
-# ---------------------------------------------------------------------------
-
-
-async def test_main_resumable_skips_processed(monkeypatch):
-    """Run main() twice with same batch — second pass processes nothing."""
-    backfill_sample = [
-        {"id": 1, "address": "ул Малышева 51", "lat": None, "lon": None},
-        {"id": 2, "address": "ул Ленина 5", "lat": None, "lon": None},
-    ]
-    processed_ids: set[int] = set()
-    db, inserted, updated = _make_db_mock(
-        backfill_sample=backfill_sample, processed_ids=processed_ids
-    )
-
-    monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
-    fake = YandexReverseResult(
-        address="ул Малышева 51",
-        snapped_lat=56.838,
-        snapped_lon=60.586,
-        precision="exact",
-        kind="house",
-        raw={"ok": True},
-    )
-
-    with (
-        patch("scripts.backfill_house_coords.SessionLocal", return_value=db),
-        patch(
-            "scripts.backfill_house_coords.forward_via_api",
-            new=AsyncMock(return_value=fake),
-        ),
-    ):
-        n1 = await main(["--batch", "resume_test"])
-        assert n1 == 2
-        assert len(inserted) == 2
-        assert len(updated) == 2
-
-        inserted.clear()
-        updated.clear()
-        n2 = await main(["--batch", "resume_test"])
-        assert n2 == 0
-        assert inserted == []
-        assert updated == []
-
-
-async def test_main_requires_api_key(monkeypatch):
-    """Without YANDEX_GEOCODER_API_KEY the script exits cleanly."""
-    monkeypatch.delenv("YANDEX_GEOCODER_API_KEY", raising=False)
-    with pytest.raises(SystemExit):
-        await main(["--batch", "no_key"])
-
-
-async def test_main_audit_only_flag_routes_to_audit_loop(monkeypatch):
-    """--audit-only switches sample query + loop, no UPDATE expected."""
-    audit_sample = [
-        {"id": 50, "address": "ул Малышева 51", "lat": 56.838, "lon": 60.586},
-    ]
-    db, inserted, updated = _make_db_mock(audit_sample=audit_sample, distance_value=8.0)
-    monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
-    fake = YandexReverseResult(
-        address="Россия, Екатеринбург, улица Малышева, 51",
-        snapped_lat=56.838004,
-        snapped_lon=60.586155,
-        precision="exact",
-        kind="house",
-        raw={"r": 1},
-    )
-    with (
-        patch("scripts.backfill_house_coords.SessionLocal", return_value=db),
-        patch(
-            "scripts.backfill_house_coords.reverse_via_api",
-            new=AsyncMock(return_value=fake),
-        ),
-    ):
-        n = await main(["--batch", "audit_run", "--audit-only"])
-    assert n == 1
-    assert updated == []  # audit mode never updates houses
-    assert inserted[0]["audit_status"] == "ok"
-    assert inserted[0]["distance_m"] == 8.0

tradein-mvp/backend/tests/test_geocoder_nominatim_lookup.py

@@ -0,0 +1,91 @@
+"""Тесты `_nominatim_lookup` — city реально доходит до исходящего HTTP-запроса.
+
+#2593 (часть 3): Yandex Geocoder полностью удалён из проекта, вместе с ним ушли
+`_yandex_reverse.py` + `tests/test_audit_address_mismatch.py` +
+`tests/test_backfill_house_coords.py` — они были единственной проверкой, что
+`city`/`city_hint` реально передаётся во внешний геокодер, а не только влияет на
+cache-ключ (см. `tests/test_geocoder_city_hint.py`, который мокает
+`_nominatim_lookup`/`_nominatim_suggest` целиком и потому не видит их внутренности).
+
+Nominatim теперь единственный живой внешний провайдер (`_nominatim_lookup`
+docstring, `app/services/geocoder.py`) — этот файл закрывает получившуюся дыру:
+мокает HTTP-транспорт (`httpx.MockTransport`, паттерн из `test_geocoder_bbox.py` /
+`tests/services/test_dadata.py`) и проверяет параметр `q` реального исходящего
+GET-запроса к `nominatim.openstreetmap.org/search`.
+"""
+
+from __future__ import annotations
+
+import os
+
+os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")
+
+from unittest.mock import patch
+
+import httpx
+
+from app.services.geocoder import _nominatim_lookup
+
+# EKB-центр (Плотинка) — внутри tight EKB bbox, `_nominatim_query` его примет
+# без похода во второй (typo-variant) тир.
+_SAMPLE_ITEM = {
+    "lat": "56.838",
+    "lon": "60.605",
+    "class": "building",
+    "display_name": "ул. Малышева, 30, Екатеринбург",
+    "address": {"state": "Свердловская область"},
+}
+
+# Snapshot реального httpx.AsyncClient ДО patch'а — фабрика ниже использует именно
+# его с подменённым transport (паттерн tests/services/test_dadata.py: избегает
+# recursion, если бы `httpx.AsyncClient` патчился поверх самого себя).
+_REAL_ASYNC_CLIENT = httpx.AsyncClient
+
+
+def _async_client_factory(transport: httpx.MockTransport):
+    def factory(*_: object, **__: object) -> httpx.AsyncClient:
+        return _REAL_ASYNC_CLIENT(transport=transport)
+
+    return factory
+
+
+def _capturing_transport(captured_q: list[str]) -> httpx.MockTransport:
+    def handler(request: httpx.Request) -> httpx.Response:
+        captured_q.append(request.url.params.get("q", ""))
+        return httpx.Response(200, json=[_SAMPLE_ITEM])
+
+    return httpx.MockTransport(handler)
+
+
+async def test_nominatim_lookup_sends_city_hint_in_query_param() -> None:
+    """city_hint="Нижний Тагил" должен попасть в q= реального GET-запроса.
+
+    Регрессия, о которой явно предупреждает docstring `_nominatim_lookup` (#2580 C):
+    city_hint обязан влиять на сам запрос к провайдеру, не только на cache-ключ.
+    """
+    captured_q: list[str] = []
+    transport = _capturing_transport(captured_q)
+
+    with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)):
+        result = await _nominatim_lookup("Ленина, 1", city_hint="Нижний Тагил")
+
+    assert captured_q, "запрос к Nominatim не был отправлен"
+    assert captured_q[0] == "Нижний Тагил, Ленина, 1"
+    assert result is not None
+    assert result.provider == "nominatim"
+
+
+async def test_nominatim_lookup_no_city_sends_bare_address() -> None:
+    """Без city_hint и без маркера города в тексте — q= остаётся bare-адресом.
+
+    Guard против регрессии в молчаливый дефолт на конкретный город (#2576/#2593)
+    — до фикса #2576 сюда молча подставлялся "Екатеринбург".
+    """
+    captured_q: list[str] = []
+    transport = _capturing_transport(captured_q)
+
+    with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)):
+        result = await _nominatim_lookup("Малышева, 30")
+
+    assert captured_q == ["Малышева, 30"]
+    assert result is not None