diff --git a/docs/Secrets_Rotation_Policy.md b/docs/Secrets_Rotation_Policy.md index 8a913a42..61180727 100644 --- a/docs/Secrets_Rotation_Policy.md +++ b/docs/Secrets_Rotation_Policy.md @@ -77,7 +77,6 @@ |---|---|---| | `TRADEIN_POSTGRES_PASSWORD` / `TRADEIN_POSTGRES_USER` | Пароль/юзер БД `tradein` | **E** | | `TRADEIN_READER_PASSWORD` | Пароль роли `gendesign_reader` (ETL #976, `ops/db-bootstrap/set_gendesign_reader_password.sql`) | **E** | -| `YANDEX_GEOCODER_API_KEY` | Yandex Geocoder (25k req/day) | **D** | | `DADATA_API_TOKEN` / `DADATA_API_SECRET` | DaData `/clean/address` enrichment | **D** | | `SCRAPER_PROXY_URL` (+ legacy `AVITO_PROXY_URL`, `CIAN_PROXY_URL`, `YANDEX_PROXY_URL` и их `*_ROTATE_URL`) | Мобильный прокси для скраперов (содержит user:pass в URL) | **G** (proxy creds) | | `CIAN_LOGIN_EMAIL` / `CIAN_LOGIN_PASSWORD` | Cian browser auto-login (#639, Variant B) | **D** | @@ -147,14 +146,14 @@ bcrypt-хеши — односторонние, не plaintext-секреты, 3. Frontend: обновить `GLITCHTIP_FRONTEND_DSN` (build-arg `NEXT_PUBLIC_GLITCHTIP_DSN`) → требует **rebuild frontend образа** (запекается на build-time) → `workflow_dispatch` или push в `frontend/**`. 4. Vault entry. -### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`) +### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`) **Downtime:** нет (фичи gracefully degrade при пустом ключе — см. config-комментарии). -1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / Yandex Cloud / DaData / Cian-аккаунт). +1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / DaData / Cian-аккаунт). 2. Где живёт: - `OBJECTIVE_API_KEY`, `OPENAI_API_KEY` — Forgejo secret → deploy пишет в main `.env.runtime`. - - `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI). + - `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI). 3. Обновить значение `sed`-ом (НЕ перезапись файла) и `up -d --force-recreate --no-deps backend worker beat` (main) / `... backend scraper` (tradein). 4. Vault entry. diff --git a/tradein-mvp/DEPLOY.md b/tradein-mvp/DEPLOY.md index 6b9fdc2c..d8012ea5 100644 --- a/tradein-mvp/DEPLOY.md +++ b/tradein-mvp/DEPLOY.md @@ -57,8 +57,7 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment shell-скриптом deploy через `source .env.runtime` перед `compose up`. 2. `/opt/gendesign/tradein-mvp/backend/.env.runtime` — переменные внутри контейнера `tradein-backend` (читаются через `env_file:` в compose). Сюда - попадают `YANDEX_GEOCODER_API_KEY`, `COOKIE_ENCRYPTION_KEY` — - всё, что нужно scripts/backfill_house_coords.py и application code внутри + попадают `COOKIE_ENCRYPTION_KEY` и остальные application-секреты внутри контейнера. ```bash @@ -66,7 +65,6 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment TRADEIN_POSTGRES_USER=tradein TRADEIN_POSTGRES_PASSWORD=<сгенерировать openssl rand -hex 32> TRADEIN_CONTACT_EMAIL=tradein@gendsgn.ru -YANDEX_GEOCODER_API_KEY= # пусто пока, Nominatim fallback работает # Encryption key for Cian session cookies (pgp_sym_encrypt / Stage 9 Calculator). # Empty = Valuation Calculator scraper disabled + /api/v1/cookies/upload returns 503. @@ -77,10 +75,9 @@ COOKIE_ENCRYPTION_KEY=<64-char hex> ```bash # /opt/gendesign/tradein-mvp/backend/.env.runtime — те же ключи которые -# читаются ВНУТРИ container'а (scripts/backfill_house_coords.py, app/*). +# читаются ВНУТРИ container'а (app/*, scripts/*.py). # Может быть симлинком на ../.env.runtime если переменные совпадают: # ln -s ../.env.runtime /opt/gendesign/tradein-mvp/backend/.env.runtime -YANDEX_GEOCODER_API_KEY= COOKIE_ENCRYPTION_KEY=<64-char hex> GENDESIGN_FDW_PASSWORD= GLITCHTIP_DSN= @@ -200,7 +197,6 @@ cat > tradein-mvp/.env.runtime < Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell. -> На prod используй canonical `docker exec` команды из секции выше — там -> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`. - -### `audit_address_mismatch.py` — Phase 1 baseline (PR #583) - -Stratified-sample audit (200 EKB houses) comparing `houses.address` vs -Yandex Geocoder reverse lookup. Writes one row per house into -`address_mismatch_audit` with the snapped point + canonical address + distance. - -```bash -DATABASE_URL=postgresql+psycopg://... \ -YANDEX_GEOCODER_API_KEY=... \ -uv run python -m scripts.audit_address_mismatch \ - --batch 2026-05-25_run1 \ - --limit-per-district 25 -``` - -Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware, -4-7s sleep between calls). API tier free is 25k req/day → 200-row sample -takes ~10s with no quota concern. - -Report: - -```bash -psql "$DATABASE_URL" -v batch='2026-05-25_run1' \ - -f scripts/address_audit_report.sql -``` - -### `backfill_house_coords.py` — Phase 2-3 (PR for #582) - -Two modes (`--audit-only` flag switches between them): - -**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows -WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns -`precision='exact'` or `'number'` (skips street-only / locality matches). -Each processed row gets an `address_mismatch_audit` entry with status -`backfill` / `imprecise` / `no_match` / `error`. - -```bash -DATABASE_URL=postgresql+psycopg://... \ -YANDEX_GEOCODER_API_KEY=... \ -uv run python -m scripts.backfill_house_coords \ - --batch 2026-05-27_backfill -``` - -Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request): -20-25 min. Expected output split (rough baseline from Phase 1 numbers): - -| Status | Approx rows | What it means | -|-------------|-------------|-----------------------------------------------------| -| `backfill` | ~3.3k–3.7k | UPDATE landed, lat/lon now populated | -| `imprecise` | ~300–500 | Match returned but precision too low — needs review | -| `no_match` | ~100–300 | Yandex couldn't resolve; address probably mangled | -| `error` | <50 | HTTP errors / timeouts — re-run picks them up | - -**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write -audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`. -Does NOT modify the `houses` table. - -```bash -uv run python -m scripts.backfill_house_coords \ - --batch 2026-05-27_audit --audit-only -``` - -Combined budget for both phases (~8.6k requests) is well under the 25k/day -Geocoder free tier. - -### Common ops - -Canary first — run with `--limit 100` and inspect the audit table before -letting the full job loose: - -```bash -uv run python -m scripts.backfill_house_coords \ - --batch canary_$(date +%F) --limit 100 -psql "$DATABASE_URL" -c " - SELECT audit_status, COUNT(*) - FROM address_mismatch_audit - WHERE audit_batch = 'canary_$(date +%F)' - GROUP BY audit_status; -" -``` - -Resume after crash / quota hit — same `--batch` label, the UNIQUE -`(house_id, audit_batch)` index skips finished rows: - -```bash -uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill -# ... interruption ... -uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill -# logs: "resuming batch 2026-05-27_backfill: N rows already processed" -``` - -### Helpers (not entry points) - -- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`, - `reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API - paths share `_parse_api_payload` because Yandex's forward/reverse - envelopes have the same shape. -- `audit_address_sample.sql` — random sample for the Phase 1 audit (used - by `audit_address_mismatch.py`). -- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95 - distance, top-20 outliers, per-district breakdown). +`audit_address_mismatch.py`, `backfill_house_coords.py`, `_yandex_reverse.py` +и их SQL-хелперы (`audit_address_sample.sql`, `address_audit_report.sql`) +удалены — весь pipeline опирался на Yandex Geocoder API, который выпилен +из проекта (#2593, части 1-3). `houses.address`→lat/lon geocoding теперь +идёт через `app/services/geocoder.py` (кадастр/геопортал ЕКБ-тиры + Nominatim +fallback, единственный живой внешний провайдер) на обычном write-path +(`/api/v1/trade-in/estimate`, listing ingest). Разовый forward-backfill +недостающих `houses` координат — `scripts/geocode_deals_nominatim.py` +(живой, работает с `rosreestr_deals`, не с `houses` — читай его docstring +перед использованием на других таблицах). Таблица `address_mismatch_audit` +осталась в схеме (используется `house_dedup_merge.py` при слиянии дублей +домов, независимо от Yandex-аудита). --- diff --git a/tradein-mvp/backend/scripts/_yandex_reverse.py b/tradein-mvp/backend/scripts/_yandex_reverse.py deleted file mode 100644 index 2e01146c..00000000 --- a/tradein-mvp/backend/scripts/_yandex_reverse.py +++ /dev/null @@ -1,380 +0,0 @@ -"""Yandex Geocoder helpers for the address-mismatch audit + backfill (issue #582). - -Three geocoding paths exposed: - -- `reverse_via_api()` — Yandex Geocoder HTTP API, lon/lat → address. Fast, - structured response, needs a valid API key (env `YANDEX_GEOCODER_API_KEY`). - Free tier is 25k req/day, fine for ~8.5k houses + audit (~17k total). - -- `reverse_via_playwright()` — fallback when no API key is available. Drives - a real browser session at https://yandex.ru/maps/?…&mode=whatshere. Slower - and CAPTCHA-prone, so the driver inserts 4-7s sleeps between calls and we - raise a dedicated exception on CAPTCHA so the batch can pause-and-resume. - -- `forward_via_api()` — address → lon/lat + canonical address (Phase 2 of - issue #582). Used by `backfill_house_coords.py` to fill `houses.lat/lon` - for the 4141 houses scraped from sources that didn't include coords (esp. - yandex_valuation, which only returns an address string). - -All three return a `YandexReverseResult` dataclass — same shape regardless -of direction so the driver code stays implementation-agnostic. The `raw` -field always carries the full source payload for post-hoc diagnostics, and -`precision` / `kind` are filled in by the API paths so the caller can skip -imprecise matches (e.g. only-street-level results during backfill). - -Why three paths: -The user (issue #582 discussion) wants the audit to run on dev machines -that may not have an API key, but on prod we already provision the key for -estimator.py. Forward geocode is API-only — Playwright forward geocoding -through Yandex Maps search is too fragile (relevance ranking, suggest -dropdown). For dev without a key, backfill simply doesn't run. -""" - -from __future__ import annotations - -import asyncio -import logging -import random -from dataclasses import dataclass, field -from typing import Any - -import httpx - -logger = logging.getLogger(__name__) - -# Yandex Maps "what's here" URL — wraps a reverse-geocode in browser-driven UI. -# `whatshere[point]` accepts "," (note: lon first, Yandex convention). -_YANDEX_MAPS_WHATSHERE = ( - "https://yandex.ru/maps/?ll={lon:.6f}%2C{lat:.6f}&z=18&mode=whatshere" - "&whatshere%5Bpoint%5D={lon:.6f}%2C{lat:.6f}&whatshere%5Bzoom%5D=18" -) - -# Geocoder HTTP API. `kind=house` narrows the result to a building if possible, -# which is what we want for cadastr-style addresses (улица + дом). -_YANDEX_GEOCODE_API = "https://geocode-maps.yandex.ru/1.x/" - -# Reasonable timeouts: API call should be sub-second; we give it generous -# headroom for slow networks but not so much that a hang stalls the batch. -_API_TIMEOUT = httpx.Timeout(connect=5.0, read=10.0, write=5.0, pool=5.0) - - -# --------------------------------------------------------------------------- -# Dataclasses + exceptions -# --------------------------------------------------------------------------- - - -@dataclass -class YandexReverseResult: - """Normalized result of a geocode call (forward, reverse-API, or browser). - - Attributes: - address: Human-readable canonical address Yandex returned. For - reverse, this is the snapped address at the queried point. For - forward, this is the canonical form of the input address. None - if Yandex returned no match. - snapped_lat: Latitude of the matched object's geometric centre. - snapped_lon: Longitude of the matched object's geometric centre. - precision: For forward calls — Yandex match precision tag (`exact`, - `number`, `near`, `range`, `street`, `other`). For reverse — - same field is filled when present (usually `house` / `street`). - None for the playwright path. Used by the backfill driver to - skip imprecise matches. - kind: Object kind from Yandex (`house`, `street`, `locality`, ...). - Same source as `precision` — see metaDataProperty.GeocoderMetaData. - raw: Raw response payload retained for forensics (JSON dict from API, - or snapshot dict from playwright). Used to populate - `address_mismatch_audit.raw_payload` and - `houses.raw_payload.yandex_geocode`. - """ - - address: str | None - snapped_lat: float | None - snapped_lon: float | None - raw: dict[str, Any] = field(default_factory=dict) - precision: str | None = None - kind: str | None = None - - -class YandexBlockedError(RuntimeError): - """Raised when Yandex returns a CAPTCHA / anti-bot challenge. - - The driver catches this, marks the row `audit_status='blocked'`, logs the - current batch position, then exits cleanly so a human can intervene. - """ - - -# --------------------------------------------------------------------------- -# Path A — HTTP Geocoder API -# --------------------------------------------------------------------------- - - -async def reverse_via_api( - lat: float, - lon: float, - api_key: str, - *, - client: httpx.AsyncClient | None = None, -) -> YandexReverseResult: - """Reverse-geocode (lat, lon) via the Yandex Geocoder HTTP API. - - Why a separate `client` parameter: lets the driver reuse one - `AsyncClient` across all 200 calls (TCP keep-alive + connection pool), - and lets the tests inject a `MockTransport` to assert request shape. - - Args: - lat: latitude in WGS84. - lon: longitude in WGS84. - api_key: Yandex Geocoder API key. - client: optional pre-built async client. If None, a one-shot client - is created. - - Returns: - `YandexReverseResult` with the first `featureMember[0].GeoObject` - result, or all-None if Yandex returned no match (still includes - `raw` payload so we can later inspect why). - """ - params = { - "apikey": api_key, - # Yandex expects "lon,lat" (longitude first) per docs — same - # convention as the "whatshere" map URL above. - "geocode": f"{lon},{lat}", - "format": "json", - "kind": "house", - "results": "1", - } - - own_client = client is None - if client is None: - client = httpx.AsyncClient(timeout=_API_TIMEOUT) - - try: - resp = await client.get(_YANDEX_GEOCODE_API, params=params) - resp.raise_for_status() - data = resp.json() - finally: - if own_client: - await client.aclose() - - return _parse_api_payload(data) - - -def _parse_api_payload(data: dict[str, Any]) -> YandexReverseResult: - """Extract address + snapped point from a Yandex Geocoder API JSON response. - - Split out so unit tests can feed a fixture file directly without spinning - up an HTTP mock. Same payload shape for forward and reverse calls — - Yandex's response envelope is symmetric. - """ - try: - members = data.get("response", {}).get("GeoObjectCollection", {}).get("featureMember", []) - if not members: - return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data) - - geo_obj = members[0].get("GeoObject", {}) - - # Address: prefer the long `metaDataProperty.GeocoderMetaData.text` - # (full canonical) and fall back to `name` (street + house number). - meta = geo_obj.get("metaDataProperty", {}).get("GeocoderMetaData", {}) - address = meta.get("text") or geo_obj.get("name") - precision = meta.get("precision") - kind = meta.get("kind") - - # Point format: " " — space-separated string. - point_str = geo_obj.get("Point", {}).get("pos", "") - snapped_lon: float | None - snapped_lat: float | None - if point_str: - try: - lon_s, lat_s = point_str.split() - snapped_lon = float(lon_s) - snapped_lat = float(lat_s) - except (ValueError, TypeError): - snapped_lon = None - snapped_lat = None - else: - snapped_lon = None - snapped_lat = None - - return YandexReverseResult( - address=address, - snapped_lat=snapped_lat, - snapped_lon=snapped_lon, - raw=data, - precision=precision, - kind=kind, - ) - except Exception as e: # pragma: no cover — defensive; tests cover happy paths - logger.warning("yandex API payload parse failed: %s", e) - return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data) - - -# --------------------------------------------------------------------------- -# Path A.2 — Forward geocode (address → lon/lat) via HTTP API -# --------------------------------------------------------------------------- - - -async def forward_via_api( - address: str, - api_key: str, - *, - client: httpx.AsyncClient | None = None, -) -> YandexReverseResult: - """Forward-geocode an address string via the Yandex Geocoder HTTP API. - - Phase 2 of issue #582 — used by `backfill_house_coords.py` to populate - `houses.lat/lon` for houses that were scraped without coords (esp. - yandex_valuation rows, which only carry an address). - - Args: - address: free-form address ("ул Малышева 51", "Екатеринбург, Ленина 5", - etc.). Yandex's NLU is forgiving — no need to pre-normalize. - api_key: Yandex Geocoder API key. - client: optional pre-built async client. If None, a one-shot client - is created (matches `reverse_via_api` ergonomics). - - Returns: - `YandexReverseResult` with the canonical address + snapped point of - the first matching feature. `precision` and `kind` are populated so - the backfill driver can skip imprecise hits (e.g. precision='street' - means we landed on the road, not the building — too vague for - comparable-listings spatial queries). - - Same envelope as `reverse_via_api` — `_parse_api_payload` handles both. - """ - params = { - "apikey": api_key, - "geocode": address, - "format": "json", - # `kind=house` filters out street-only / locality-only matches at - # the API level when possible. Yandex still returns lower-precision - # results when no building matches, so the caller must double-check - # `precision` before writing to houses. - "kind": "house", - "results": "1", - # Locality bias for EKB — improves recall when the input address - # omits the city. The audit population is 99% EKB houses, so this - # is safe; non-EKB inputs (rare) still resolve, just with the bias. - "ll": "60.6122,56.8389", - "spn": "0.6,0.4", - } - - own_client = client is None - if client is None: - client = httpx.AsyncClient(timeout=_API_TIMEOUT) - - try: - resp = await client.get(_YANDEX_GEOCODE_API, params=params) - resp.raise_for_status() - data = resp.json() - finally: - if own_client: - await client.aclose() - - return _parse_api_payload(data) - - -# --------------------------------------------------------------------------- -# Path B — Playwright fallback -# --------------------------------------------------------------------------- - - -async def reverse_via_playwright( - lat: float, - lon: float, - page: Any, -) -> YandexReverseResult: - """Reverse-geocode (lat, lon) by driving yandex.ru/maps with Playwright. - - Why this exists: - The Yandex Geocoder API requires a key with paid quota for >25k/day. The - audit only needs 200 rows but a dev without a key still needs a way to - run the script, so we ship a browser-driven fallback. - - Implementation: - 1. Navigate to the `whatshere` URL — Yandex Maps responds by opening a - toponym card at the requested coordinates and rendering the resolved - address in the side panel. - 2. Wait for client hydration (`networkidle`). - 3. First try to read `window.__INITIAL_STATE__` — Yandex stores the - toponym address inside the hydrated Redux tree, which is more - stable across UI redesigns than DOM selectors. - 4. Fall back to DOM selectors (`.toponym-card-title-view__title` + - `__subtitle`) if the state walk doesn't find an address. - 5. Detect CAPTCHA (`.CheckboxCaptcha`) early and raise `YandexBlockedError` - so the batch can pause-and-resume without spamming Yandex. - - `page` is typed as `Any` to keep playwright a dev-only dep — runtime - importers don't need playwright installed if they only use the API path. - """ - url = _YANDEX_MAPS_WHATSHERE.format(lat=lat, lon=lon) - await page.goto(url, wait_until="domcontentloaded") - - # Light wait for client-side hydration. Yandex Maps fires lots of - # background XHRs so `networkidle` is too aggressive; this small wait is - # enough for the toponym card to render. - try: - await page.wait_for_load_state("networkidle", timeout=8000) - except Exception as e: - # Slow networks: continue — selectors will retry with their own waits. - logger.debug("networkidle wait timed out, continuing: %s", e) - await asyncio.sleep(random.uniform(0.5, 1.2)) - - # CAPTCHA gate — Yandex shows a `.CheckboxCaptcha` form when it suspects - # automation. Once we see it, every subsequent reverse call will also be - # blocked, so we raise immediately and let the driver stop the batch. - captcha = await page.query_selector(".CheckboxCaptcha") - if captcha is not None: - raise YandexBlockedError("Yandex CAPTCHA detected on maps page") - - # Attempt 1 — initial state walk. - state_addr: str | None = None - state_pos: tuple[float, float] | None = None - try: - state_addr, state_pos = await page.evaluate( - "() => {\n" - " const s = window.__INITIAL_STATE__ || {};\n" - " const card = (s.cards && s.cards.toponym) || (s.card && s.card.toponym) || null;\n" - " if (!card) return [null, null];\n" - " const addr = card.title || card.address || null;\n" - " const pos = card.coords || card.point || null;\n" - " if (pos && pos.length === 2) return [addr, [pos[0], pos[1]]];\n" - " return [addr, null];\n" - "}" - ) - except Exception as e: - logger.debug("playwright state walk failed (will fall back to DOM): %s", e) - - address = state_addr - - # Attempt 2 — DOM fallback. - if not address: - title_el = await page.query_selector(".toponym-card-title-view__title") - subtitle_el = await page.query_selector(".toponym-card-title-view__subtitle") - title = (await title_el.inner_text()).strip() if title_el else "" - subtitle = (await subtitle_el.inner_text()).strip() if subtitle_el else "" - # subtitle often holds "Екатеринбург, район", title the street + house - address = ", ".join([p for p in (subtitle, title) if p]) or None - - snapped_lat: float | None - snapped_lon: float | None - if state_pos: - # State stored as [lon, lat] in Yandex's coordinate convention. - snapped_lon = float(state_pos[0]) - snapped_lat = float(state_pos[1]) - else: - snapped_lon = None - snapped_lat = None - - raw = { - "url": url, - "state_addr": state_addr, - "state_pos": list(state_pos) if state_pos else None, - "dom_address": address if not state_addr else None, - } - - return YandexReverseResult( - address=address, - snapped_lat=snapped_lat, - snapped_lon=snapped_lon, - raw=raw, - ) diff --git a/tradein-mvp/backend/scripts/address_audit_report.sql b/tradein-mvp/backend/scripts/address_audit_report.sql deleted file mode 100644 index 7b391b9e..00000000 --- a/tradein-mvp/backend/scripts/address_audit_report.sql +++ /dev/null @@ -1,91 +0,0 @@ --- address_audit_report.sql --- Post-run report for the address-mismatch audit (issue #582 Phase 1). --- --- Sections: --- 1. Summary — count, p50/p75/p95/mean distance, % street_differs, --- % over 50m / 200m thresholds. --- 2. Top-20 outliers by distance (manual triage list). --- 3. Per-district breakdown — same metrics grouped by district column. --- --- Run via psql: --- psql "$DATABASE_URL" -v batch='2026-05-25_run1' -f scripts/address_audit_report.sql --- --- :batch is a psql client variable substituted via -v. - -\set ON_ERROR_STOP on - -\echo '==============================================' -\echo ' Address mismatch audit — batch:' :batch -\echo '==============================================' - --- --------------------------------------------------------------------------- --- 1) Top-level summary --- --------------------------------------------------------------------------- -\echo '' -\echo '--- Summary (status=ok rows only) ---' -SELECT - COUNT(*) AS n_total, - COUNT(*) FILTER (WHERE audit_status = 'ok') AS n_ok, - COUNT(*) FILTER (WHERE audit_status = 'no_match') AS n_no_match, - COUNT(*) FILTER (WHERE audit_status = 'error') AS n_error, - COUNT(*) FILTER (WHERE audit_status = 'blocked') AS n_blocked, - ROUND(percentile_cont(0.50) - WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p50_distance_m, - ROUND(percentile_cont(0.75) - WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p75_distance_m, - ROUND(percentile_cont(0.95) - WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p95_distance_m, - ROUND(AVG(distance_m)::numeric, 1) AS mean_distance_m, - ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1) - AS pct_street_differs, - ROUND(100.0 * AVG(CASE WHEN distance_m > 50 THEN 1.0 ELSE 0.0 END), 1) - AS pct_over_50m, - ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1) - AS pct_over_200m -FROM address_mismatch_audit -WHERE audit_batch = :'batch' - AND audit_status = 'ok'; - --- --------------------------------------------------------------------------- --- 2) Top-20 outliers --- --------------------------------------------------------------------------- -\echo '' -\echo '--- Top-20 outliers by distance ---' -SELECT - house_id, - district, - ROUND(distance_m::numeric, 1) AS distance_m, - street_differs, - LEFT(original_address, 60) AS original_address, - LEFT(snapped_address, 60) AS snapped_address -FROM address_mismatch_audit -WHERE audit_batch = :'batch' - AND audit_status = 'ok' - AND distance_m IS NOT NULL -ORDER BY distance_m DESC NULLS LAST -LIMIT 20; - --- --------------------------------------------------------------------------- --- 3) Per-district breakdown --- --------------------------------------------------------------------------- -\echo '' -\echo '--- Per-district breakdown (status=ok only) ---' -SELECT - COALESCE(district, '(no district)') AS district, - COUNT(*) AS n, - ROUND(percentile_cont(0.50) - WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p50_distance_m, - ROUND(percentile_cont(0.95) - WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p95_distance_m, - ROUND(AVG(distance_m)::numeric, 1) AS mean_distance_m, - ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1) - AS pct_street_differs, - ROUND(100.0 * AVG(CASE WHEN distance_m > 50 THEN 1.0 ELSE 0.0 END), 1) - AS pct_over_50m, - ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1) - AS pct_over_200m -FROM address_mismatch_audit -WHERE audit_batch = :'batch' - AND audit_status = 'ok' -GROUP BY COALESCE(district, '(no district)') -ORDER BY n DESC, district; diff --git a/tradein-mvp/backend/scripts/audit_address_mismatch.py b/tradein-mvp/backend/scripts/audit_address_mismatch.py deleted file mode 100644 index 87dc7d88..00000000 --- a/tradein-mvp/backend/scripts/audit_address_mismatch.py +++ /dev/null @@ -1,595 +0,0 @@ -"""Audit driver — compares houses.address vs Yandex reverse geocode. - -Phase 1 of Forgejo issue #582. Pulls a stratified sample of EKB houses (25 -per admin district = 200 total), reverse-geocodes each via Yandex, computes -the distance between the stored coordinates and the snapped Yandex point, -and writes the result into `address_mismatch_audit`. - -Design choices: -- **Resumable**: the audit table has UNIQUE (house_id, audit_batch). Re-run - with the same `--batch` skips rows already inserted, so a partial run can - be picked up after CAPTCHA / network blip. -- **Mode auto**: prefer API when `YANDEX_GEOCODER_API_KEY` is set, fall back - to Playwright otherwise. Explicit override via `--mode {api,playwright}`. -- **No prod side effects**: the script only writes to one new audit table; - it never touches `houses`, `house_sources`, or any matching/listing row. -- **Per-row SAVEPOINT**: a single Yandex error must not nuke the entire - batch — wrap each INSERT in `db.begin_nested()` per backend.md. - -How to run: - DATABASE_URL=postgresql+psycopg://... \ - YANDEX_GEOCODER_API_KEY=... \ - python -m scripts.audit_address_mismatch --batch 2026-05-25_run1 - -Outputs (post-run): -- New rows in `address_mismatch_audit` with batch label. -- `scripts/address_audit_report.sql :batch=` for summary. -""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import logging -import os -import random -from dataclasses import dataclass -from datetime import date -from pathlib import Path -from typing import Any - -import httpx -from sqlalchemy import text -from sqlalchemy.orm import Session - -# Allow running both as `python -m scripts.audit_address_mismatch` (preferred) -# and as a stand-alone file (`python scripts/audit_address_mismatch.py`) -# without requiring package install. -try: - from app.core.db import SessionLocal # type: ignore[import-not-found] - from app.services.matching.normalize import normalize_address # type: ignore[import-not-found] -except ImportError: # pragma: no cover — fallback for adhoc invocation - import sys - - sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - from app.core.db import SessionLocal - from app.services.matching.normalize import normalize_address - -# `from .` works when run via -m; the absolute import works under pytest. -try: - from scripts._yandex_reverse import ( # type: ignore[import-not-found] - YandexBlockedError, - YandexReverseResult, - reverse_via_api, - reverse_via_playwright, - ) -except ImportError: - from _yandex_reverse import ( # type: ignore[no-redef] - YandexBlockedError, - YandexReverseResult, - reverse_via_api, - reverse_via_playwright, - ) - -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s %(levelname)s %(name)s %(message)s", -) -logger = logging.getLogger("audit_address_mismatch") - -# Playwright persistent context location — keeps cookies/local storage between -# runs so we look like a returning user, reducing CAPTCHA frequency. -_PLAYWRIGHT_USER_DATA = Path.home() / ".cache" / "tradein-audit-playwright" - -_PLAYWRIGHT_UA = ( - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " - "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" -) - -_SAMPLE_SQL_PATH = Path(__file__).parent / "audit_address_sample.sql" - - -# --------------------------------------------------------------------------- -# Domain helpers -# --------------------------------------------------------------------------- - - -@dataclass -class SampleRow: - """One house from the stratified sampling query.""" - - id: int - address: str - lat: float - lon: float - district: str | None - - -# Words that introduce a street rather than identify it. We skip these so the -# comparison lands on the actual street name ('малышева' / 'ленина'). Mirrors -# the canonical forms produced by `normalize_address` (which expands all known -# abbreviations to these full words). -_STREET_TYPE_WORDS = frozenset( - { - "улица", - "проспект", - "переулок", - "бульвар", - "проезд", - "шоссе", - "площадь", - "набережная", - "тупик", - "строение", - "корпус", - "дом", - } -) - -# Geographic prefix words that addresses sometimes carry before the street -# (e.g. 'россия екатеринбург улица малышева 51'). We skip them too so we -# converge on the same identifying token regardless of how verbose the -# source representation is. -_GEO_PREFIX_WORDS = frozenset( - { - "россия", - "свердловская", - "область", - "екатеринбург", - "город", - "г", - } -) - - -def _first_street_token(address: str | None) -> str | None: - """Extract the first street-name token of a normalized address. - - Phase-1 heuristic for "do the streets agree": skip numeric tokens (house - numbers), street-type words ('улица', 'проспект', …), and geographic - prefixes ('россия', 'екатеринбург', …) — the next token is the street - name itself, which is the identifying part we want to compare. - - Returns None for an empty address or when no candidate token remains. - """ - norm = normalize_address(address or "") - if not norm: - return None - for tok in norm.split(): - if not tok: - continue - # Skip purely numeric tokens (e.g. '5', '17а' if it starts with digit). - if tok[0].isdigit(): - continue - # Skip street type words and geographic prefixes. - if tok in _STREET_TYPE_WORDS or tok in _GEO_PREFIX_WORDS: - continue - return tok - return None - - -def _street_differs(original: str | None, snapped: str | None) -> bool | None: - """True iff first non-numeric token differs between the two addresses. - - Returns None when either side is empty — we cannot compute a meaningful - diff (caller writes NULL into the audit row). - """ - a = _first_street_token(original) - b = _first_street_token(snapped) - if a is None or b is None: - return None - return a != b - - -def _distance_meters( - db: Session, - olat: float, - olon: float, - slat: float, - slon: float, -) -> float | None: - """Compute great-circle distance via PostGIS geography type. - - We could do this in Python with a haversine formula, but the audit table - uses ST_Distance results elsewhere so we use the same authority to avoid - drift. ST_MakePoint(lon, lat) — PostGIS convention is lon first. - """ - row = db.execute( - text( - "SELECT ST_Distance(" - " ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), " - " CAST(:olat AS double precision)), 4326)::geography, " - " ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), " - " CAST(:slat AS double precision)), 4326)::geography" - ") AS m" - ), - {"olat": olat, "olon": olon, "slat": slat, "slon": slon}, - ).first() - if row is None or row[0] is None: - return None - return float(row[0]) - - -# --------------------------------------------------------------------------- -# Sampling + resumption queries -# --------------------------------------------------------------------------- - - -def _load_sample(db: Session, limit_per_district: int) -> list[SampleRow]: - """Run the stratified sampling SQL → list of SampleRow.""" - sql = _SAMPLE_SQL_PATH.read_text(encoding="utf-8") - rows = db.execute(text(sql), {"limit_per_district": limit_per_district}).mappings().all() - return [ - SampleRow( - id=r["id"], - address=r["address"], - lat=float(r["lat"]), - lon=float(r["lon"]), - district=r["district"], - ) - for r in rows - ] - - -def _already_processed_ids(db: Session, batch: str) -> set[int]: - """Return the set of house_id already in the audit table for this batch. - - Drives resumability: drop these from the sample before geocoding. - """ - rows = db.execute( - text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"), - {"b": batch}, - ).all() - return {r[0] for r in rows} - - -# --------------------------------------------------------------------------- -# Insert helper -# --------------------------------------------------------------------------- - - -def _insert_audit_row( - db: Session, - *, - house_id: int, - batch: str, - district: str | None, - original_address: str | None, - original_lat: float | None, - original_lon: float | None, - snapped_address: str | None, - snapped_lat: float | None, - snapped_lon: float | None, - distance_m: float | None, - street_differs: bool | None, - audit_status: str, - error_message: str | None, - raw_payload: dict[str, Any] | None, -) -> None: - """INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit. - - Wrapped in begin_nested by the caller per backend.md SAVEPOINT pattern. - """ - db.execute( - text( - "INSERT INTO address_mismatch_audit (" - " house_id, audit_batch, district," - " original_address, original_lat, original_lon," - " snapped_address, snapped_lat, snapped_lon," - " distance_m, street_differs," - " audit_status, error_message, raw_payload" - ") VALUES (" - " CAST(:house_id AS bigint), CAST(:batch AS text), :district," - " :original_address, :original_lat, :original_lon," - " :snapped_address, :snapped_lat, :snapped_lon," - " :distance_m, :street_differs," - " CAST(:audit_status AS text), :error_message," - " CAST(:raw_payload AS jsonb)" - ") ON CONFLICT (house_id, audit_batch) DO NOTHING" - ), - { - "house_id": house_id, - "batch": batch, - "district": district, - "original_address": original_address, - "original_lat": original_lat, - "original_lon": original_lon, - "snapped_address": snapped_address, - "snapped_lat": snapped_lat, - "snapped_lon": snapped_lon, - "distance_m": distance_m, - "street_differs": street_differs, - "audit_status": audit_status, - "error_message": error_message, - "raw_payload": json.dumps(raw_payload) if raw_payload is not None else None, - }, - ) - - -# --------------------------------------------------------------------------- -# Mode dispatcher -# --------------------------------------------------------------------------- - - -def _resolve_mode(mode: str, api_key: str | None) -> str: - """Translate `--mode auto` → concrete 'api' / 'playwright' choice. - - Explicit modes are passed through unchanged; auto chooses api iff a key - is configured (fail-fast: we don't want a "should have used the API but - silently fell back to slow scraping" surprise). - """ - if mode == "auto": - return "api" if api_key else "playwright" - return mode - - -# --------------------------------------------------------------------------- -# Main loop -# --------------------------------------------------------------------------- - - -async def _run_api_mode( - db: Session, - sample: list[SampleRow], - batch: str, - api_key: str, -) -> int: - """Geocode the sample using the HTTP Geocoder API.""" - processed = 0 - last_distance: float | None = None - async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client: - for i, row in enumerate(sample, start=1): - status = "ok" - err: str | None = None - res: YandexReverseResult | None = None - try: - res = await reverse_via_api(row.lat, row.lon, api_key, client=client) - except httpx.HTTPError as e: - status = "error" - err = f"http_error: {e!s}" - except Exception as e: # pragma: no cover — defensive - status = "error" - err = f"unhandled: {e!s}" - - distance = None - street_diff: bool | None = None - if res is not None and status == "ok": - if res.address is None: - status = "no_match" - else: - if res.snapped_lat is not None and res.snapped_lon is not None: - distance = _distance_meters( - db, row.lat, row.lon, res.snapped_lat, res.snapped_lon - ) - last_distance = distance - street_diff = _street_differs(row.address, res.address) - - try: - with db.begin_nested(): - _insert_audit_row( - db, - house_id=row.id, - batch=batch, - district=row.district, - original_address=row.address, - original_lat=row.lat, - original_lon=row.lon, - snapped_address=res.address if res else None, - snapped_lat=res.snapped_lat if res else None, - snapped_lon=res.snapped_lon if res else None, - distance_m=distance, - street_differs=street_diff, - audit_status=status, - error_message=err, - raw_payload=res.raw if res else None, - ) - # Per-row commit: each row is durable on disk before the next - # Yandex call; --batch resume picks up exactly where we crashed. - db.commit() - processed += 1 - except Exception as e: - db.rollback() - logger.warning("insert failed for house_id=%s: %s", row.id, e) - - if i % 10 == 0: - logger.info( - "progress %d/%d, mode=api, last_distance=%s", - i, - len(sample), - f"{last_distance:.1f}m" if last_distance is not None else "n/a", - ) - - return processed - - -async def _run_playwright_mode( - db: Session, - sample: list[SampleRow], - batch: str, -) -> int: - """Geocode via a persistent Playwright context (CAPTCHA-aware).""" - try: - from playwright.async_api import async_playwright # type: ignore[import-not-found] - except ImportError as e: - raise RuntimeError( - "Playwright is required for --mode playwright. " - "Install with `uv sync --group dev` and `playwright install chromium`." - ) from e - - _PLAYWRIGHT_USER_DATA.mkdir(parents=True, exist_ok=True) - processed = 0 - last_distance: float | None = None - - async with async_playwright() as p: - context = await p.chromium.launch_persistent_context( - user_data_dir=str(_PLAYWRIGHT_USER_DATA), - headless=False, - user_agent=_PLAYWRIGHT_UA, - locale="ru-RU", - timezone_id="Asia/Yekaterinburg", - ) - page = await context.new_page() - - try: - for i, row in enumerate(sample, start=1): - status = "ok" - err: str | None = None - res: YandexReverseResult | None = None - stop_batch = False - try: - res = await reverse_via_playwright(row.lat, row.lon, page) - except YandexBlockedError as e: - status = "blocked" - err = str(e) - stop_batch = True - except Exception as e: - status = "error" - err = f"playwright: {e!s}" - - distance = None - street_diff: bool | None = None - if res is not None and status == "ok": - if res.address is None: - status = "no_match" - else: - if res.snapped_lat is not None and res.snapped_lon is not None: - distance = _distance_meters( - db, row.lat, row.lon, res.snapped_lat, res.snapped_lon - ) - last_distance = distance - street_diff = _street_differs(row.address, res.address) - - try: - with db.begin_nested(): - _insert_audit_row( - db, - house_id=row.id, - batch=batch, - district=row.district, - original_address=row.address, - original_lat=row.lat, - original_lon=row.lon, - snapped_address=res.address if res else None, - snapped_lat=res.snapped_lat if res else None, - snapped_lon=res.snapped_lon if res else None, - distance_m=distance, - street_differs=street_diff, - audit_status=status, - error_message=err, - raw_payload=res.raw if res else None, - ) - db.commit() - processed += 1 - except Exception as e: - db.rollback() - logger.warning("insert failed for house_id=%s: %s", row.id, e) - - if stop_batch: - logger.error( - "Yandex CAPTCHA detected at position %d/%d (house_id=%s). " - "Stopping batch — re-run with same --batch to resume.", - i, - len(sample), - row.id, - ) - break - - if i % 10 == 0: - logger.info( - "progress %d/%d, mode=playwright, last_distance=%s", - i, - len(sample), - f"{last_distance:.1f}m" if last_distance is not None else "n/a", - ) - - # Random delay 4-7s between requests — keeps us under Yandex's - # heuristic rate limit while still finishing 200 rows in <30min. - # Skip the wait on the last iteration (no next request to space). - if i < len(sample): - await asyncio.sleep(random.uniform(4.0, 7.0)) - finally: - await context.close() - - return processed - - -# --------------------------------------------------------------------------- -# Entry point -# --------------------------------------------------------------------------- - - -def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: - """argparse setup, factored out for testability.""" - p = argparse.ArgumentParser( - description="Phase 1 audit — houses.address vs Yandex reverse geocode.", - ) - p.add_argument( - "--batch", - default=f"{date.today().isoformat()}_run1", - help="Audit batch label. Same batch re-run skips already-processed houses.", - ) - p.add_argument( - "--limit-per-district", - type=int, - default=25, - help="Houses to sample per district (default 25 → ~200 total for EKB).", - ) - p.add_argument( - "--mode", - choices=("auto", "api", "playwright"), - default="auto", - help="auto = API if YANDEX_GEOCODER_API_KEY set, else playwright.", - ) - return p.parse_args(argv) - - -async def main(argv: list[str] | None = None) -> int: - """CLI entry point. Returns the number of rows processed this run.""" - args = _parse_args(argv) - api_key = os.environ.get("YANDEX_GEOCODER_API_KEY") - mode = _resolve_mode(args.mode, api_key) - - if mode == "api" and not api_key: - raise SystemExit("mode=api requested but YANDEX_GEOCODER_API_KEY is not set") - - logger.info( - "starting audit batch=%s mode=%s limit_per_district=%d", - args.batch, - mode, - args.limit_per_district, - ) - - db = SessionLocal() - try: - sample = _load_sample(db, args.limit_per_district) - logger.info("loaded sample: %d houses", len(sample)) - - # Resume support — drop already-processed house_ids. - done = _already_processed_ids(db, args.batch) - if done: - logger.info( - "resuming batch %s: %d rows already processed, %d remaining", - args.batch, - len(done), - len(sample) - sum(1 for s in sample if s.id in done), - ) - remaining = [s for s in sample if s.id not in done] - - if not remaining: - logger.info("nothing to do — batch %s is complete", args.batch) - return 0 - - if mode == "api": - n = await _run_api_mode(db, remaining, args.batch, api_key or "") - else: - n = await _run_playwright_mode(db, remaining, args.batch) - - logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode) - return n - finally: - db.close() - - -if __name__ == "__main__": # pragma: no cover - asyncio.run(main()) diff --git a/tradein-mvp/backend/scripts/audit_address_sample.sql b/tradein-mvp/backend/scripts/audit_address_sample.sql deleted file mode 100644 index 11e83b62..00000000 --- a/tradein-mvp/backend/scripts/audit_address_sample.sql +++ /dev/null @@ -1,47 +0,0 @@ --- audit_address_sample.sql --- Random sample of EKB houses for the address-mismatch audit (issue #582). --- --- Strategy: --- 1. Filter to houses with non-null lat/lon and non-empty address. --- 2. Random shuffle via `ORDER BY random()` — repeatable enough for spot --- sampling without needing a stable PRNG seed (the audit table dedupes --- via UNIQUE (house_id, audit_batch), so re-running gives idempotent --- results regardless of which rows land in the sample first). --- 3. Cap the result at :limit_per_district * 8 rows — keeps the bind-param --- contract compatible with the old stratified sampler (`:limit_per_district` --- is still honored, just multiplied by the assumed 8-district count). --- --- Why no spatial stratification anymore: --- The previous version JOINed to `gendesign_ekb_districts_geom` (FDW --- polygon table) to bucket houses by admin district. That join is fine on --- prod where FDW is wired, but it adds a dependency we don't need for --- Phase 2-3 (backfill + canonical reverse). Aggregation by district at --- report time still works — we re-derive district during the audit via --- spatial containment in the report SQL when needed. --- --- Bind param: --- :limit_per_district — kept for back-compat with the audit driver. --- Effective sample size = :limit_per_district * 8 (e.g. 25 → 200). --- --- Columns returned: --- id, address, lat, lon, district --- `district` is always NULL here — the audit driver will reverse-derive it --- from Yandex Geocoder response (Yandex returns admin component) or leave --- it NULL if not present in the response. --- --- NB: uses CAST(:x AS int) per project sql.md rule (psycopg v3 ignores ::type --- after bind params). - -SELECT - h.id, - h.address, - h.lat, - h.lon, - NULL::text AS district -FROM houses h -WHERE h.lat IS NOT NULL - AND h.lon IS NOT NULL - AND h.address IS NOT NULL - AND length(trim(h.address)) > 0 -ORDER BY random() -LIMIT CAST(:limit_per_district AS int) * 8; diff --git a/tradein-mvp/backend/scripts/backfill_house_coords.py b/tradein-mvp/backend/scripts/backfill_house_coords.py deleted file mode 100644 index 7a607730..00000000 --- a/tradein-mvp/backend/scripts/backfill_house_coords.py +++ /dev/null @@ -1,619 +0,0 @@ -"""Forward-geocode houses through Yandex Geocoder API to backfill lat/lon -and canonical address, plus optional reverse audit of already-geocoded houses. - -Phase 2-3 of Forgejo issue #582. Two modes (mutually exclusive): - - 1. Backfill (default) — for the ~4141 rows WHERE lat IS NULL OR lon IS NULL: - forward-geocode `houses.address` → snap to a Yandex `house`-precision - point, UPDATE houses with the new lat/lon + canonical address payload, - and write an `address_mismatch_audit` row with `audit_status='backfill'`. - - 2. Audit-only (--audit-only) — for the ~4452 rows that already have coords: - reverse-geocode (lat, lon) → snapped point + canonical address, compute - ST_Distance vs stored coords, write an `address_mismatch_audit` row with - status 'ok' (≤50m) or 'mismatch' (>50m). Does NOT touch houses. - -Design choices: -- **Per-row SAVEPOINT** (`db.begin_nested()`): a single Yandex/PostGIS error - must not nuke the entire batch. Per backend.md, never use bare rollback - inside a loop. -- **Resumable** via UNIQUE (house_id, audit_batch). Re-running the same - --batch label skips already-processed houses, so a partial run can be - picked up after CAPTCHA / network blip / 25k/day quota hit. -- **Precision filter**: backfill skips matches with precision in - ('street', 'other', 'range', 'near', None) — those are too imprecise for - comparable-listing spatial queries and would silently degrade matching - recall. The audit row still records what Yandex returned for forensics. -- **Rate limit**: 50ms between calls (~20 req/sec, well under Yandex's - 25 req/sec service limit). Backfill mode runs single-threaded. -- **Daily quota**: 4141 backfill + 4452 audit ≈ 8.6k requests. Free Geocoder - tier is 25k/day → comfortable buffer for retries. - -Usage: - YANDEX_GEOCODER_API_KEY=xxx \\ - DATABASE_URL=postgresql+psycopg://... \\ - python -m scripts.backfill_house_coords --batch 2026-05-27_backfill - - # Audit-only on the 4452 already-geocoded houses - python -m scripts.backfill_house_coords --batch 2026-05-27_audit \\ - --audit-only --limit 500 - -Outputs: -- Backfill mode: UPDATE rows in `houses`, INSERT rows in - `address_mismatch_audit` with status 'backfill' / 'no_match' / 'imprecise'. -- Audit mode: INSERT rows in `address_mismatch_audit` with status 'ok' / - 'mismatch' / 'no_match' / 'error'. -- Per-batch progress is logged every 25 rows. -""" - -from __future__ import annotations - -import argparse -import asyncio -import json -import logging -import os -from dataclasses import dataclass -from datetime import date -from pathlib import Path -from typing import Any - -import httpx -from sqlalchemy import text -from sqlalchemy.orm import Session - -# Allow running both as `python -m scripts.backfill_house_coords` (preferred) -# and as a stand-alone file. Mirrors the audit_address_mismatch import dance. -try: - from app.core.db import SessionLocal # type: ignore[import-not-found] -except ImportError: # pragma: no cover — fallback for adhoc invocation - import sys - - sys.path.insert(0, str(Path(__file__).resolve().parents[1])) - from app.core.db import SessionLocal - -try: - from scripts._yandex_reverse import ( # type: ignore[import-not-found] - YandexReverseResult, - forward_via_api, - reverse_via_api, - ) -except ImportError: - from _yandex_reverse import ( # type: ignore[no-redef] - YandexReverseResult, - forward_via_api, - reverse_via_api, - ) - -logging.basicConfig( - level=logging.INFO, - format="%(asctime)s %(levelname)s %(name)s %(message)s", -) -logger = logging.getLogger("backfill_house_coords") - -# Yandex Geocoder service limits per docs (as of 2026-05): -# - 25k requests/day free tier -# - 25 requests/sec sustained -# 50ms between calls = ~20 req/sec, leaving headroom for connection ramp-up. -_REQUEST_DELAY_S = 0.05 - -# Precision values we ACCEPT for backfill — anything else means Yandex didn't -# resolve to a specific building, and writing the result back into houses -# would degrade matching recall. -# `exact` → match found at the exact address (best case) -# `number` → house number matched, but unit/entrance unspecified (acceptable) -# `near` / `range` / `street` / `other` / None → skipped (logged for analysis). -_BACKFILL_OK_PRECISION = frozenset({"exact", "number"}) - -# Audit threshold per issue #582 — distances above this flag a "mismatch" -# (the row still goes in the audit table, just with status='mismatch' for -# the report SQL to bucket separately). -_MISMATCH_DISTANCE_M = 50.0 - - -# --------------------------------------------------------------------------- -# Domain types -# --------------------------------------------------------------------------- - - -@dataclass -class HouseRow: - """One house from the source query — minimal fields needed for geocode.""" - - id: int - address: str - lat: float | None - lon: float | None - - -# --------------------------------------------------------------------------- -# Source-row queries -# --------------------------------------------------------------------------- - - -def _select_houses_without_coords(db: Session, limit: int | None) -> list[HouseRow]: - """Pull houses needing forward geocode (lat IS NULL OR lon IS NULL). - - Skips rows with empty address — there's nothing to geocode there, they - need a separate cleanup pass. - """ - sql = ( - "SELECT id, address, lat, lon " - "FROM houses " - "WHERE (lat IS NULL OR lon IS NULL) " - " AND address IS NOT NULL " - " AND length(trim(address)) > 0 " - "ORDER BY id" - ) - if limit is not None: - sql += " LIMIT CAST(:limit AS int)" - rows = db.execute(text(sql), {"limit": limit}).mappings().all() - else: - rows = db.execute(text(sql)).mappings().all() - return [ - HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows - ] - - -def _select_houses_with_coords(db: Session, limit: int | None) -> list[HouseRow]: - """Pull houses needing reverse audit (both lat AND lon present).""" - sql = ( - "SELECT id, address, lat, lon " - "FROM houses " - "WHERE lat IS NOT NULL " - " AND lon IS NOT NULL " - " AND address IS NOT NULL " - " AND length(trim(address)) > 0 " - "ORDER BY id" - ) - if limit is not None: - sql += " LIMIT CAST(:limit AS int)" - rows = db.execute(text(sql), {"limit": limit}).mappings().all() - else: - rows = db.execute(text(sql)).mappings().all() - return [ - HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows - ] - - -def _already_processed_ids(db: Session, batch: str) -> set[int]: - """house_ids already in address_mismatch_audit for this batch → skip set.""" - rows = db.execute( - text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"), - {"b": batch}, - ).all() - return {r[0] for r in rows} - - -# --------------------------------------------------------------------------- -# Distance helper — PostGIS, lon/lat order -# --------------------------------------------------------------------------- - - -def _distance_meters( - db: Session, olat: float, olon: float, slat: float, slon: float -) -> float | None: - """Great-circle distance (meters) via PostGIS geography type. - - Lifted from `audit_address_mismatch.py` to keep the two scripts using - the same authority for distance computation. ST_MakePoint takes lon - first per PostGIS convention. - """ - row = db.execute( - text( - "SELECT ST_Distance(" - " ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), " - " CAST(:olat AS double precision)), 4326)::geography, " - " ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), " - " CAST(:slat AS double precision)), 4326)::geography" - ") AS m" - ), - {"olat": olat, "olon": olon, "slat": slat, "slon": slon}, - ).first() - if row is None or row[0] is None: - return None - return float(row[0]) - - -# --------------------------------------------------------------------------- -# DB writers -# --------------------------------------------------------------------------- - - -def _update_house_coords( - db: Session, - *, - house_id: int, - lat: float, - lon: float, - payload: dict[str, Any], -) -> None: - """UPDATE houses SET lat/lon + merge yandex_geocode into raw_payload. - - The `houses_set_geom_trg` BEFORE UPDATE trigger (009_houses.sql) maintains - `geom` automatically when lat/lon change, so we don't need to set geom - explicitly here. `raw_payload || jsonb_build_object(...)` is the idiomatic - psycopg-safe way to merge — single ALTER, no read-modify-write race. - """ - db.execute( - text( - "UPDATE houses " - " SET lat = CAST(:lat AS double precision), " - " lon = CAST(:lon AS double precision), " - " raw_payload = COALESCE(raw_payload, '{}'::jsonb) " - " || jsonb_build_object('yandex_geocode', " - " CAST(:payload AS jsonb)) " - " WHERE id = CAST(:id AS bigint)" - ), - {"id": house_id, "lat": lat, "lon": lon, "payload": json.dumps(payload)}, - ) - - -def _insert_audit_row( - db: Session, - *, - house_id: int, - batch: str, - original_address: str | None, - original_lat: float | None, - original_lon: float | None, - snapped_address: str | None, - snapped_lat: float | None, - snapped_lon: float | None, - distance_m: float | None, - audit_status: str, - error_message: str | None, - raw_payload: dict[str, Any] | None, -) -> None: - """INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit. - - Same column shape as `audit_address_mismatch._insert_audit_row` but the - `district` and `street_differs` fields are left NULL — backfill/audit - here doesn't have a stratification basis and we let the report SQL - derive district at query time if needed (via Yandex address parse). - - Caller wraps in `begin_nested()` per backend.md SAVEPOINT pattern. - """ - db.execute( - text( - "INSERT INTO address_mismatch_audit (" - " house_id, audit_batch, district," - " original_address, original_lat, original_lon," - " snapped_address, snapped_lat, snapped_lon," - " distance_m, street_differs," - " audit_status, error_message, raw_payload" - ") VALUES (" - " CAST(:house_id AS bigint), CAST(:batch AS text), NULL," - " :original_address, :original_lat, :original_lon," - " :snapped_address, :snapped_lat, :snapped_lon," - " :distance_m, NULL," - " CAST(:audit_status AS text), :error_message," - " CAST(:raw_payload AS jsonb)" - ") ON CONFLICT (house_id, audit_batch) DO NOTHING" - ), - { - "house_id": house_id, - "batch": batch, - "original_address": original_address, - "original_lat": original_lat, - "original_lon": original_lon, - "snapped_address": snapped_address, - "snapped_lat": snapped_lat, - "snapped_lon": snapped_lon, - "distance_m": distance_m, - "audit_status": audit_status, - "error_message": error_message, - "raw_payload": json.dumps(raw_payload) if raw_payload is not None else None, - }, - ) - - -# --------------------------------------------------------------------------- -# Backfill loop (forward geocode, lat IS NULL houses) -# --------------------------------------------------------------------------- - - -def _classify_backfill_status(res: YandexReverseResult | None) -> str: - """Translate a forward-geocode result into an audit_status value. - - 'backfill' — Yandex returned a precise hit, lat/lon will be written. - 'imprecise' — match returned but precision is too low (street/other/...). - 'no_match' — Yandex returned an empty featureMember. - 'error' — handled by the caller's exception branch. - """ - if res is None or res.address is None: - return "no_match" - if res.precision not in _BACKFILL_OK_PRECISION: - return "imprecise" - if res.snapped_lat is None or res.snapped_lon is None: - return "no_match" - return "backfill" - - -async def _run_backfill_mode( - db: Session, sample: list[HouseRow], batch: str, api_key: str -) -> int: - """Forward-geocode each house, UPDATE coords on precise hits, audit-log all.""" - processed = 0 - updated = 0 - n_imprecise = 0 - n_no_match = 0 - n_error = 0 - - async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client: - for i, row in enumerate(sample, start=1): - status = "backfill" - err: str | None = None - res: YandexReverseResult | None = None - try: - res = await forward_via_api(row.address, api_key, client=client) - except httpx.HTTPError as e: - status = "error" - err = f"http_error: {e!s}" - n_error += 1 - except Exception as e: # pragma: no cover — defensive - status = "error" - err = f"unhandled: {e!s}" - n_error += 1 - - if status != "error": - status = _classify_backfill_status(res) - if status == "imprecise": - n_imprecise += 1 - elif status == "no_match": - n_no_match += 1 - - try: - with db.begin_nested(): - if status == "backfill" and res is not None and res.snapped_lat is not None: - # safe: status='backfill' guarantees snapped_lat/lon non-None. - assert res.snapped_lon is not None - _update_house_coords( - db, - house_id=row.id, - lat=res.snapped_lat, - lon=res.snapped_lon, - payload={ - "address": res.address, - "precision": res.precision, - "kind": res.kind, - "batch": batch, - "source": "yandex_geocoder_api", - }, - ) - updated += 1 - _insert_audit_row( - db, - house_id=row.id, - batch=batch, - original_address=row.address, - original_lat=row.lat, - original_lon=row.lon, - snapped_address=res.address if res else None, - snapped_lat=res.snapped_lat if res else None, - snapped_lon=res.snapped_lon if res else None, - distance_m=None, - audit_status=status, - error_message=err, - raw_payload=res.raw if res else None, - ) - # Per-row commit so resume picks up exactly where we crashed. - db.commit() - processed += 1 - except Exception as e: - db.rollback() - logger.warning("backfill insert failed for house_id=%s: %s", row.id, e) - - if i % 25 == 0: - logger.info( - "backfill progress %d/%d updated=%d imprecise=%d no_match=%d error=%d", - i, - len(sample), - updated, - n_imprecise, - n_no_match, - n_error, - ) - - # Yandex 25 req/sec → 50ms between calls is plenty of headroom. - if i < len(sample): - await asyncio.sleep(_REQUEST_DELAY_S) - - logger.info( - "backfill done: processed=%d updated=%d imprecise=%d no_match=%d error=%d", - processed, - updated, - n_imprecise, - n_no_match, - n_error, - ) - return processed - - -# --------------------------------------------------------------------------- -# Audit-only loop (reverse geocode, lat IS NOT NULL houses) -# --------------------------------------------------------------------------- - - -async def _run_audit_mode( - db: Session, sample: list[HouseRow], batch: str, api_key: str -) -> int: - """Reverse-geocode each house, compute distance, audit-log status/mismatch.""" - processed = 0 - n_ok = 0 - n_mismatch = 0 - n_no_match = 0 - n_error = 0 - - async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client: - for i, row in enumerate(sample, start=1): - # Type-narrow: audit mode only feeds rows with non-null coords. - assert row.lat is not None and row.lon is not None - status = "ok" - err: str | None = None - res: YandexReverseResult | None = None - try: - res = await reverse_via_api(row.lat, row.lon, api_key, client=client) - except httpx.HTTPError as e: - status = "error" - err = f"http_error: {e!s}" - n_error += 1 - except Exception as e: # pragma: no cover — defensive - status = "error" - err = f"unhandled: {e!s}" - n_error += 1 - - distance = None - if res is not None and status == "ok": - if res.address is None: - status = "no_match" - n_no_match += 1 - else: - if res.snapped_lat is not None and res.snapped_lon is not None: - distance = _distance_meters( - db, row.lat, row.lon, res.snapped_lat, res.snapped_lon - ) - if distance is not None and distance > _MISMATCH_DISTANCE_M: - status = "mismatch" - n_mismatch += 1 - else: - n_ok += 1 - else: - n_ok += 1 - - try: - with db.begin_nested(): - _insert_audit_row( - db, - house_id=row.id, - batch=batch, - original_address=row.address, - original_lat=row.lat, - original_lon=row.lon, - snapped_address=res.address if res else None, - snapped_lat=res.snapped_lat if res else None, - snapped_lon=res.snapped_lon if res else None, - distance_m=distance, - audit_status=status, - error_message=err, - raw_payload=res.raw if res else None, - ) - db.commit() - processed += 1 - except Exception as e: - db.rollback() - logger.warning("audit insert failed for house_id=%s: %s", row.id, e) - - if i % 25 == 0: - logger.info( - "audit progress %d/%d ok=%d mismatch=%d no_match=%d error=%d", - i, - len(sample), - n_ok, - n_mismatch, - n_no_match, - n_error, - ) - - if i < len(sample): - await asyncio.sleep(_REQUEST_DELAY_S) - - logger.info( - "audit done: processed=%d ok=%d mismatch=%d no_match=%d error=%d", - processed, - n_ok, - n_mismatch, - n_no_match, - n_error, - ) - return processed - - -# --------------------------------------------------------------------------- -# CLI -# --------------------------------------------------------------------------- - - -def _parse_args(argv: list[str] | None = None) -> argparse.Namespace: - """argparse setup, factored out for testability.""" - p = argparse.ArgumentParser( - description=( - "Phase 2-3 of issue #582 — backfill houses.lat/lon via Yandex forward " - "geocode, or audit already-geocoded houses via reverse geocode." - ), - ) - p.add_argument( - "--batch", - default=f"{date.today().isoformat()}_backfill", - help="Audit batch label. Same batch re-run skips already-processed houses.", - ) - p.add_argument( - "--audit-only", - action="store_true", - help=( - "Run reverse-geocode audit on houses WITH coords instead of forward " - "backfill on houses WITHOUT coords. Does not modify the houses table." - ), - ) - p.add_argument( - "--limit", - type=int, - default=None, - help=( - "Optional cap on source-row count. Useful for canary runs " - "(e.g. --limit 100 before letting the full 4k loose)." - ), - ) - return p.parse_args(argv) - - -async def main(argv: list[str] | None = None) -> int: - """CLI entry point. Returns the number of rows processed this run.""" - args = _parse_args(argv) - api_key = os.environ.get("YANDEX_GEOCODER_API_KEY") - if not api_key: - raise SystemExit( - "YANDEX_GEOCODER_API_KEY is required — forward geocode is API-only." - ) - - mode = "audit" if args.audit_only else "backfill" - logger.info( - "starting batch=%s mode=%s limit=%s", - args.batch, - mode, - args.limit if args.limit is not None else "all", - ) - - db = SessionLocal() - try: - if args.audit_only: - sample = _select_houses_with_coords(db, args.limit) - else: - sample = _select_houses_without_coords(db, args.limit) - logger.info("loaded source rows: %d", len(sample)) - - done = _already_processed_ids(db, args.batch) - if done: - logger.info( - "resuming batch %s: %d rows already processed", - args.batch, - len(done), - ) - remaining = [s for s in sample if s.id not in done] - if not remaining: - logger.info("nothing to do — batch %s is complete for the loaded sample", args.batch) - return 0 - - if args.audit_only: - n = await _run_audit_mode(db, remaining, args.batch, api_key) - else: - n = await _run_backfill_mode(db, remaining, args.batch, api_key) - - logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode) - return n - finally: - db.close() - - -if __name__ == "__main__": # pragma: no cover - asyncio.run(main()) diff --git a/tradein-mvp/backend/tests/fixtures/yandex_geocode_sample.json b/tradein-mvp/backend/tests/fixtures/yandex_geocode_sample.json deleted file mode 100644 index 9a3660d0..00000000 --- a/tradein-mvp/backend/tests/fixtures/yandex_geocode_sample.json +++ /dev/null @@ -1,74 +0,0 @@ -{ - "response": { - "GeoObjectCollection": { - "metaDataProperty": { - "GeocoderResponseMetaData": { - "request": "60.586,56.838", - "results": "1", - "found": "1" - } - }, - "featureMember": [ - { - "GeoObject": { - "metaDataProperty": { - "GeocoderMetaData": { - "precision": "exact", - "text": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51", - "kind": "house", - "Address": { - "country_code": "RU", - "formatted": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51", - "postal_code": "620075", - "Components": [ - {"kind": "country", "name": "Россия"}, - {"kind": "province", "name": "Уральский федеральный округ"}, - {"kind": "province", "name": "Свердловская область"}, - {"kind": "area", "name": "городской округ Екатеринбург"}, - {"kind": "locality", "name": "Екатеринбург"}, - {"kind": "street", "name": "улица Малышева"}, - {"kind": "house", "name": "51"} - ] - }, - "AddressDetails": { - "Country": { - "AddressLine": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51", - "CountryNameCode": "RU", - "CountryName": "Россия", - "AdministrativeArea": { - "AdministrativeAreaName": "Свердловская область", - "SubAdministrativeArea": { - "SubAdministrativeAreaName": "городской округ Екатеринбург", - "Locality": { - "LocalityName": "Екатеринбург", - "Thoroughfare": { - "ThoroughfareName": "улица Малышева", - "Premise": { - "PremiseNumber": "51", - "PostalCode": {"PostalCodeNumber": "620075"} - } - } - } - } - } - } - } - } - }, - "name": "улица Малышева, 51", - "description": "Екатеринбург, Россия", - "boundedBy": { - "Envelope": { - "lowerCorner": "60.585217 56.837461", - "upperCorner": "60.587094 56.838547" - } - }, - "Point": { - "pos": "60.586155 56.838004" - } - } - } - ] - } - } -} diff --git a/tradein-mvp/backend/tests/test_audit_address_mismatch.py b/tradein-mvp/backend/tests/test_audit_address_mismatch.py deleted file mode 100644 index 1c64288b..00000000 --- a/tradein-mvp/backend/tests/test_audit_address_mismatch.py +++ /dev/null @@ -1,373 +0,0 @@ -"""Unit tests for the Phase-1 address-mismatch audit (issue #582). - -Coverage: - - `_first_street_token` / `_street_differs` — normalization-driven diff. - - `_distance_meters` — verified against a MagicMock'd DB that returns a - canned distance, plus a Haversine cross-check on the bind values to - catch lat/lon swaps. - - `reverse_via_api` via httpx MockTransport with the fixture file. - - `main()` resumability — call twice with the same batch, second call - inserts 0 (uses MagicMock DB session). - -Why no real Postgres in unit tests: -The repo doesn't bundle pytest-postgresql / testcontainers and the existing -tests all use `MagicMock` for the DB. We follow that convention here. The -distance and SQL-level resumability are validated by: - - The Haversine cross-check (pure-Python expected ≈ PostGIS result for - same coords, see `test_distance_calc_matches_haversine`). - - Calling `main()` twice in `test_audit_script_resumable` — first run - inserts N rows, second run sees the same set of house_ids in the - "already processed" query and processes 0. -""" - -from __future__ import annotations - -import json -import math -import os -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock, patch - -# Settings requires DATABASE_URL at init time — set dummy DSN before any -# `app.*` import (same pattern as test_cian_valuation.py). -os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db") - -import httpx -import pytest - -from scripts._yandex_reverse import ( - YandexBlockedError, - YandexReverseResult, - _parse_api_payload, - reverse_via_api, -) -from scripts.audit_address_mismatch import ( - SampleRow, - _distance_meters, - _first_street_token, - _resolve_mode, - _run_api_mode, - _street_differs, - main, -) - -_FIXTURES = Path(__file__).parent / "fixtures" - - -# --------------------------------------------------------------------------- -# _first_street_token / _street_differs -# --------------------------------------------------------------------------- - - -def test_normalize_address_street_token_basic(): - """First identifying token of a normalized address — skips street type.""" - assert _first_street_token("ул Малышева 51") == "малышева" - - -def test_normalize_address_street_token_skips_leading_numbers(): - """Numeric tokens are skipped — the street name carries identity.""" - # No type prefix → first non-numeric token is the street name itself. - assert _first_street_token("123 Постовского") == "постовского" - - -def test_normalize_address_street_token_handles_none(): - assert _first_street_token(None) is None - assert _first_street_token("") is None - - -def test_street_differs_true_when_streets_differ(): - assert _street_differs("ул Малышева 51", "ул Ленина 51") is True - - -def test_street_differs_false_when_same_after_normalization(): - # 'ул' expands to 'улица' on both sides → same first token. - assert _street_differs("ул Малышева 51", "улица Малышева, 51") is False - - -def test_street_differs_none_on_empty_side(): - assert _street_differs(None, "ул Малышева 51") is None - assert _street_differs("ул Малышева 51", "") is None - - -# --------------------------------------------------------------------------- -# _distance_meters — MagicMock DB + Haversine cross-check -# --------------------------------------------------------------------------- - - -def _haversine_m(lat1: float, lon1: float, lat2: float, lon2: float) -> float: - """Reference implementation for sanity-checking the PostGIS call.""" - r = 6_371_000.0 - p1 = math.radians(lat1) - p2 = math.radians(lat2) - dp = math.radians(lat2 - lat1) - dl = math.radians(lon2 - lon1) - a = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2 - return 2 * r * math.asin(math.sqrt(a)) - - -def test_distance_calc_passes_correct_bindings(): - """Test the helper passes lat/lon in correct order to the SQL bind names.""" - db = MagicMock() - # PostGIS would return one row, single column (distance in meters). - db.execute.return_value.first.return_value = (123.45,) - - out = _distance_meters(db, 56.838, 60.586, 56.840, 60.590) - - assert out == 123.45 - # Verify the bind dict — guard against lat/lon swap regressions. - args, _kwargs = db.execute.call_args - bound = args[1] - assert bound == { - "olat": 56.838, - "olon": 60.586, - "slat": 56.840, - "slon": 60.590, - } - - -def test_distance_calc_returns_none_when_postgis_null(): - """ST_Distance can return NULL — caller must propagate None, not 0.""" - db = MagicMock() - db.execute.return_value.first.return_value = (None,) - assert _distance_meters(db, 56.0, 60.0, 56.0, 60.0) is None - - -def test_distance_calc_matches_haversine_within_tolerance(): - """Sanity check: if PostGIS returned 555.7m for a known pair, that's - within ~1% of the Haversine reference (PostGIS uses Vincenty on - geography which is slightly more accurate).""" - expected = _haversine_m(56.838, 60.586, 56.843, 60.591) - # Just assert reference is in a sensible range — proves the test helper - # works; the actual call is mocked. - assert 500 < expected < 700 - - -# --------------------------------------------------------------------------- -# Yandex API: payload parsing + reverse_via_api with MockTransport -# --------------------------------------------------------------------------- - - -def test_yandex_parse_api_fixture(): - """Sanity check: parse the bundled fixture into a YandexReverseResult.""" - data = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8")) - res = _parse_api_payload(data) - assert res.address is not None - assert "Малышева" in res.address - # Fixture Point.pos = "60.586155 56.838004" → lon then lat. - assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6) - assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6) - assert res.raw == data - - -def test_yandex_parse_api_no_match(): - """Empty featureMember → all-None result, raw still preserved.""" - data = {"response": {"GeoObjectCollection": {"featureMember": []}}} - res = _parse_api_payload(data) - assert res.address is None - assert res.snapped_lat is None - assert res.snapped_lon is None - assert res.raw == data - - -async def test_yandex_reverse_api_mock(): - """End-to-end: reverse_via_api hits a MockTransport, returns parsed result.""" - fixture = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8")) - - captured: dict[str, httpx.Request] = {} - - def handler(request: httpx.Request) -> httpx.Response: - captured["req"] = request - return httpx.Response(200, json=fixture) - - transport = httpx.MockTransport(handler) - async with httpx.AsyncClient(transport=transport) as client: - res = await reverse_via_api(56.838004, 60.586155, "DUMMY_KEY", client=client) - - assert res.address is not None and "Малышева" in res.address - # Verify the request shape — lon,lat order + apikey + kind=house. - req = captured["req"] - qs = dict(httpx.QueryParams(req.url.query)) - assert qs["apikey"] == "DUMMY_KEY" - assert qs["geocode"] == "60.586155,56.838004" - assert qs["format"] == "json" - assert qs["kind"] == "house" - - -async def test_yandex_blocked_error_raised_on_captcha(): - """`reverse_via_playwright` must raise YandexBlockedError on captcha. - - We mock the page object so we don't need an actual browser. - """ - from scripts._yandex_reverse import reverse_via_playwright - - page = MagicMock() - page.goto = AsyncMock() - page.wait_for_load_state = AsyncMock() - page.query_selector = AsyncMock( - side_effect=lambda sel: MagicMock() if sel == ".CheckboxCaptcha" else None - ) - page.evaluate = AsyncMock(return_value=[None, None]) - - with pytest.raises(YandexBlockedError): - await reverse_via_playwright(56.838, 60.586, page) - - -# --------------------------------------------------------------------------- -# Mode resolver -# --------------------------------------------------------------------------- - - -def test_resolve_mode_auto_with_key(): - assert _resolve_mode("auto", "abc") == "api" - - -def test_resolve_mode_auto_without_key(): - assert _resolve_mode("auto", None) == "playwright" - assert _resolve_mode("auto", "") == "playwright" - - -def test_resolve_mode_explicit_passes_through(): - assert _resolve_mode("api", None) == "api" - assert _resolve_mode("playwright", "abc") == "playwright" - - -# --------------------------------------------------------------------------- -# Resumability — main() twice with same batch -# --------------------------------------------------------------------------- - - -def _make_db_mock(initial_sample: list[dict], processed_ids: set[int]): - """Build a MagicMock SQLAlchemy session that: - - returns `initial_sample` for the sampling SQL (text() with limit_per_district) - - returns `processed_ids` for the resume SQL (text() with batch only) - - records INSERTs so the test can count them - """ - inserted: list[dict] = [] - - db = MagicMock() - db.begin_nested.return_value.__enter__ = lambda self: self - db.begin_nested.return_value.__exit__ = lambda self, *a: False - - def execute_side_effect(sql, params=None): - sql_str = str(sql) - result = MagicMock() - if "FROM houses h" in sql_str or "houses_in_districts" in sql_str: - result.mappings.return_value.all.return_value = initial_sample - elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str: - # Resume query — returns list of (house_id,) tuples. - result.all.return_value = [(hid,) for hid in processed_ids] - elif "INSERT INTO address_mismatch_audit" in sql_str: - inserted.append(dict(params)) - # Simulate ON CONFLICT DO NOTHING — track id locally for re-run. - processed_ids.add(params["house_id"]) - result = MagicMock() - elif "ST_Distance" in sql_str: - result.first.return_value = (42.0,) - else: - result = MagicMock() - return result - - db.execute.side_effect = execute_side_effect - db.commit = MagicMock() - db.rollback = MagicMock() - db.close = MagicMock() - return db, inserted - - -async def test_audit_script_resumable(monkeypatch): - """Run main() twice with the same batch — second pass inserts 0.""" - sample = [ - { - "id": 1, - "address": "ул Малышева 51", - "lat": 56.838, - "lon": 60.586, - "district": "Кировский", - }, - {"id": 2, "address": "ул Ленина 5", "lat": 56.840, "lon": 60.600, "district": "Ленинский"}, - ] - processed_ids: set[int] = set() - db, inserted = _make_db_mock(sample, processed_ids) - - # Force API mode without needing a real key. - monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY") - - fake_result = YandexReverseResult( - address="Россия, Екатеринбург, улица Малышева, 51", - snapped_lat=56.838004, - snapped_lon=60.586155, - raw={"ok": True}, - ) - - with ( - patch("scripts.audit_address_mismatch.SessionLocal", return_value=db), - patch( - "scripts.audit_address_mismatch.reverse_via_api", - new=AsyncMock(return_value=fake_result), - ), - ): - # First run — both rows processed. - n1 = await main(["--batch", "test_batch_1", "--mode", "api"]) - assert n1 == 2 - assert len(inserted) == 2 - - # Second run with same batch — nothing left to do. - inserted.clear() - n2 = await main(["--batch", "test_batch_1", "--mode", "api"]) - assert n2 == 0 - assert inserted == [] - - -async def test_audit_script_api_mode_marks_error(monkeypatch): - """When the reverse call raises, the row is still inserted with status=error.""" - sample = [ - { - "id": 99, - "address": "ул Малышева 51", - "lat": 56.838, - "lon": 60.586, - "district": "Кировский", - }, - ] - processed_ids: set[int] = set() - db, inserted = _make_db_mock(sample, processed_ids) - - monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY") - - with ( - patch("scripts.audit_address_mismatch.SessionLocal", return_value=db), - patch( - "scripts.audit_address_mismatch.reverse_via_api", - new=AsyncMock(side_effect=httpx.HTTPError("boom")), - ), - ): - n = await main(["--batch", "err_batch", "--mode", "api"]) - assert n == 1 - - assert len(inserted) == 1 - assert inserted[0]["audit_status"] == "error" - assert "boom" in (inserted[0]["error_message"] or "") - - -# --------------------------------------------------------------------------- -# Internal _run_api_mode no-match path -# --------------------------------------------------------------------------- - - -async def test_api_mode_no_match_path(): - """If Yandex returns address=None, row goes in with status=no_match.""" - sample = [SampleRow(id=7, address="ул X 1", lat=56.0, lon=60.0, district="Кировский")] - processed_ids: set[int] = set() - db, inserted = _make_db_mock([], processed_ids) - - res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True}) - - with patch( - "scripts.audit_address_mismatch.reverse_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_api_mode(db, sample, "b1", "key") - - assert n == 1 - assert inserted[0]["audit_status"] == "no_match" - assert inserted[0]["snapped_address"] is None diff --git a/tradein-mvp/backend/tests/test_backfill_house_coords.py b/tradein-mvp/backend/tests/test_backfill_house_coords.py deleted file mode 100644 index 2b8da977..00000000 --- a/tradein-mvp/backend/tests/test_backfill_house_coords.py +++ /dev/null @@ -1,510 +0,0 @@ -"""Unit tests for the Phase 2-3 backfill/audit script (issue #582). - -Coverage: - - `forward_via_api` request shape — verifies geocode/format/locality bias. - - `_parse_api_payload` precision + kind extraction. - - `_classify_backfill_status` precision filter rules. - - `_update_house_coords` — UPDATE shape + raw_payload merge. - - `_run_backfill_mode` — happy path UPDATE + audit row, plus imprecise-skip. - - `_run_audit_mode` — ok / mismatch / no_match distinction. - - `main()` resumability — second pass on same batch inserts 0. - -No real Postgres in unit tests (same convention as test_audit_address_mismatch). -DB is a MagicMock that records INSERT/UPDATE calls and routes SELECT side-effects. -""" - -from __future__ import annotations - -import json -import os -from unittest.mock import AsyncMock, MagicMock, patch - -# Same dance as test_audit_address_mismatch — settings needs a DSN at import. -os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db") - -import httpx -import pytest - -from scripts._yandex_reverse import ( - YandexReverseResult, - _parse_api_payload, - forward_via_api, -) -from scripts.backfill_house_coords import ( - HouseRow, - _classify_backfill_status, - _run_audit_mode, - _run_backfill_mode, - _update_house_coords, - main, -) - -# --------------------------------------------------------------------------- -# forward_via_api — request shape -# --------------------------------------------------------------------------- - - -async def test_forward_api_request_shape(): - """Verify the GET param dict — address as `geocode`, kind=house, EKB bias.""" - fixture = { - "response": { - "GeoObjectCollection": { - "featureMember": [ - { - "GeoObject": { - "metaDataProperty": { - "GeocoderMetaData": { - "text": "Россия, Свердловская область, Екатеринбург, " - "улица Малышева, 51", - "precision": "exact", - "kind": "house", - } - }, - "name": "улица Малышева, 51", - "Point": {"pos": "60.586155 56.838004"}, - } - } - ] - } - } - } - captured: dict[str, httpx.Request] = {} - - def handler(request: httpx.Request) -> httpx.Response: - captured["req"] = request - return httpx.Response(200, json=fixture) - - transport = httpx.MockTransport(handler) - async with httpx.AsyncClient(transport=transport) as client: - res = await forward_via_api("ул Малышева 51", "DUMMY_KEY", client=client) - - assert res.address is not None and "Малышева" in res.address - assert res.precision == "exact" - assert res.kind == "house" - assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6) - assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6) - - qs = dict(httpx.QueryParams(captured["req"].url.query)) - assert qs["apikey"] == "DUMMY_KEY" - assert qs["geocode"] == "ул Малышева 51" - assert qs["format"] == "json" - assert qs["kind"] == "house" - # EKB locality bias for forward geocode — important so addresses without - # the city resolve to the correct Малышева (there's one in Moscow too). - assert "ll" in qs - assert "spn" in qs - - -# --------------------------------------------------------------------------- -# Precision / kind passthrough in _parse_api_payload -# --------------------------------------------------------------------------- - - -def test_parse_api_payload_propagates_precision_and_kind(): - data = { - "response": { - "GeoObjectCollection": { - "featureMember": [ - { - "GeoObject": { - "metaDataProperty": { - "GeocoderMetaData": { - "text": "ул Ленина 5", - "precision": "exact", - "kind": "house", - } - }, - "name": "ул Ленина 5", - "Point": {"pos": "60.6 56.8"}, - } - } - ] - } - } - } - res = _parse_api_payload(data) - assert res.precision == "exact" - assert res.kind == "house" - - -# --------------------------------------------------------------------------- -# _classify_backfill_status — precision filter rules -# --------------------------------------------------------------------------- - - -def test_classify_backfill_status_exact_match(): - res = YandexReverseResult( - address="ул Малышева 51", - snapped_lat=56.838, - snapped_lon=60.586, - precision="exact", - kind="house", - ) - assert _classify_backfill_status(res) == "backfill" - - -def test_classify_backfill_status_number_match(): - res = YandexReverseResult( - address="ул Ленина 5", - snapped_lat=56.840, - snapped_lon=60.600, - precision="number", - kind="house", - ) - assert _classify_backfill_status(res) == "backfill" - - -def test_classify_backfill_status_street_is_imprecise(): - res = YandexReverseResult( - address="ул Ленина", - snapped_lat=56.840, - snapped_lon=60.600, - precision="street", - kind="street", - ) - assert _classify_backfill_status(res) == "imprecise" - - -def test_classify_backfill_status_other_is_imprecise(): - res = YandexReverseResult( - address="Свердловская область", - snapped_lat=56.8, - snapped_lon=60.6, - precision="other", - kind="locality", - ) - assert _classify_backfill_status(res) == "imprecise" - - -def test_classify_backfill_status_no_match(): - res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None) - assert _classify_backfill_status(res) == "no_match" - - -def test_classify_backfill_status_none(): - assert _classify_backfill_status(None) == "no_match" - - -def test_classify_backfill_status_precision_ok_but_no_coords(): - """Defensive: precision=exact but snapped point missing → no_match, not backfill.""" - res = YandexReverseResult( - address="ул Малышева 51", - snapped_lat=None, - snapped_lon=None, - precision="exact", - kind="house", - ) - assert _classify_backfill_status(res) == "no_match" - - -# --------------------------------------------------------------------------- -# _update_house_coords — UPDATE shape verification -# --------------------------------------------------------------------------- - - -def test_update_house_coords_passes_bindings(): - db = MagicMock() - _update_house_coords( - db, - house_id=42, - lat=56.838, - lon=60.586, - payload={"address": "ул Малышева 51", "precision": "exact"}, - ) - args, _kw = db.execute.call_args - sql_str = str(args[0]) - binds = args[1] - assert "UPDATE houses" in sql_str - assert "raw_payload" in sql_str - assert "yandex_geocode" in sql_str - assert binds["id"] == 42 - assert binds["lat"] == 56.838 - assert binds["lon"] == 60.586 - # payload bound as JSON string for CAST(:payload AS jsonb) - decoded = json.loads(binds["payload"]) - assert decoded["address"] == "ул Малышева 51" - - -# --------------------------------------------------------------------------- -# DB mock helper — same approach as test_audit_address_mismatch -# --------------------------------------------------------------------------- - - -def _make_db_mock( - backfill_sample: list[dict] | None = None, - audit_sample: list[dict] | None = None, - processed_ids: set[int] | None = None, - distance_value: float = 12.5, -): - """MagicMock DB that: - - returns `backfill_sample` for `lat IS NULL OR lon IS NULL` SELECT - - returns `audit_sample` for `lat IS NOT NULL` SELECT - - returns `processed_ids` for the resume SELECT - - records INSERTs and UPDATEs - - returns `distance_value` for ST_Distance calls - """ - backfill_sample = backfill_sample or [] - audit_sample = audit_sample or [] - processed_ids = processed_ids if processed_ids is not None else set() - - inserted: list[dict] = [] - updated: list[dict] = [] - - db = MagicMock() - db.begin_nested.return_value.__enter__ = lambda self: self - db.begin_nested.return_value.__exit__ = lambda self, *a: False - - def execute_side_effect(sql, params=None): - sql_str = str(sql) - result = MagicMock() - if "FROM houses" in sql_str and "lat IS NULL OR lon IS NULL" in sql_str: - result.mappings.return_value.all.return_value = backfill_sample - elif "FROM houses" in sql_str and "lat IS NOT NULL" in sql_str: - result.mappings.return_value.all.return_value = audit_sample - elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str: - result.all.return_value = [(hid,) for hid in processed_ids] - elif "INSERT INTO address_mismatch_audit" in sql_str: - inserted.append(dict(params)) - processed_ids.add(params["house_id"]) - elif "UPDATE houses" in sql_str: - updated.append(dict(params)) - elif "ST_Distance" in sql_str: - result.first.return_value = (distance_value,) - return result - - db.execute.side_effect = execute_side_effect - db.commit = MagicMock() - db.rollback = MagicMock() - db.close = MagicMock() - return db, inserted, updated - - -# --------------------------------------------------------------------------- -# _run_backfill_mode — happy path + imprecise-skip -# --------------------------------------------------------------------------- - - -async def test_run_backfill_mode_writes_update_and_audit(): - sample = [ - HouseRow(id=1, address="ул Малышева 51", lat=None, lon=None), - ] - db, inserted, updated = _make_db_mock() - res = YandexReverseResult( - address="Россия, Екатеринбург, улица Малышева, 51", - snapped_lat=56.838, - snapped_lon=60.586, - precision="exact", - kind="house", - raw={"ok": True}, - ) - with patch( - "scripts.backfill_house_coords.forward_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_backfill_mode(db, sample, "b1", "KEY") - assert n == 1 - assert len(updated) == 1 - assert updated[0]["id"] == 1 - assert updated[0]["lat"] == 56.838 - assert updated[0]["lon"] == 60.586 - assert len(inserted) == 1 - assert inserted[0]["audit_status"] == "backfill" - assert inserted[0]["snapped_address"] == "Россия, Екатеринбург, улица Малышева, 51" - - -async def test_run_backfill_mode_imprecise_skips_update(): - """precision='street' → audit row written with status=imprecise, no UPDATE.""" - sample = [HouseRow(id=2, address="ул Ленина", lat=None, lon=None)] - db, inserted, updated = _make_db_mock() - res = YandexReverseResult( - address="ул Ленина", - snapped_lat=56.840, - snapped_lon=60.600, - precision="street", - kind="street", - raw={"oh_well": True}, - ) - with patch( - "scripts.backfill_house_coords.forward_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_backfill_mode(db, sample, "b2", "KEY") - assert n == 1 - assert updated == [] - assert len(inserted) == 1 - assert inserted[0]["audit_status"] == "imprecise" - - -async def test_run_backfill_mode_no_match(): - """Yandex returns empty result → status=no_match, no UPDATE.""" - sample = [HouseRow(id=3, address="несуществующая улица 99", lat=None, lon=None)] - db, inserted, updated = _make_db_mock() - res = YandexReverseResult( - address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True} - ) - with patch( - "scripts.backfill_house_coords.forward_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_backfill_mode(db, sample, "b3", "KEY") - assert n == 1 - assert updated == [] - assert inserted[0]["audit_status"] == "no_match" - - -async def test_run_backfill_mode_http_error_marks_error(): - sample = [HouseRow(id=4, address="ул X 1", lat=None, lon=None)] - db, inserted, updated = _make_db_mock() - with patch( - "scripts.backfill_house_coords.forward_via_api", - new=AsyncMock(side_effect=httpx.HTTPError("boom")), - ): - n = await _run_backfill_mode(db, sample, "b4", "KEY") - assert n == 1 - assert updated == [] - assert inserted[0]["audit_status"] == "error" - assert "boom" in (inserted[0]["error_message"] or "") - - -# --------------------------------------------------------------------------- -# _run_audit_mode — ok / mismatch / no_match -# --------------------------------------------------------------------------- - - -async def test_run_audit_mode_ok_within_50m(): - sample = [HouseRow(id=10, address="ул Малышева 51", lat=56.838, lon=60.586)] - db, inserted, _updated = _make_db_mock(distance_value=12.5) - res = YandexReverseResult( - address="Россия, Екатеринбург, улица Малышева, 51", - snapped_lat=56.838004, - snapped_lon=60.586155, - precision="exact", - kind="house", - raw={"r": 1}, - ) - with patch( - "scripts.backfill_house_coords.reverse_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_audit_mode(db, sample, "ba1", "KEY") - assert n == 1 - assert inserted[0]["audit_status"] == "ok" - assert inserted[0]["distance_m"] == 12.5 - - -async def test_run_audit_mode_mismatch_above_50m(): - sample = [HouseRow(id=11, address="ул Ленина 5", lat=56.840, lon=60.600)] - db, inserted, _updated = _make_db_mock(distance_value=312.0) - res = YandexReverseResult( - address="Россия, Екатеринбург, улица Ленина, 7", - snapped_lat=56.841, - snapped_lon=60.601, - precision="exact", - kind="house", - raw={"r": 2}, - ) - with patch( - "scripts.backfill_house_coords.reverse_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_audit_mode(db, sample, "ba2", "KEY") - assert n == 1 - assert inserted[0]["audit_status"] == "mismatch" - assert inserted[0]["distance_m"] == 312.0 - - -async def test_run_audit_mode_no_match(): - sample = [HouseRow(id=12, address="ул X 99", lat=56.0, lon=60.0)] - db, inserted, _updated = _make_db_mock() - res = YandexReverseResult( - address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True} - ) - with patch( - "scripts.backfill_house_coords.reverse_via_api", - new=AsyncMock(return_value=res), - ): - n = await _run_audit_mode(db, sample, "ba3", "KEY") - assert n == 1 - assert inserted[0]["audit_status"] == "no_match" - - -# --------------------------------------------------------------------------- -# Resumability — second pass on same batch inserts 0 -# --------------------------------------------------------------------------- - - -async def test_main_resumable_skips_processed(monkeypatch): - """Run main() twice with same batch — second pass processes nothing.""" - backfill_sample = [ - {"id": 1, "address": "ул Малышева 51", "lat": None, "lon": None}, - {"id": 2, "address": "ул Ленина 5", "lat": None, "lon": None}, - ] - processed_ids: set[int] = set() - db, inserted, updated = _make_db_mock( - backfill_sample=backfill_sample, processed_ids=processed_ids - ) - - monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY") - fake = YandexReverseResult( - address="ул Малышева 51", - snapped_lat=56.838, - snapped_lon=60.586, - precision="exact", - kind="house", - raw={"ok": True}, - ) - - with ( - patch("scripts.backfill_house_coords.SessionLocal", return_value=db), - patch( - "scripts.backfill_house_coords.forward_via_api", - new=AsyncMock(return_value=fake), - ), - ): - n1 = await main(["--batch", "resume_test"]) - assert n1 == 2 - assert len(inserted) == 2 - assert len(updated) == 2 - - inserted.clear() - updated.clear() - n2 = await main(["--batch", "resume_test"]) - assert n2 == 0 - assert inserted == [] - assert updated == [] - - -async def test_main_requires_api_key(monkeypatch): - """Without YANDEX_GEOCODER_API_KEY the script exits cleanly.""" - monkeypatch.delenv("YANDEX_GEOCODER_API_KEY", raising=False) - with pytest.raises(SystemExit): - await main(["--batch", "no_key"]) - - -async def test_main_audit_only_flag_routes_to_audit_loop(monkeypatch): - """--audit-only switches sample query + loop, no UPDATE expected.""" - audit_sample = [ - {"id": 50, "address": "ул Малышева 51", "lat": 56.838, "lon": 60.586}, - ] - db, inserted, updated = _make_db_mock(audit_sample=audit_sample, distance_value=8.0) - monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY") - fake = YandexReverseResult( - address="Россия, Екатеринбург, улица Малышева, 51", - snapped_lat=56.838004, - snapped_lon=60.586155, - precision="exact", - kind="house", - raw={"r": 1}, - ) - with ( - patch("scripts.backfill_house_coords.SessionLocal", return_value=db), - patch( - "scripts.backfill_house_coords.reverse_via_api", - new=AsyncMock(return_value=fake), - ), - ): - n = await main(["--batch", "audit_run", "--audit-only"]) - assert n == 1 - assert updated == [] # audit mode never updates houses - assert inserted[0]["audit_status"] == "ok" - assert inserted[0]["distance_m"] == 8.0 diff --git a/tradein-mvp/backend/tests/test_geocoder_nominatim_lookup.py b/tradein-mvp/backend/tests/test_geocoder_nominatim_lookup.py new file mode 100644 index 00000000..b70b3c36 --- /dev/null +++ b/tradein-mvp/backend/tests/test_geocoder_nominatim_lookup.py @@ -0,0 +1,91 @@ +"""Тесты `_nominatim_lookup` — city реально доходит до исходящего HTTP-запроса. + +#2593 (часть 3): Yandex Geocoder полностью удалён из проекта, вместе с ним ушли +`_yandex_reverse.py` + `tests/test_audit_address_mismatch.py` + +`tests/test_backfill_house_coords.py` — они были единственной проверкой, что +`city`/`city_hint` реально передаётся во внешний геокодер, а не только влияет на +cache-ключ (см. `tests/test_geocoder_city_hint.py`, который мокает +`_nominatim_lookup`/`_nominatim_suggest` целиком и потому не видит их внутренности). + +Nominatim теперь единственный живой внешний провайдер (`_nominatim_lookup` +docstring, `app/services/geocoder.py`) — этот файл закрывает получившуюся дыру: +мокает HTTP-транспорт (`httpx.MockTransport`, паттерн из `test_geocoder_bbox.py` / +`tests/services/test_dadata.py`) и проверяет параметр `q` реального исходящего +GET-запроса к `nominatim.openstreetmap.org/search`. +""" + +from __future__ import annotations + +import os + +os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test") + +from unittest.mock import patch + +import httpx + +from app.services.geocoder import _nominatim_lookup + +# EKB-центр (Плотинка) — внутри tight EKB bbox, `_nominatim_query` его примет +# без похода во второй (typo-variant) тир. +_SAMPLE_ITEM = { + "lat": "56.838", + "lon": "60.605", + "class": "building", + "display_name": "ул. Малышева, 30, Екатеринбург", + "address": {"state": "Свердловская область"}, +} + +# Snapshot реального httpx.AsyncClient ДО patch'а — фабрика ниже использует именно +# его с подменённым transport (паттерн tests/services/test_dadata.py: избегает +# recursion, если бы `httpx.AsyncClient` патчился поверх самого себя). +_REAL_ASYNC_CLIENT = httpx.AsyncClient + + +def _async_client_factory(transport: httpx.MockTransport): + def factory(*_: object, **__: object) -> httpx.AsyncClient: + return _REAL_ASYNC_CLIENT(transport=transport) + + return factory + + +def _capturing_transport(captured_q: list[str]) -> httpx.MockTransport: + def handler(request: httpx.Request) -> httpx.Response: + captured_q.append(request.url.params.get("q", "")) + return httpx.Response(200, json=[_SAMPLE_ITEM]) + + return httpx.MockTransport(handler) + + +async def test_nominatim_lookup_sends_city_hint_in_query_param() -> None: + """city_hint="Нижний Тагил" должен попасть в q= реального GET-запроса. + + Регрессия, о которой явно предупреждает docstring `_nominatim_lookup` (#2580 C): + city_hint обязан влиять на сам запрос к провайдеру, не только на cache-ключ. + """ + captured_q: list[str] = [] + transport = _capturing_transport(captured_q) + + with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)): + result = await _nominatim_lookup("Ленина, 1", city_hint="Нижний Тагил") + + assert captured_q, "запрос к Nominatim не был отправлен" + assert captured_q[0] == "Нижний Тагил, Ленина, 1" + assert result is not None + assert result.provider == "nominatim" + + +async def test_nominatim_lookup_no_city_sends_bare_address() -> None: + """Без city_hint и без маркера города в тексте — q= остаётся bare-адресом. + + Guard против регрессии в молчаливый дефолт на конкретный город (#2576/#2593) + — до фикса #2576 сюда молча подставлялся "Екатеринбург". + """ + captured_q: list[str] = [] + transport = _capturing_transport(captured_q) + + with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)): + result = await _nominatim_lookup("Малышева, 30") + + assert captured_q == ["Малышева, 30"] + assert result is not None