lekss361/gendesign
1
0
Fork 0
Code Issues 143 Pull requests Projects Releases Packages Wiki Activity Actions

chore(tradein/geocoder): удалить остатки скриптов Яндекс-геокодера, часть 3 (#2593) #2599

Merged
lekss361 merged 2 commits from chore/tradein-remove-yandex-scripts into main 2026-07-31 20:53:43 +00:00
Conversation 0 Commits 2 Files changed 13 +109 -2842
13 changed files with 109 additions and 2842 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
docs/Secrets_Rotation_Policy.md
View file

@ -77,7 +77,6 @@
|---|---|---|
| `TRADEIN_POSTGRES_PASSWORD` / `TRADEIN_POSTGRES_USER` | Пароль/юзер БД `tradein` | **E** |
| `TRADEIN_READER_PASSWORD` | Пароль роли `gendesign_reader` (ETL #976, `ops/db-bootstrap/set_gendesign_reader_password.sql`) | **E** |
| `YANDEX_GEOCODER_API_KEY` | Yandex Geocoder (25k req/day) | **D** |
| `DADATA_API_TOKEN` / `DADATA_API_SECRET` | DaData `/clean/address` enrichment | **D** |
| `SCRAPER_PROXY_URL` (+ legacy `AVITO_PROXY_URL`, `CIAN_PROXY_URL`, `YANDEX_PROXY_URL` и их `*_ROTATE_URL`) | Мобильный прокси для скраперов (содержит user:pass в URL) | **G** (proxy creds) |
| `CIAN_LOGIN_EMAIL` / `CIAN_LOGIN_PASSWORD` | Cian browser auto-login (#639, Variant B) | **D** |
@ -147,14 +146,14 @@ bcrypt-хеши — односторонние, не plaintext-секреты,
3. Frontend: обновить `GLITCHTIP_FRONTEND_DSN` (build-arg `NEXT_PUBLIC_GLITCHTIP_DSN`) → требует **rebuild frontend образа** (запекается на build-time) → `workflow_dispatch` или push в `frontend/**`.
4. Vault entry.
### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`)
### Класс D — 3rd-party API keys (`OBJECTIVE_API_KEY`, `OPENAI_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*`)
**Downtime:** нет (фичи gracefully degrade при пустом ключе — см. config-комментарии).
1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / Yandex Cloud / DaData / Cian-аккаунт).
1. Перевыпустить/ротировать ключ в кабинете провайдера (Объектив / OpenAI / DaData / Cian-аккаунт).
2. Где живёт:
- `OBJECTIVE_API_KEY`, `OPENAI_API_KEY` — Forgejo secret → deploy пишет в main `.env.runtime`.
- `YANDEX_GEOCODER_API_KEY`, `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI).
- `DADATA_*`, `CIAN_LOGIN_*` — tradein `.env.runtime` (правится **на VPS вручную**, не из CI).
3. Обновить значение `sed`-ом (НЕ перезапись файла) и `up -d --force-recreate --no-deps backend worker beat` (main) / `... backend scraper` (tradein).
4. Vault entry.

6
tradein-mvp/.env.example
View file

@ -6,12 +6,6 @@ DATABASE_URL=postgresql+psycopg://tradein:tradein@postgres:5432/tradein
CORS_ORIGINS=["http://localhost:8080","http://localhost:3000"]
ENVIRONMENT=dev
# Yandex Geocoder API key (25k req/day free tier).
# Required for backfill scripts (scripts/backfill_house_coords.py + audit_address_mismatch.py).
# Empty = Nominatim fallback для backend геокодинга; backfill scripts требуют этот ключ
# и упадут с SystemExit без него.
YANDEX_GEOCODER_API_KEY=
# DaData /clean/address — обогащение target адреса в estimate flow (PR Q1).
# Возвращает canonical-форму, kadastr_num, ФИАС, координаты, ближайшее метро.
# Demo tier: 100 req/день — хватит для тестов и low-traffic prod.

8
tradein-mvp/DEPLOY.md
View file

@ -57,8 +57,7 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment
shell-скриптом deploy через `source .env.runtime` перед `compose up`.
2. `/opt/gendesign/tradein-mvp/backend/.env.runtime` — переменные внутри
контейнера `tradein-backend` (читаются через `env_file:` в compose). Сюда
попадают `YANDEX_GEOCODER_API_KEY`, `COOKIE_ENCRYPTION_KEY` —
всё, что нужно scripts/backfill_house_coords.py и application code внутри
попадают `COOKIE_ENCRYPTION_KEY` и остальные application-секреты внутри
контейнера.
```bash
@ -66,7 +65,6 @@ import /opt/gendesign/tradein-mvp/deploy/Caddyfile.tradein-fragment
TRADEIN_POSTGRES_USER=tradein
TRADEIN_POSTGRES_PASSWORD=<сгенерировать openssl rand -hex 32>
TRADEIN_CONTACT_EMAIL=tradein@gendsgn.ru
YANDEX_GEOCODER_API_KEY= # пусто пока, Nominatim fallback работает
# Encryption key for Cian session cookies (pgp_sym_encrypt / Stage 9 Calculator).
# Empty = Valuation Calculator scraper disabled + /api/v1/cookies/upload returns 503.
@ -77,10 +75,9 @@ COOKIE_ENCRYPTION_KEY=<64-char hex>
```bash
# /opt/gendesign/tradein-mvp/backend/.env.runtime — те же ключи которые
# читаются ВНУТРИ container'а (scripts/backfill_house_coords.py, app/*).
# читаются ВНУТРИ container'а (app/*, scripts/*.py).
# Может быть симлинком на ../.env.runtime если переменные совпадают:
# ln -s ../.env.runtime /opt/gendesign/tradein-mvp/backend/.env.runtime
YANDEX_GEOCODER_API_KEY=<key или пусто>
COOKIE_ENCRYPTION_KEY=<64-char hex>
GENDESIGN_FDW_PASSWORD=<password или пусто>
GLITCHTIP_DSN=<dsn или пусто>
@ -200,7 +197,6 @@ cat > tradein-mvp/.env.runtime <<EOF
TRADEIN_POSTGRES_USER=tradein
TRADEIN_POSTGRES_PASSWORD=$(openssl rand -hex 32)
TRADEIN_CONTACT_EMAIL=tradein@gendsgn.ru
YANDEX_GEOCODER_API_KEY=
EOF
chmod 600 tradein-mvp/.env.runtime

150
tradein-mvp/backend/scripts/README.md
View file

@ -10,144 +10,20 @@ each row is wrapped in a SAVEPOINT (`db.begin_nested()`) per `.claude/rules/back
---
## Production usage (canonical)
## Address audit + backfill (issue #582) — REMOVED (#2593)
Scripts ship inside the `tradein-backend` image (PR F — `COPY scripts ./scripts`
в `backend/Dockerfile`). На VPS они уже в `/app/scripts/` — никаких manual
`docker cp` не нужно.
`YANDEX_GEOCODER_API_KEY` подтягивается из `/opt/gendesign/tradein-mvp/backend/
.env.runtime` через `env_file:` в `docker-compose.prod.yml` — никакого `-e` в
`docker exec` не нужно.
```bash
# Backfill (forward geocode 4170 houses без coords)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --batch 2026-05-27_backfill'
# Audit-only (reverse geocode проверка для уже geocoded houses)
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --audit-only --batch 2026-05-27_audit'
# Canary first
ssh gendesign 'docker exec tradein-backend python -m scripts.backfill_house_coords --limit 100 --batch canary_$(date +%F)'
```
После изменения `backend/.env.runtime` нужен `--force-recreate` контейнера
(см. `.claude/rules/deploy.md`):
```bash
ssh gendesign 'cd /opt/gendesign/tradein-mvp && docker compose -p gendesign-tradein -f docker-compose.prod.yml up -d --force-recreate --no-deps backend'
```
---
## Address audit + backfill (issue #582)
End-to-end address quality pipeline. Three scripts, two helpers, two SQL files.
> Локальные примеры ниже — для dev-машины с `uv run` и переменными в shell.
> На prod используй canonical `docker exec` команды из секции выше — там
> `YANDEX_GEOCODER_API_KEY` уже подгружен из `backend/.env.runtime`.
### `audit_address_mismatch.py` — Phase 1 baseline (PR #583)
Stratified-sample audit (200 EKB houses) comparing `houses.address` vs
Yandex Geocoder reverse lookup. Writes one row per house into
`address_mismatch_audit` with the snapped point + canonical address + distance.
```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.audit_address_mismatch \
--batch 2026-05-25_run1 \
--limit-per-district 25
```
Mode `auto` picks API if the key is set, otherwise Playwright (CAPTCHA-aware,
4-7s sleep between calls). API tier free is 25k req/day → 200-row sample
takes ~10s with no quota concern.
Report:
```bash
psql "$DATABASE_URL" -v batch='2026-05-25_run1' \
-f scripts/address_audit_report.sql
```
### `backfill_house_coords.py` — Phase 2-3 (PR for #582)
Two modes (`--audit-only` flag switches between them):
**Backfill (default)** — forward-geocode `houses.address` for the ~4141 rows
WHERE `lat IS NULL OR lon IS NULL`. Only writes back if Yandex returns
`precision='exact'` or `'number'` (skips street-only / locality matches).
Each processed row gets an `address_mismatch_audit` entry with status
`backfill` / `imprecise` / `no_match` / `error`.
```bash
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
uv run python -m scripts.backfill_house_coords \
--batch 2026-05-27_backfill
```
Expected duration (~4141 rows, 50ms between calls, ~250ms RTT per request):
20-25 min. Expected output split (rough baseline from Phase 1 numbers):
| Status | Approx rows | What it means |
|-------------|-------------|-----------------------------------------------------|
| `backfill` | ~3.3k–3.7k | UPDATE landed, lat/lon now populated |
| `imprecise` | ~300–500 | Match returned but precision too low — needs review |
| `no_match` | ~100–300 | Yandex couldn't resolve; address probably mangled |
| `error` | <50 | HTTP errors / timeouts — re-run picks them up |
**Audit-only** — reverse-geocode the ~4452 houses WITH coords, write
audit rows with status `ok` (≤50m) / `mismatch` (>50m) / `no_match` / `error`.
Does NOT modify the `houses` table.
```bash
uv run python -m scripts.backfill_house_coords \
--batch 2026-05-27_audit --audit-only
```
Combined budget for both phases (~8.6k requests) is well under the 25k/day
Geocoder free tier.
### Common ops
Canary first — run with `--limit 100` and inspect the audit table before
letting the full job loose:
```bash
uv run python -m scripts.backfill_house_coords \
--batch canary_$(date +%F) --limit 100
psql "$DATABASE_URL" -c "
SELECT audit_status, COUNT(*)
FROM address_mismatch_audit
WHERE audit_batch = 'canary_$(date +%F)'
GROUP BY audit_status;
"
```
Resume after crash / quota hit — same `--batch` label, the UNIQUE
`(house_id, audit_batch)` index skips finished rows:
```bash
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# ... interruption ...
uv run python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# logs: "resuming batch 2026-05-27_backfill: N rows already processed"
```
### Helpers (not entry points)
- `_yandex_reverse.py` — `forward_via_api()`, `reverse_via_api()`,
`reverse_via_playwright()`, `YandexReverseResult` dataclass. Both API
paths share `_parse_api_payload` because Yandex's forward/reverse
envelopes have the same shape.
- `audit_address_sample.sql` — random sample for the Phase 1 audit (used
by `audit_address_mismatch.py`).
- `address_audit_report.sql` — psql-driven post-run summary (p50/p75/p95
distance, top-20 outliers, per-district breakdown).
`audit_address_mismatch.py`, `backfill_house_coords.py`, `_yandex_reverse.py`
и их SQL-хелперы (`audit_address_sample.sql`, `address_audit_report.sql`)
удалены — весь pipeline опирался на Yandex Geocoder API, который выпилен
из проекта (#2593, части 1-3). `houses.address`→lat/lon geocoding теперь
идёт через `app/services/geocoder.py` (кадастр/геопортал ЕКБ-тиры + Nominatim
fallback, единственный живой внешний провайдер) на обычном write-path
(`/api/v1/trade-in/estimate`, listing ingest). Разовый forward-backfill
недостающих `houses` координат — `scripts/geocode_deals_nominatim.py`
(живой, работает с `rosreestr_deals`, не с `houses` — читай его docstring
перед использованием на других таблицах). Таблица `address_mismatch_audit`
осталась в схеме (используется `house_dedup_merge.py` при слиянии дублей
домов, независимо от Yandex-аудита).
---

380
tradein-mvp/backend/scripts/_yandex_reverse.py
View file

@ -1,380 +0,0 @@
"""Yandex Geocoder helpers for the address-mismatch audit + backfill (issue #582).
Three geocoding paths exposed:
- `reverse_via_api()` — Yandex Geocoder HTTP API, lon/lat → address. Fast,
structured response, needs a valid API key (env `YANDEX_GEOCODER_API_KEY`).
Free tier is 25k req/day, fine for ~8.5k houses + audit (~17k total).
- `reverse_via_playwright()` — fallback when no API key is available. Drives
a real browser session at https://yandex.ru/maps/?…&mode=whatshere. Slower
and CAPTCHA-prone, so the driver inserts 4-7s sleeps between calls and we
raise a dedicated exception on CAPTCHA so the batch can pause-and-resume.
- `forward_via_api()` — address → lon/lat + canonical address (Phase 2 of
issue #582). Used by `backfill_house_coords.py` to fill `houses.lat/lon`
for the 4141 houses scraped from sources that didn't include coords (esp.
yandex_valuation, which only returns an address string).
All three return a `YandexReverseResult` dataclass — same shape regardless
of direction so the driver code stays implementation-agnostic. The `raw`
field always carries the full source payload for post-hoc diagnostics, and
`precision` / `kind` are filled in by the API paths so the caller can skip
imprecise matches (e.g. only-street-level results during backfill).
Why three paths:
The user (issue #582 discussion) wants the audit to run on dev machines
that may not have an API key, but on prod we already provision the key for
estimator.py. Forward geocode is API-only — Playwright forward geocoding
through Yandex Maps search is too fragile (relevance ranking, suggest
dropdown). For dev without a key, backfill simply doesn't run.
"""
from __future__ import annotations
import asyncio
import logging
import random
from dataclasses import dataclass, field
from typing import Any
import httpx
logger = logging.getLogger(__name__)
# Yandex Maps "what's here" URL — wraps a reverse-geocode in browser-driven UI.
# `whatshere[point]` accepts "<lon>,<lat>" (note: lon first, Yandex convention).
_YANDEX_MAPS_WHATSHERE = (
"https://yandex.ru/maps/?ll={lon:.6f}%2C{lat:.6f}&z=18&mode=whatshere"
"&whatshere%5Bpoint%5D={lon:.6f}%2C{lat:.6f}&whatshere%5Bzoom%5D=18"
)
# Geocoder HTTP API. `kind=house` narrows the result to a building if possible,
# which is what we want for cadastr-style addresses (улица + дом).
_YANDEX_GEOCODE_API = "https://geocode-maps.yandex.ru/1.x/"
# Reasonable timeouts: API call should be sub-second; we give it generous
# headroom for slow networks but not so much that a hang stalls the batch.
_API_TIMEOUT = httpx.Timeout(connect=5.0, read=10.0, write=5.0, pool=5.0)
# ---------------------------------------------------------------------------
# Dataclasses + exceptions
# ---------------------------------------------------------------------------
@dataclass
class YandexReverseResult:
"""Normalized result of a geocode call (forward, reverse-API, or browser).
Attributes:
address: Human-readable canonical address Yandex returned. For
reverse, this is the snapped address at the queried point. For
forward, this is the canonical form of the input address. None
if Yandex returned no match.
snapped_lat: Latitude of the matched object's geometric centre.
snapped_lon: Longitude of the matched object's geometric centre.
precision: For forward calls — Yandex match precision tag (`exact`,
`number`, `near`, `range`, `street`, `other`). For reverse —
same field is filled when present (usually `house` / `street`).
None for the playwright path. Used by the backfill driver to
skip imprecise matches.
kind: Object kind from Yandex (`house`, `street`, `locality`, ...).
Same source as `precision` — see metaDataProperty.GeocoderMetaData.
raw: Raw response payload retained for forensics (JSON dict from API,
or snapshot dict from playwright). Used to populate
`address_mismatch_audit.raw_payload` and
`houses.raw_payload.yandex_geocode`.
"""
address: str | None
snapped_lat: float | None
snapped_lon: float | None
raw: dict[str, Any] = field(default_factory=dict)
precision: str | None = None
kind: str | None = None
class YandexBlockedError(RuntimeError):
"""Raised when Yandex returns a CAPTCHA / anti-bot challenge.
The driver catches this, marks the row `audit_status='blocked'`, logs the
current batch position, then exits cleanly so a human can intervene.
"""
# ---------------------------------------------------------------------------
# Path A — HTTP Geocoder API
# ---------------------------------------------------------------------------
async def reverse_via_api(
lat: float,
lon: float,
api_key: str,
*,
client: httpx.AsyncClient | None = None,
) -> YandexReverseResult:
"""Reverse-geocode (lat, lon) via the Yandex Geocoder HTTP API.
Why a separate `client` parameter: lets the driver reuse one
`AsyncClient` across all 200 calls (TCP keep-alive + connection pool),
and lets the tests inject a `MockTransport` to assert request shape.
Args:
lat: latitude in WGS84.
lon: longitude in WGS84.
api_key: Yandex Geocoder API key.
client: optional pre-built async client. If None, a one-shot client
is created.
Returns:
`YandexReverseResult` with the first `featureMember[0].GeoObject`
result, or all-None if Yandex returned no match (still includes
`raw` payload so we can later inspect why).
"""
params = {
"apikey": api_key,
# Yandex expects "lon,lat" (longitude first) per docs — same
# convention as the "whatshere" map URL above.
"geocode": f"{lon},{lat}",
"format": "json",
"kind": "house",
"results": "1",
}
own_client = client is None
if client is None:
client = httpx.AsyncClient(timeout=_API_TIMEOUT)
try:
resp = await client.get(_YANDEX_GEOCODE_API, params=params)
resp.raise_for_status()
data = resp.json()
finally:
if own_client:
await client.aclose()
return _parse_api_payload(data)
def _parse_api_payload(data: dict[str, Any]) -> YandexReverseResult:
"""Extract address + snapped point from a Yandex Geocoder API JSON response.
Split out so unit tests can feed a fixture file directly without spinning
up an HTTP mock. Same payload shape for forward and reverse calls —
Yandex's response envelope is symmetric.
"""
try:
members = data.get("response", {}).get("GeoObjectCollection", {}).get("featureMember", [])
if not members:
return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data)
geo_obj = members[0].get("GeoObject", {})
# Address: prefer the long `metaDataProperty.GeocoderMetaData.text`
# (full canonical) and fall back to `name` (street + house number).
meta = geo_obj.get("metaDataProperty", {}).get("GeocoderMetaData", {})
address = meta.get("text") or geo_obj.get("name")
precision = meta.get("precision")
kind = meta.get("kind")
# Point format: "<lon> <lat>" — space-separated string.
point_str = geo_obj.get("Point", {}).get("pos", "")
snapped_lon: float | None
snapped_lat: float | None
if point_str:
try:
lon_s, lat_s = point_str.split()
snapped_lon = float(lon_s)
snapped_lat = float(lat_s)
except (ValueError, TypeError):
snapped_lon = None
snapped_lat = None
else:
snapped_lon = None
snapped_lat = None
return YandexReverseResult(
address=address,
snapped_lat=snapped_lat,
snapped_lon=snapped_lon,
raw=data,
precision=precision,
kind=kind,
)
except Exception as e: # pragma: no cover — defensive; tests cover happy paths
logger.warning("yandex API payload parse failed: %s", e)
return YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw=data)
# ---------------------------------------------------------------------------
# Path A.2 — Forward geocode (address → lon/lat) via HTTP API
# ---------------------------------------------------------------------------
async def forward_via_api(
address: str,
api_key: str,
*,
client: httpx.AsyncClient | None = None,
) -> YandexReverseResult:
"""Forward-geocode an address string via the Yandex Geocoder HTTP API.
Phase 2 of issue #582 — used by `backfill_house_coords.py` to populate
`houses.lat/lon` for houses that were scraped without coords (esp.
yandex_valuation rows, which only carry an address).
Args:
address: free-form address ("ул Малышева 51", "Екатеринбург, Ленина 5",
etc.). Yandex's NLU is forgiving — no need to pre-normalize.
api_key: Yandex Geocoder API key.
client: optional pre-built async client. If None, a one-shot client
is created (matches `reverse_via_api` ergonomics).
Returns:
`YandexReverseResult` with the canonical address + snapped point of
the first matching feature. `precision` and `kind` are populated so
the backfill driver can skip imprecise hits (e.g. precision='street'
means we landed on the road, not the building — too vague for
comparable-listings spatial queries).
Same envelope as `reverse_via_api` — `_parse_api_payload` handles both.
"""
params = {
"apikey": api_key,
"geocode": address,
"format": "json",
# `kind=house` filters out street-only / locality-only matches at
# the API level when possible. Yandex still returns lower-precision
# results when no building matches, so the caller must double-check
# `precision` before writing to houses.
"kind": "house",
"results": "1",
# Locality bias for EKB — improves recall when the input address
# omits the city. The audit population is 99% EKB houses, so this
# is safe; non-EKB inputs (rare) still resolve, just with the bias.
"ll": "60.6122,56.8389",
"spn": "0.6,0.4",
}
own_client = client is None
if client is None:
client = httpx.AsyncClient(timeout=_API_TIMEOUT)
try:
resp = await client.get(_YANDEX_GEOCODE_API, params=params)
resp.raise_for_status()
data = resp.json()
finally:
if own_client:
await client.aclose()
return _parse_api_payload(data)
# ---------------------------------------------------------------------------
# Path B — Playwright fallback
# ---------------------------------------------------------------------------
async def reverse_via_playwright(
lat: float,
lon: float,
page: Any,
) -> YandexReverseResult:
"""Reverse-geocode (lat, lon) by driving yandex.ru/maps with Playwright.
Why this exists:
The Yandex Geocoder API requires a key with paid quota for >25k/day. The
audit only needs 200 rows but a dev without a key still needs a way to
run the script, so we ship a browser-driven fallback.
Implementation:
1. Navigate to the `whatshere` URL — Yandex Maps responds by opening a
toponym card at the requested coordinates and rendering the resolved
address in the side panel.
2. Wait for client hydration (`networkidle`).
3. First try to read `window.__INITIAL_STATE__` — Yandex stores the
toponym address inside the hydrated Redux tree, which is more
stable across UI redesigns than DOM selectors.
4. Fall back to DOM selectors (`.toponym-card-title-view__title` +
`__subtitle`) if the state walk doesn't find an address.
5. Detect CAPTCHA (`.CheckboxCaptcha`) early and raise `YandexBlockedError`
so the batch can pause-and-resume without spamming Yandex.
`page` is typed as `Any` to keep playwright a dev-only dep — runtime
importers don't need playwright installed if they only use the API path.
"""
url = _YANDEX_MAPS_WHATSHERE.format(lat=lat, lon=lon)
await page.goto(url, wait_until="domcontentloaded")
# Light wait for client-side hydration. Yandex Maps fires lots of
# background XHRs so `networkidle` is too aggressive; this small wait is
# enough for the toponym card to render.
try:
await page.wait_for_load_state("networkidle", timeout=8000)
except Exception as e:
# Slow networks: continue — selectors will retry with their own waits.
logger.debug("networkidle wait timed out, continuing: %s", e)
await asyncio.sleep(random.uniform(0.5, 1.2))
# CAPTCHA gate — Yandex shows a `.CheckboxCaptcha` form when it suspects
# automation. Once we see it, every subsequent reverse call will also be
# blocked, so we raise immediately and let the driver stop the batch.
captcha = await page.query_selector(".CheckboxCaptcha")
if captcha is not None:
raise YandexBlockedError("Yandex CAPTCHA detected on maps page")
# Attempt 1 — initial state walk.
state_addr: str | None = None
state_pos: tuple[float, float] | None = None
try:
state_addr, state_pos = await page.evaluate(
"() => {\n"
" const s = window.__INITIAL_STATE__ || {};\n"
" const card = (s.cards && s.cards.toponym) || (s.card && s.card.toponym) || null;\n"
" if (!card) return [null, null];\n"
" const addr = card.title || card.address || null;\n"
" const pos = card.coords || card.point || null;\n"
" if (pos && pos.length === 2) return [addr, [pos[0], pos[1]]];\n"
" return [addr, null];\n"
"}"
)
except Exception as e:
logger.debug("playwright state walk failed (will fall back to DOM): %s", e)
address = state_addr
# Attempt 2 — DOM fallback.
if not address:
title_el = await page.query_selector(".toponym-card-title-view__title")
subtitle_el = await page.query_selector(".toponym-card-title-view__subtitle")
title = (await title_el.inner_text()).strip() if title_el else ""
subtitle = (await subtitle_el.inner_text()).strip() if subtitle_el else ""
# subtitle often holds "Екатеринбург, район", title the street + house
address = ", ".join([p for p in (subtitle, title) if p]) or None
snapped_lat: float | None
snapped_lon: float | None
if state_pos:
# State stored as [lon, lat] in Yandex's coordinate convention.
snapped_lon = float(state_pos[0])
snapped_lat = float(state_pos[1])
else:
snapped_lon = None
snapped_lat = None
raw = {
"url": url,
"state_addr": state_addr,
"state_pos": list(state_pos) if state_pos else None,
"dom_address": address if not state_addr else None,
}
return YandexReverseResult(
address=address,
snapped_lat=snapped_lat,
snapped_lon=snapped_lon,
raw=raw,
)

91
tradein-mvp/backend/scripts/address_audit_report.sql
View file

@ -1,91 +0,0 @@
-- address_audit_report.sql
-- Post-run report for the address-mismatch audit (issue #582 Phase 1).
--
-- Sections:
-- 1. Summary — count, p50/p75/p95/mean distance, % street_differs,
-- % over 50m / 200m thresholds.
-- 2. Top-20 outliers by distance (manual triage list).
-- 3. Per-district breakdown — same metrics grouped by district column.
--
-- Run via psql:
-- psql "$DATABASE_URL" -v batch='2026-05-25_run1' -f scripts/address_audit_report.sql
--
-- :batch is a psql client variable substituted via -v.
\set ON_ERROR_STOP on
\echo '=============================================='
\echo ' Address mismatch audit — batch:' :batch
\echo '=============================================='
-- ---------------------------------------------------------------------------
-- 1) Top-level summary
-- ---------------------------------------------------------------------------
\echo ''
\echo '--- Summary (status=ok rows only) ---'
SELECT
COUNT(*) AS n_total,
COUNT(*) FILTER (WHERE audit_status = 'ok') AS n_ok,
COUNT(*) FILTER (WHERE audit_status = 'no_match') AS n_no_match,
COUNT(*) FILTER (WHERE audit_status = 'error') AS n_error,
COUNT(*) FILTER (WHERE audit_status = 'blocked') AS n_blocked,
ROUND(percentile_cont(0.50)
WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p50_distance_m,
ROUND(percentile_cont(0.75)
WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p75_distance_m,
ROUND(percentile_cont(0.95)
WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p95_distance_m,
ROUND(AVG(distance_m)::numeric, 1) AS mean_distance_m,
ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1)
AS pct_street_differs,
ROUND(100.0 * AVG(CASE WHEN distance_m > 50 THEN 1.0 ELSE 0.0 END), 1)
AS pct_over_50m,
ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1)
AS pct_over_200m
FROM address_mismatch_audit
WHERE audit_batch = :'batch'
AND audit_status = 'ok';
-- ---------------------------------------------------------------------------
-- 2) Top-20 outliers
-- ---------------------------------------------------------------------------
\echo ''
\echo '--- Top-20 outliers by distance ---'
SELECT
house_id,
district,
ROUND(distance_m::numeric, 1) AS distance_m,
street_differs,
LEFT(original_address, 60) AS original_address,
LEFT(snapped_address, 60) AS snapped_address
FROM address_mismatch_audit
WHERE audit_batch = :'batch'
AND audit_status = 'ok'
AND distance_m IS NOT NULL
ORDER BY distance_m DESC NULLS LAST
LIMIT 20;
-- ---------------------------------------------------------------------------
-- 3) Per-district breakdown
-- ---------------------------------------------------------------------------
\echo ''
\echo '--- Per-district breakdown (status=ok only) ---'
SELECT
COALESCE(district, '(no district)') AS district,
COUNT(*) AS n,
ROUND(percentile_cont(0.50)
WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p50_distance_m,
ROUND(percentile_cont(0.95)
WITHIN GROUP (ORDER BY distance_m)::numeric, 1) AS p95_distance_m,
ROUND(AVG(distance_m)::numeric, 1) AS mean_distance_m,
ROUND(100.0 * AVG(CASE WHEN street_differs THEN 1.0 ELSE 0.0 END), 1)
AS pct_street_differs,
ROUND(100.0 * AVG(CASE WHEN distance_m > 50 THEN 1.0 ELSE 0.0 END), 1)
AS pct_over_50m,
ROUND(100.0 * AVG(CASE WHEN distance_m > 200 THEN 1.0 ELSE 0.0 END), 1)
AS pct_over_200m
FROM address_mismatch_audit
WHERE audit_batch = :'batch'
AND audit_status = 'ok'
GROUP BY COALESCE(district, '(no district)')
ORDER BY n DESC, district;

595
tradein-mvp/backend/scripts/audit_address_mismatch.py
View file

@ -1,595 +0,0 @@
"""Audit driver — compares houses.address vs Yandex reverse geocode.
Phase 1 of Forgejo issue #582. Pulls a stratified sample of EKB houses (25
per admin district = 200 total), reverse-geocodes each via Yandex, computes
the distance between the stored coordinates and the snapped Yandex point,
and writes the result into `address_mismatch_audit`.
Design choices:
- **Resumable**: the audit table has UNIQUE (house_id, audit_batch). Re-run
with the same `--batch` skips rows already inserted, so a partial run can
be picked up after CAPTCHA / network blip.
- **Mode auto**: prefer API when `YANDEX_GEOCODER_API_KEY` is set, fall back
to Playwright otherwise. Explicit override via `--mode {api,playwright}`.
- **No prod side effects**: the script only writes to one new audit table;
it never touches `houses`, `house_sources`, or any matching/listing row.
- **Per-row SAVEPOINT**: a single Yandex error must not nuke the entire
batch — wrap each INSERT in `db.begin_nested()` per backend.md.
How to run:
DATABASE_URL=postgresql+psycopg://... \
YANDEX_GEOCODER_API_KEY=... \
python -m scripts.audit_address_mismatch --batch 2026-05-25_run1
Outputs (post-run):
- New rows in `address_mismatch_audit` with batch label.
- `scripts/address_audit_report.sql :batch=<id>` for summary.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import logging
import os
import random
from dataclasses import dataclass
from datetime import date
from pathlib import Path
from typing import Any
import httpx
from sqlalchemy import text
from sqlalchemy.orm import Session
# Allow running both as `python -m scripts.audit_address_mismatch` (preferred)
# and as a stand-alone file (`python scripts/audit_address_mismatch.py`)
# without requiring package install.
try:
from app.core.db import SessionLocal # type: ignore[import-not-found]
from app.services.matching.normalize import normalize_address # type: ignore[import-not-found]
except ImportError: # pragma: no cover — fallback for adhoc invocation
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from app.core.db import SessionLocal
from app.services.matching.normalize import normalize_address
# `from .` works when run via -m; the absolute import works under pytest.
try:
from scripts._yandex_reverse import ( # type: ignore[import-not-found]
YandexBlockedError,
YandexReverseResult,
reverse_via_api,
reverse_via_playwright,
)
except ImportError:
from _yandex_reverse import ( # type: ignore[no-redef]
YandexBlockedError,
YandexReverseResult,
reverse_via_api,
reverse_via_playwright,
)
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger("audit_address_mismatch")
# Playwright persistent context location — keeps cookies/local storage between
# runs so we look like a returning user, reducing CAPTCHA frequency.
_PLAYWRIGHT_USER_DATA = Path.home() / ".cache" / "tradein-audit-playwright"
_PLAYWRIGHT_UA = (
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
)
_SAMPLE_SQL_PATH = Path(__file__).parent / "audit_address_sample.sql"
# ---------------------------------------------------------------------------
# Domain helpers
# ---------------------------------------------------------------------------
@dataclass
class SampleRow:
"""One house from the stratified sampling query."""
id: int
address: str
lat: float
lon: float
district: str | None
# Words that introduce a street rather than identify it. We skip these so the
# comparison lands on the actual street name ('малышева' / 'ленина'). Mirrors
# the canonical forms produced by `normalize_address` (which expands all known
# abbreviations to these full words).
_STREET_TYPE_WORDS = frozenset(
{
"улица",
"проспект",
"переулок",
"бульвар",
"проезд",
"шоссе",
"площадь",
"набережная",
"тупик",
"строение",
"корпус",
"дом",
}
)
# Geographic prefix words that addresses sometimes carry before the street
# (e.g. 'россия екатеринбург улица малышева 51'). We skip them too so we
# converge on the same identifying token regardless of how verbose the
# source representation is.
_GEO_PREFIX_WORDS = frozenset(
{
"россия",
"свердловская",
"область",
"екатеринбург",
"город",
"г",
}
)
def _first_street_token(address: str | None) -> str | None:
"""Extract the first street-name token of a normalized address.
Phase-1 heuristic for "do the streets agree": skip numeric tokens (house
numbers), street-type words ('улица', 'проспект', …), and geographic
prefixes ('россия', 'екатеринбург', …) — the next token is the street
name itself, which is the identifying part we want to compare.
Returns None for an empty address or when no candidate token remains.
"""
norm = normalize_address(address or "")
if not norm:
return None
for tok in norm.split():
if not tok:
continue
# Skip purely numeric tokens (e.g. '5', '17а' if it starts with digit).
if tok[0].isdigit():
continue
# Skip street type words and geographic prefixes.
if tok in _STREET_TYPE_WORDS or tok in _GEO_PREFIX_WORDS:
continue
return tok
return None
def _street_differs(original: str | None, snapped: str | None) -> bool | None:
"""True iff first non-numeric token differs between the two addresses.
Returns None when either side is empty — we cannot compute a meaningful
diff (caller writes NULL into the audit row).
"""
a = _first_street_token(original)
b = _first_street_token(snapped)
if a is None or b is None:
return None
return a != b
def _distance_meters(
db: Session,
olat: float,
olon: float,
slat: float,
slon: float,
) -> float | None:
"""Compute great-circle distance via PostGIS geography type.
We could do this in Python with a haversine formula, but the audit table
uses ST_Distance results elsewhere so we use the same authority to avoid
drift. ST_MakePoint(lon, lat) — PostGIS convention is lon first.
"""
row = db.execute(
text(
"SELECT ST_Distance("
" ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), "
" CAST(:olat AS double precision)), 4326)::geography, "
" ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), "
" CAST(:slat AS double precision)), 4326)::geography"
") AS m"
),
{"olat": olat, "olon": olon, "slat": slat, "slon": slon},
).first()
if row is None or row[0] is None:
return None
return float(row[0])
# ---------------------------------------------------------------------------
# Sampling + resumption queries
# ---------------------------------------------------------------------------
def _load_sample(db: Session, limit_per_district: int) -> list[SampleRow]:
"""Run the stratified sampling SQL → list of SampleRow."""
sql = _SAMPLE_SQL_PATH.read_text(encoding="utf-8")
rows = db.execute(text(sql), {"limit_per_district": limit_per_district}).mappings().all()
return [
SampleRow(
id=r["id"],
address=r["address"],
lat=float(r["lat"]),
lon=float(r["lon"]),
district=r["district"],
)
for r in rows
]
def _already_processed_ids(db: Session, batch: str) -> set[int]:
"""Return the set of house_id already in the audit table for this batch.
Drives resumability: drop these from the sample before geocoding.
"""
rows = db.execute(
text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"),
{"b": batch},
).all()
return {r[0] for r in rows}
# ---------------------------------------------------------------------------
# Insert helper
# ---------------------------------------------------------------------------
def _insert_audit_row(
db: Session,
*,
house_id: int,
batch: str,
district: str | None,
original_address: str | None,
original_lat: float | None,
original_lon: float | None,
snapped_address: str | None,
snapped_lat: float | None,
snapped_lon: float | None,
distance_m: float | None,
street_differs: bool | None,
audit_status: str,
error_message: str | None,
raw_payload: dict[str, Any] | None,
) -> None:
"""INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit.
Wrapped in begin_nested by the caller per backend.md SAVEPOINT pattern.
"""
db.execute(
text(
"INSERT INTO address_mismatch_audit ("
" house_id, audit_batch, district,"
" original_address, original_lat, original_lon,"
" snapped_address, snapped_lat, snapped_lon,"
" distance_m, street_differs,"
" audit_status, error_message, raw_payload"
") VALUES ("
" CAST(:house_id AS bigint), CAST(:batch AS text), :district,"
" :original_address, :original_lat, :original_lon,"
" :snapped_address, :snapped_lat, :snapped_lon,"
" :distance_m, :street_differs,"
" CAST(:audit_status AS text), :error_message,"
" CAST(:raw_payload AS jsonb)"
") ON CONFLICT (house_id, audit_batch) DO NOTHING"
),
{
"house_id": house_id,
"batch": batch,
"district": district,
"original_address": original_address,
"original_lat": original_lat,
"original_lon": original_lon,
"snapped_address": snapped_address,
"snapped_lat": snapped_lat,
"snapped_lon": snapped_lon,
"distance_m": distance_m,
"street_differs": street_differs,
"audit_status": audit_status,
"error_message": error_message,
"raw_payload": json.dumps(raw_payload) if raw_payload is not None else None,
},
)
# ---------------------------------------------------------------------------
# Mode dispatcher
# ---------------------------------------------------------------------------
def _resolve_mode(mode: str, api_key: str | None) -> str:
"""Translate `--mode auto` → concrete 'api' / 'playwright' choice.
Explicit modes are passed through unchanged; auto chooses api iff a key
is configured (fail-fast: we don't want a "should have used the API but
silently fell back to slow scraping" surprise).
"""
if mode == "auto":
return "api" if api_key else "playwright"
return mode
# ---------------------------------------------------------------------------
# Main loop
# ---------------------------------------------------------------------------
async def _run_api_mode(
db: Session,
sample: list[SampleRow],
batch: str,
api_key: str,
) -> int:
"""Geocode the sample using the HTTP Geocoder API."""
processed = 0
last_distance: float | None = None
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
for i, row in enumerate(sample, start=1):
status = "ok"
err: str | None = None
res: YandexReverseResult | None = None
try:
res = await reverse_via_api(row.lat, row.lon, api_key, client=client)
except httpx.HTTPError as e:
status = "error"
err = f"http_error: {e!s}"
except Exception as e: # pragma: no cover — defensive
status = "error"
err = f"unhandled: {e!s}"
distance = None
street_diff: bool | None = None
if res is not None and status == "ok":
if res.address is None:
status = "no_match"
else:
if res.snapped_lat is not None and res.snapped_lon is not None:
distance = _distance_meters(
db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
)
last_distance = distance
street_diff = _street_differs(row.address, res.address)
try:
with db.begin_nested():
_insert_audit_row(
db,
house_id=row.id,
batch=batch,
district=row.district,
original_address=row.address,
original_lat=row.lat,
original_lon=row.lon,
snapped_address=res.address if res else None,
snapped_lat=res.snapped_lat if res else None,
snapped_lon=res.snapped_lon if res else None,
distance_m=distance,
street_differs=street_diff,
audit_status=status,
error_message=err,
raw_payload=res.raw if res else None,
)
# Per-row commit: each row is durable on disk before the next
# Yandex call; --batch resume picks up exactly where we crashed.
db.commit()
processed += 1
except Exception as e:
db.rollback()
logger.warning("insert failed for house_id=%s: %s", row.id, e)
if i % 10 == 0:
logger.info(
"progress %d/%d, mode=api, last_distance=%s",
i,
len(sample),
f"{last_distance:.1f}m" if last_distance is not None else "n/a",
)
return processed
async def _run_playwright_mode(
db: Session,
sample: list[SampleRow],
batch: str,
) -> int:
"""Geocode via a persistent Playwright context (CAPTCHA-aware)."""
try:
from playwright.async_api import async_playwright # type: ignore[import-not-found]
except ImportError as e:
raise RuntimeError(
"Playwright is required for --mode playwright. "
"Install with `uv sync --group dev` and `playwright install chromium`."
) from e
_PLAYWRIGHT_USER_DATA.mkdir(parents=True, exist_ok=True)
processed = 0
last_distance: float | None = None
async with async_playwright() as p:
context = await p.chromium.launch_persistent_context(
user_data_dir=str(_PLAYWRIGHT_USER_DATA),
headless=False,
user_agent=_PLAYWRIGHT_UA,
locale="ru-RU",
timezone_id="Asia/Yekaterinburg",
)
page = await context.new_page()
try:
for i, row in enumerate(sample, start=1):
status = "ok"
err: str | None = None
res: YandexReverseResult | None = None
stop_batch = False
try:
res = await reverse_via_playwright(row.lat, row.lon, page)
except YandexBlockedError as e:
status = "blocked"
err = str(e)
stop_batch = True
except Exception as e:
status = "error"
err = f"playwright: {e!s}"
distance = None
street_diff: bool | None = None
if res is not None and status == "ok":
if res.address is None:
status = "no_match"
else:
if res.snapped_lat is not None and res.snapped_lon is not None:
distance = _distance_meters(
db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
)
last_distance = distance
street_diff = _street_differs(row.address, res.address)
try:
with db.begin_nested():
_insert_audit_row(
db,
house_id=row.id,
batch=batch,
district=row.district,
original_address=row.address,
original_lat=row.lat,
original_lon=row.lon,
snapped_address=res.address if res else None,
snapped_lat=res.snapped_lat if res else None,
snapped_lon=res.snapped_lon if res else None,
distance_m=distance,
street_differs=street_diff,
audit_status=status,
error_message=err,
raw_payload=res.raw if res else None,
)
db.commit()
processed += 1
except Exception as e:
db.rollback()
logger.warning("insert failed for house_id=%s: %s", row.id, e)
if stop_batch:
logger.error(
"Yandex CAPTCHA detected at position %d/%d (house_id=%s). "
"Stopping batch — re-run with same --batch to resume.",
i,
len(sample),
row.id,
)
break
if i % 10 == 0:
logger.info(
"progress %d/%d, mode=playwright, last_distance=%s",
i,
len(sample),
f"{last_distance:.1f}m" if last_distance is not None else "n/a",
)
# Random delay 4-7s between requests — keeps us under Yandex's
# heuristic rate limit while still finishing 200 rows in <30min.
# Skip the wait on the last iteration (no next request to space).
if i < len(sample):
await asyncio.sleep(random.uniform(4.0, 7.0))
finally:
await context.close()
return processed
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
"""argparse setup, factored out for testability."""
p = argparse.ArgumentParser(
description="Phase 1 audit — houses.address vs Yandex reverse geocode.",
)
p.add_argument(
"--batch",
default=f"{date.today().isoformat()}_run1",
help="Audit batch label. Same batch re-run skips already-processed houses.",
)
p.add_argument(
"--limit-per-district",
type=int,
default=25,
help="Houses to sample per district (default 25 → ~200 total for EKB).",
)
p.add_argument(
"--mode",
choices=("auto", "api", "playwright"),
default="auto",
help="auto = API if YANDEX_GEOCODER_API_KEY set, else playwright.",
)
return p.parse_args(argv)
async def main(argv: list[str] | None = None) -> int:
"""CLI entry point. Returns the number of rows processed this run."""
args = _parse_args(argv)
api_key = os.environ.get("YANDEX_GEOCODER_API_KEY")
mode = _resolve_mode(args.mode, api_key)
if mode == "api" and not api_key:
raise SystemExit("mode=api requested but YANDEX_GEOCODER_API_KEY is not set")
logger.info(
"starting audit batch=%s mode=%s limit_per_district=%d",
args.batch,
mode,
args.limit_per_district,
)
db = SessionLocal()
try:
sample = _load_sample(db, args.limit_per_district)
logger.info("loaded sample: %d houses", len(sample))
# Resume support — drop already-processed house_ids.
done = _already_processed_ids(db, args.batch)
if done:
logger.info(
"resuming batch %s: %d rows already processed, %d remaining",
args.batch,
len(done),
len(sample) - sum(1 for s in sample if s.id in done),
)
remaining = [s for s in sample if s.id not in done]
if not remaining:
logger.info("nothing to do — batch %s is complete", args.batch)
return 0
if mode == "api":
n = await _run_api_mode(db, remaining, args.batch, api_key or "")
else:
n = await _run_playwright_mode(db, remaining, args.batch)
logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode)
return n
finally:
db.close()
if __name__ == "__main__": # pragma: no cover
asyncio.run(main())

47
tradein-mvp/backend/scripts/audit_address_sample.sql
View file

@ -1,47 +0,0 @@
-- audit_address_sample.sql
-- Random sample of EKB houses for the address-mismatch audit (issue #582).
--
-- Strategy:
-- 1. Filter to houses with non-null lat/lon and non-empty address.
-- 2. Random shuffle via `ORDER BY random()` — repeatable enough for spot
-- sampling without needing a stable PRNG seed (the audit table dedupes
-- via UNIQUE (house_id, audit_batch), so re-running gives idempotent
-- results regardless of which rows land in the sample first).
-- 3. Cap the result at :limit_per_district * 8 rows — keeps the bind-param
-- contract compatible with the old stratified sampler (`:limit_per_district`
-- is still honored, just multiplied by the assumed 8-district count).
--
-- Why no spatial stratification anymore:
-- The previous version JOINed to `gendesign_ekb_districts_geom` (FDW
-- polygon table) to bucket houses by admin district. That join is fine on
-- prod where FDW is wired, but it adds a dependency we don't need for
-- Phase 2-3 (backfill + canonical reverse). Aggregation by district at
-- report time still works — we re-derive district during the audit via
-- spatial containment in the report SQL when needed.
--
-- Bind param:
-- :limit_per_district — kept for back-compat with the audit driver.
-- Effective sample size = :limit_per_district * 8 (e.g. 25 → 200).
--
-- Columns returned:
-- id, address, lat, lon, district
-- `district` is always NULL here — the audit driver will reverse-derive it
-- from Yandex Geocoder response (Yandex returns admin component) or leave
-- it NULL if not present in the response.
--
-- NB: uses CAST(:x AS int) per project sql.md rule (psycopg v3 ignores ::type
-- after bind params).
SELECT
h.id,
h.address,
h.lat,
h.lon,
NULL::text AS district
FROM houses h
WHERE h.lat IS NOT NULL
AND h.lon IS NOT NULL
AND h.address IS NOT NULL
AND length(trim(h.address)) > 0
ORDER BY random()
LIMIT CAST(:limit_per_district AS int) * 8;

619
tradein-mvp/backend/scripts/backfill_house_coords.py
View file

@ -1,619 +0,0 @@
"""Forward-geocode houses through Yandex Geocoder API to backfill lat/lon
and canonical address, plus optional reverse audit of already-geocoded houses.
Phase 2-3 of Forgejo issue #582. Two modes (mutually exclusive):
1. Backfill (default) — for the ~4141 rows WHERE lat IS NULL OR lon IS NULL:
forward-geocode `houses.address` → snap to a Yandex `house`-precision
point, UPDATE houses with the new lat/lon + canonical address payload,
and write an `address_mismatch_audit` row with `audit_status='backfill'`.
2. Audit-only (--audit-only) — for the ~4452 rows that already have coords:
reverse-geocode (lat, lon) → snapped point + canonical address, compute
ST_Distance vs stored coords, write an `address_mismatch_audit` row with
status 'ok' (≤50m) or 'mismatch' (>50m). Does NOT touch houses.
Design choices:
- **Per-row SAVEPOINT** (`db.begin_nested()`): a single Yandex/PostGIS error
must not nuke the entire batch. Per backend.md, never use bare rollback
inside a loop.
- **Resumable** via UNIQUE (house_id, audit_batch). Re-running the same
--batch label skips already-processed houses, so a partial run can be
picked up after CAPTCHA / network blip / 25k/day quota hit.
- **Precision filter**: backfill skips matches with precision in
('street', 'other', 'range', 'near', None) — those are too imprecise for
comparable-listing spatial queries and would silently degrade matching
recall. The audit row still records what Yandex returned for forensics.
- **Rate limit**: 50ms between calls (~20 req/sec, well under Yandex's
25 req/sec service limit). Backfill mode runs single-threaded.
- **Daily quota**: 4141 backfill + 4452 audit ≈ 8.6k requests. Free Geocoder
tier is 25k/day → comfortable buffer for retries.
Usage:
YANDEX_GEOCODER_API_KEY=xxx \\
DATABASE_URL=postgresql+psycopg://... \\
python -m scripts.backfill_house_coords --batch 2026-05-27_backfill
# Audit-only on the 4452 already-geocoded houses
python -m scripts.backfill_house_coords --batch 2026-05-27_audit \\
--audit-only --limit 500
Outputs:
- Backfill mode: UPDATE rows in `houses`, INSERT rows in
`address_mismatch_audit` with status 'backfill' / 'no_match' / 'imprecise'.
- Audit mode: INSERT rows in `address_mismatch_audit` with status 'ok' /
'mismatch' / 'no_match' / 'error'.
- Per-batch progress is logged every 25 rows.
"""
from __future__ import annotations
import argparse
import asyncio
import json
import logging
import os
from dataclasses import dataclass
from datetime import date
from pathlib import Path
from typing import Any
import httpx
from sqlalchemy import text
from sqlalchemy.orm import Session
# Allow running both as `python -m scripts.backfill_house_coords` (preferred)
# and as a stand-alone file. Mirrors the audit_address_mismatch import dance.
try:
from app.core.db import SessionLocal # type: ignore[import-not-found]
except ImportError: # pragma: no cover — fallback for adhoc invocation
import sys
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from app.core.db import SessionLocal
try:
from scripts._yandex_reverse import ( # type: ignore[import-not-found]
YandexReverseResult,
forward_via_api,
reverse_via_api,
)
except ImportError:
from _yandex_reverse import ( # type: ignore[no-redef]
YandexReverseResult,
forward_via_api,
reverse_via_api,
)
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s %(message)s",
)
logger = logging.getLogger("backfill_house_coords")
# Yandex Geocoder service limits per docs (as of 2026-05):
# - 25k requests/day free tier
# - 25 requests/sec sustained
# 50ms between calls = ~20 req/sec, leaving headroom for connection ramp-up.
_REQUEST_DELAY_S = 0.05
# Precision values we ACCEPT for backfill — anything else means Yandex didn't
# resolve to a specific building, and writing the result back into houses
# would degrade matching recall.
# `exact` → match found at the exact address (best case)
# `number` → house number matched, but unit/entrance unspecified (acceptable)
# `near` / `range` / `street` / `other` / None → skipped (logged for analysis).
_BACKFILL_OK_PRECISION = frozenset({"exact", "number"})
# Audit threshold per issue #582 — distances above this flag a "mismatch"
# (the row still goes in the audit table, just with status='mismatch' for
# the report SQL to bucket separately).
_MISMATCH_DISTANCE_M = 50.0
# ---------------------------------------------------------------------------
# Domain types
# ---------------------------------------------------------------------------
@dataclass
class HouseRow:
"""One house from the source query — minimal fields needed for geocode."""
id: int
address: str
lat: float | None
lon: float | None
# ---------------------------------------------------------------------------
# Source-row queries
# ---------------------------------------------------------------------------
def _select_houses_without_coords(db: Session, limit: int | None) -> list[HouseRow]:
"""Pull houses needing forward geocode (lat IS NULL OR lon IS NULL).
Skips rows with empty address — there's nothing to geocode there, they
need a separate cleanup pass.
"""
sql = (
"SELECT id, address, lat, lon "
"FROM houses "
"WHERE (lat IS NULL OR lon IS NULL) "
" AND address IS NOT NULL "
" AND length(trim(address)) > 0 "
"ORDER BY id"
)
if limit is not None:
sql += " LIMIT CAST(:limit AS int)"
rows = db.execute(text(sql), {"limit": limit}).mappings().all()
else:
rows = db.execute(text(sql)).mappings().all()
return [
HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows
]
def _select_houses_with_coords(db: Session, limit: int | None) -> list[HouseRow]:
"""Pull houses needing reverse audit (both lat AND lon present)."""
sql = (
"SELECT id, address, lat, lon "
"FROM houses "
"WHERE lat IS NOT NULL "
" AND lon IS NOT NULL "
" AND address IS NOT NULL "
" AND length(trim(address)) > 0 "
"ORDER BY id"
)
if limit is not None:
sql += " LIMIT CAST(:limit AS int)"
rows = db.execute(text(sql), {"limit": limit}).mappings().all()
else:
rows = db.execute(text(sql)).mappings().all()
return [
HouseRow(id=r["id"], address=r["address"], lat=r["lat"], lon=r["lon"]) for r in rows
]
def _already_processed_ids(db: Session, batch: str) -> set[int]:
"""house_ids already in address_mismatch_audit for this batch → skip set."""
rows = db.execute(
text("SELECT house_id FROM address_mismatch_audit WHERE audit_batch = CAST(:b AS text)"),
{"b": batch},
).all()
return {r[0] for r in rows}
# ---------------------------------------------------------------------------
# Distance helper — PostGIS, lon/lat order
# ---------------------------------------------------------------------------
def _distance_meters(
db: Session, olat: float, olon: float, slat: float, slon: float
) -> float | None:
"""Great-circle distance (meters) via PostGIS geography type.
Lifted from `audit_address_mismatch.py` to keep the two scripts using
the same authority for distance computation. ST_MakePoint takes lon
first per PostGIS convention.
"""
row = db.execute(
text(
"SELECT ST_Distance("
" ST_SetSRID(ST_MakePoint(CAST(:olon AS double precision), "
" CAST(:olat AS double precision)), 4326)::geography, "
" ST_SetSRID(ST_MakePoint(CAST(:slon AS double precision), "
" CAST(:slat AS double precision)), 4326)::geography"
") AS m"
),
{"olat": olat, "olon": olon, "slat": slat, "slon": slon},
).first()
if row is None or row[0] is None:
return None
return float(row[0])
# ---------------------------------------------------------------------------
# DB writers
# ---------------------------------------------------------------------------
def _update_house_coords(
db: Session,
*,
house_id: int,
lat: float,
lon: float,
payload: dict[str, Any],
) -> None:
"""UPDATE houses SET lat/lon + merge yandex_geocode into raw_payload.
The `houses_set_geom_trg` BEFORE UPDATE trigger (009_houses.sql) maintains
`geom` automatically when lat/lon change, so we don't need to set geom
explicitly here. `raw_payload || jsonb_build_object(...)` is the idiomatic
psycopg-safe way to merge — single ALTER, no read-modify-write race.
"""
db.execute(
text(
"UPDATE houses "
" SET lat = CAST(:lat AS double precision), "
" lon = CAST(:lon AS double precision), "
" raw_payload = COALESCE(raw_payload, '{}'::jsonb) "
" || jsonb_build_object('yandex_geocode', "
" CAST(:payload AS jsonb)) "
" WHERE id = CAST(:id AS bigint)"
),
{"id": house_id, "lat": lat, "lon": lon, "payload": json.dumps(payload)},
)
def _insert_audit_row(
db: Session,
*,
house_id: int,
batch: str,
original_address: str | None,
original_lat: float | None,
original_lon: float | None,
snapped_address: str | None,
snapped_lat: float | None,
snapped_lon: float | None,
distance_m: float | None,
audit_status: str,
error_message: str | None,
raw_payload: dict[str, Any] | None,
) -> None:
"""INSERT … ON CONFLICT DO NOTHING into address_mismatch_audit.
Same column shape as `audit_address_mismatch._insert_audit_row` but the
`district` and `street_differs` fields are left NULL — backfill/audit
here doesn't have a stratification basis and we let the report SQL
derive district at query time if needed (via Yandex address parse).
Caller wraps in `begin_nested()` per backend.md SAVEPOINT pattern.
"""
db.execute(
text(
"INSERT INTO address_mismatch_audit ("
" house_id, audit_batch, district,"
" original_address, original_lat, original_lon,"
" snapped_address, snapped_lat, snapped_lon,"
" distance_m, street_differs,"
" audit_status, error_message, raw_payload"
") VALUES ("
" CAST(:house_id AS bigint), CAST(:batch AS text), NULL,"
" :original_address, :original_lat, :original_lon,"
" :snapped_address, :snapped_lat, :snapped_lon,"
" :distance_m, NULL,"
" CAST(:audit_status AS text), :error_message,"
" CAST(:raw_payload AS jsonb)"
") ON CONFLICT (house_id, audit_batch) DO NOTHING"
),
{
"house_id": house_id,
"batch": batch,
"original_address": original_address,
"original_lat": original_lat,
"original_lon": original_lon,
"snapped_address": snapped_address,
"snapped_lat": snapped_lat,
"snapped_lon": snapped_lon,
"distance_m": distance_m,
"audit_status": audit_status,
"error_message": error_message,
"raw_payload": json.dumps(raw_payload) if raw_payload is not None else None,
},
)
# ---------------------------------------------------------------------------
# Backfill loop (forward geocode, lat IS NULL houses)
# ---------------------------------------------------------------------------
def _classify_backfill_status(res: YandexReverseResult | None) -> str:
"""Translate a forward-geocode result into an audit_status value.
'backfill' — Yandex returned a precise hit, lat/lon will be written.
'imprecise' — match returned but precision is too low (street/other/...).
'no_match' — Yandex returned an empty featureMember.
'error' — handled by the caller's exception branch.
"""
if res is None or res.address is None:
return "no_match"
if res.precision not in _BACKFILL_OK_PRECISION:
return "imprecise"
if res.snapped_lat is None or res.snapped_lon is None:
return "no_match"
return "backfill"
async def _run_backfill_mode(
db: Session, sample: list[HouseRow], batch: str, api_key: str
) -> int:
"""Forward-geocode each house, UPDATE coords on precise hits, audit-log all."""
processed = 0
updated = 0
n_imprecise = 0
n_no_match = 0
n_error = 0
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
for i, row in enumerate(sample, start=1):
status = "backfill"
err: str | None = None
res: YandexReverseResult | None = None
try:
res = await forward_via_api(row.address, api_key, client=client)
except httpx.HTTPError as e:
status = "error"
err = f"http_error: {e!s}"
n_error += 1
except Exception as e: # pragma: no cover — defensive
status = "error"
err = f"unhandled: {e!s}"
n_error += 1
if status != "error":
status = _classify_backfill_status(res)
if status == "imprecise":
n_imprecise += 1
elif status == "no_match":
n_no_match += 1
try:
with db.begin_nested():
if status == "backfill" and res is not None and res.snapped_lat is not None:
# safe: status='backfill' guarantees snapped_lat/lon non-None.
assert res.snapped_lon is not None
_update_house_coords(
db,
house_id=row.id,
lat=res.snapped_lat,
lon=res.snapped_lon,
payload={
"address": res.address,
"precision": res.precision,
"kind": res.kind,
"batch": batch,
"source": "yandex_geocoder_api",
},
)
updated += 1
_insert_audit_row(
db,
house_id=row.id,
batch=batch,
original_address=row.address,
original_lat=row.lat,
original_lon=row.lon,
snapped_address=res.address if res else None,
snapped_lat=res.snapped_lat if res else None,
snapped_lon=res.snapped_lon if res else None,
distance_m=None,
audit_status=status,
error_message=err,
raw_payload=res.raw if res else None,
)
# Per-row commit so resume picks up exactly where we crashed.
db.commit()
processed += 1
except Exception as e:
db.rollback()
logger.warning("backfill insert failed for house_id=%s: %s", row.id, e)
if i % 25 == 0:
logger.info(
"backfill progress %d/%d updated=%d imprecise=%d no_match=%d error=%d",
i,
len(sample),
updated,
n_imprecise,
n_no_match,
n_error,
)
# Yandex 25 req/sec → 50ms between calls is plenty of headroom.
if i < len(sample):
await asyncio.sleep(_REQUEST_DELAY_S)
logger.info(
"backfill done: processed=%d updated=%d imprecise=%d no_match=%d error=%d",
processed,
updated,
n_imprecise,
n_no_match,
n_error,
)
return processed
# ---------------------------------------------------------------------------
# Audit-only loop (reverse geocode, lat IS NOT NULL houses)
# ---------------------------------------------------------------------------
async def _run_audit_mode(
db: Session, sample: list[HouseRow], batch: str, api_key: str
) -> int:
"""Reverse-geocode each house, compute distance, audit-log status/mismatch."""
processed = 0
n_ok = 0
n_mismatch = 0
n_no_match = 0
n_error = 0
async with httpx.AsyncClient(timeout=httpx.Timeout(10.0)) as client:
for i, row in enumerate(sample, start=1):
# Type-narrow: audit mode only feeds rows with non-null coords.
assert row.lat is not None and row.lon is not None
status = "ok"
err: str | None = None
res: YandexReverseResult | None = None
try:
res = await reverse_via_api(row.lat, row.lon, api_key, client=client)
except httpx.HTTPError as e:
status = "error"
err = f"http_error: {e!s}"
n_error += 1
except Exception as e: # pragma: no cover — defensive
status = "error"
err = f"unhandled: {e!s}"
n_error += 1
distance = None
if res is not None and status == "ok":
if res.address is None:
status = "no_match"
n_no_match += 1
else:
if res.snapped_lat is not None and res.snapped_lon is not None:
distance = _distance_meters(
db, row.lat, row.lon, res.snapped_lat, res.snapped_lon
)
if distance is not None and distance > _MISMATCH_DISTANCE_M:
status = "mismatch"
n_mismatch += 1
else:
n_ok += 1
else:
n_ok += 1
try:
with db.begin_nested():
_insert_audit_row(
db,
house_id=row.id,
batch=batch,
original_address=row.address,
original_lat=row.lat,
original_lon=row.lon,
snapped_address=res.address if res else None,
snapped_lat=res.snapped_lat if res else None,
snapped_lon=res.snapped_lon if res else None,
distance_m=distance,
audit_status=status,
error_message=err,
raw_payload=res.raw if res else None,
)
db.commit()
processed += 1
except Exception as e:
db.rollback()
logger.warning("audit insert failed for house_id=%s: %s", row.id, e)
if i % 25 == 0:
logger.info(
"audit progress %d/%d ok=%d mismatch=%d no_match=%d error=%d",
i,
len(sample),
n_ok,
n_mismatch,
n_no_match,
n_error,
)
if i < len(sample):
await asyncio.sleep(_REQUEST_DELAY_S)
logger.info(
"audit done: processed=%d ok=%d mismatch=%d no_match=%d error=%d",
processed,
n_ok,
n_mismatch,
n_no_match,
n_error,
)
return processed
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
"""argparse setup, factored out for testability."""
p = argparse.ArgumentParser(
description=(
"Phase 2-3 of issue #582 — backfill houses.lat/lon via Yandex forward "
"geocode, or audit already-geocoded houses via reverse geocode."
),
)
p.add_argument(
"--batch",
default=f"{date.today().isoformat()}_backfill",
help="Audit batch label. Same batch re-run skips already-processed houses.",
)
p.add_argument(
"--audit-only",
action="store_true",
help=(
"Run reverse-geocode audit on houses WITH coords instead of forward "
"backfill on houses WITHOUT coords. Does not modify the houses table."
),
)
p.add_argument(
"--limit",
type=int,
default=None,
help=(
"Optional cap on source-row count. Useful for canary runs "
"(e.g. --limit 100 before letting the full 4k loose)."
),
)
return p.parse_args(argv)
async def main(argv: list[str] | None = None) -> int:
"""CLI entry point. Returns the number of rows processed this run."""
args = _parse_args(argv)
api_key = os.environ.get("YANDEX_GEOCODER_API_KEY")
if not api_key:
raise SystemExit(
"YANDEX_GEOCODER_API_KEY is required — forward geocode is API-only."
)
mode = "audit" if args.audit_only else "backfill"
logger.info(
"starting batch=%s mode=%s limit=%s",
args.batch,
mode,
args.limit if args.limit is not None else "all",
)
db = SessionLocal()
try:
if args.audit_only:
sample = _select_houses_with_coords(db, args.limit)
else:
sample = _select_houses_without_coords(db, args.limit)
logger.info("loaded source rows: %d", len(sample))
done = _already_processed_ids(db, args.batch)
if done:
logger.info(
"resuming batch %s: %d rows already processed",
args.batch,
len(done),
)
remaining = [s for s in sample if s.id not in done]
if not remaining:
logger.info("nothing to do — batch %s is complete for the loaded sample", args.batch)
return 0
if args.audit_only:
n = await _run_audit_mode(db, remaining, args.batch, api_key)
else:
n = await _run_backfill_mode(db, remaining, args.batch, api_key)
logger.info("done: processed=%d batch=%s mode=%s", n, args.batch, mode)
return n
finally:
db.close()
if __name__ == "__main__": # pragma: no cover
asyncio.run(main())

74
tradein-mvp/backend/tests/fixtures/yandex_geocode_sample.json vendored
View file

@ -1,74 +0,0 @@
{
"response": {
"GeoObjectCollection": {
"metaDataProperty": {
"GeocoderResponseMetaData": {
"request": "60.586,56.838",
"results": "1",
"found": "1"
}
},
"featureMember": [
{
"GeoObject": {
"metaDataProperty": {
"GeocoderMetaData": {
"precision": "exact",
"text": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
"kind": "house",
"Address": {
"country_code": "RU",
"formatted": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
"postal_code": "620075",
"Components": [
{"kind": "country", "name": "Россия"},
{"kind": "province", "name": "Уральский федеральный округ"},
{"kind": "province", "name": "Свердловская область"},
{"kind": "area", "name": "городской округ Екатеринбург"},
{"kind": "locality", "name": "Екатеринбург"},
{"kind": "street", "name": "улица Малышева"},
{"kind": "house", "name": "51"}
]
},
"AddressDetails": {
"Country": {
"AddressLine": "Россия, Свердловская область, Екатеринбург, улица Малышева, 51",
"CountryNameCode": "RU",
"CountryName": "Россия",
"AdministrativeArea": {
"AdministrativeAreaName": "Свердловская область",
"SubAdministrativeArea": {
"SubAdministrativeAreaName": "городской округ Екатеринбург",
"Locality": {
"LocalityName": "Екатеринбург",
"Thoroughfare": {
"ThoroughfareName": "улица Малышева",
"Premise": {
"PremiseNumber": "51",
"PostalCode": {"PostalCodeNumber": "620075"}
}
}
}
}
}
}
}
}
},
"name": "улица Малышева, 51",
"description": "Екатеринбург, Россия",
"boundedBy": {
"Envelope": {
"lowerCorner": "60.585217 56.837461",
"upperCorner": "60.587094 56.838547"
}
},
"Point": {
"pos": "60.586155 56.838004"
}
}
}
]
}
}
}

373
tradein-mvp/backend/tests/test_audit_address_mismatch.py
View file

@ -1,373 +0,0 @@
"""Unit tests for the Phase-1 address-mismatch audit (issue #582).
Coverage:
- `_first_street_token` / `_street_differs` — normalization-driven diff.
- `_distance_meters` — verified against a MagicMock'd DB that returns a
canned distance, plus a Haversine cross-check on the bind values to
catch lat/lon swaps.
- `reverse_via_api` via httpx MockTransport with the fixture file.
- `main()` resumability — call twice with the same batch, second call
inserts 0 (uses MagicMock DB session).
Why no real Postgres in unit tests:
The repo doesn't bundle pytest-postgresql / testcontainers and the existing
tests all use `MagicMock` for the DB. We follow that convention here. The
distance and SQL-level resumability are validated by:
- The Haversine cross-check (pure-Python expected ≈ PostGIS result for
same coords, see `test_distance_calc_matches_haversine`).
- Calling `main()` twice in `test_audit_script_resumable` — first run
inserts N rows, second run sees the same set of house_ids in the
"already processed" query and processes 0.
"""
from __future__ import annotations
import json
import math
import os
from pathlib import Path
from unittest.mock import AsyncMock, MagicMock, patch
# Settings requires DATABASE_URL at init time — set dummy DSN before any
# `app.*` import (same pattern as test_cian_valuation.py).
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
import httpx
import pytest
from scripts._yandex_reverse import (
YandexBlockedError,
YandexReverseResult,
_parse_api_payload,
reverse_via_api,
)
from scripts.audit_address_mismatch import (
SampleRow,
_distance_meters,
_first_street_token,
_resolve_mode,
_run_api_mode,
_street_differs,
main,
)
_FIXTURES = Path(__file__).parent / "fixtures"
# ---------------------------------------------------------------------------
# _first_street_token / _street_differs
# ---------------------------------------------------------------------------
def test_normalize_address_street_token_basic():
"""First identifying token of a normalized address — skips street type."""
assert _first_street_token("ул Малышева 51") == "малышева"
def test_normalize_address_street_token_skips_leading_numbers():
"""Numeric tokens are skipped — the street name carries identity."""
# No type prefix → first non-numeric token is the street name itself.
assert _first_street_token("123 Постовского") == "постовского"
def test_normalize_address_street_token_handles_none():
assert _first_street_token(None) is None
assert _first_street_token("") is None
def test_street_differs_true_when_streets_differ():
assert _street_differs("ул Малышева 51", "ул Ленина 51") is True
def test_street_differs_false_when_same_after_normalization():
# 'ул' expands to 'улица' on both sides → same first token.
assert _street_differs("ул Малышева 51", "улица Малышева, 51") is False
def test_street_differs_none_on_empty_side():
assert _street_differs(None, "ул Малышева 51") is None
assert _street_differs("ул Малышева 51", "") is None
# ---------------------------------------------------------------------------
# _distance_meters — MagicMock DB + Haversine cross-check
# ---------------------------------------------------------------------------
def _haversine_m(lat1: float, lon1: float, lat2: float, lon2: float) -> float:
"""Reference implementation for sanity-checking the PostGIS call."""
r = 6_371_000.0
p1 = math.radians(lat1)
p2 = math.radians(lat2)
dp = math.radians(lat2 - lat1)
dl = math.radians(lon2 - lon1)
a = math.sin(dp / 2) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2) ** 2
return 2 * r * math.asin(math.sqrt(a))
def test_distance_calc_passes_correct_bindings():
"""Test the helper passes lat/lon in correct order to the SQL bind names."""
db = MagicMock()
# PostGIS would return one row, single column (distance in meters).
db.execute.return_value.first.return_value = (123.45,)
out = _distance_meters(db, 56.838, 60.586, 56.840, 60.590)
assert out == 123.45
# Verify the bind dict — guard against lat/lon swap regressions.
args, _kwargs = db.execute.call_args
bound = args[1]
assert bound == {
"olat": 56.838,
"olon": 60.586,
"slat": 56.840,
"slon": 60.590,
}
def test_distance_calc_returns_none_when_postgis_null():
"""ST_Distance can return NULL — caller must propagate None, not 0."""
db = MagicMock()
db.execute.return_value.first.return_value = (None,)
assert _distance_meters(db, 56.0, 60.0, 56.0, 60.0) is None
def test_distance_calc_matches_haversine_within_tolerance():
"""Sanity check: if PostGIS returned 555.7m for a known pair, that's
within ~1% of the Haversine reference (PostGIS uses Vincenty on
geography which is slightly more accurate)."""
expected = _haversine_m(56.838, 60.586, 56.843, 60.591)
# Just assert reference is in a sensible range — proves the test helper
# works; the actual call is mocked.
assert 500 < expected < 700
# ---------------------------------------------------------------------------
# Yandex API: payload parsing + reverse_via_api with MockTransport
# ---------------------------------------------------------------------------
def test_yandex_parse_api_fixture():
"""Sanity check: parse the bundled fixture into a YandexReverseResult."""
data = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8"))
res = _parse_api_payload(data)
assert res.address is not None
assert "Малышева" in res.address
# Fixture Point.pos = "60.586155 56.838004" → lon then lat.
assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6)
assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6)
assert res.raw == data
def test_yandex_parse_api_no_match():
"""Empty featureMember → all-None result, raw still preserved."""
data = {"response": {"GeoObjectCollection": {"featureMember": []}}}
res = _parse_api_payload(data)
assert res.address is None
assert res.snapped_lat is None
assert res.snapped_lon is None
assert res.raw == data
async def test_yandex_reverse_api_mock():
"""End-to-end: reverse_via_api hits a MockTransport, returns parsed result."""
fixture = json.loads((_FIXTURES / "yandex_geocode_sample.json").read_text("utf-8"))
captured: dict[str, httpx.Request] = {}
def handler(request: httpx.Request) -> httpx.Response:
captured["req"] = request
return httpx.Response(200, json=fixture)
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport) as client:
res = await reverse_via_api(56.838004, 60.586155, "DUMMY_KEY", client=client)
assert res.address is not None and "Малышева" in res.address
# Verify the request shape — lon,lat order + apikey + kind=house.
req = captured["req"]
qs = dict(httpx.QueryParams(req.url.query))
assert qs["apikey"] == "DUMMY_KEY"
assert qs["geocode"] == "60.586155,56.838004"
assert qs["format"] == "json"
assert qs["kind"] == "house"
async def test_yandex_blocked_error_raised_on_captcha():
"""`reverse_via_playwright` must raise YandexBlockedError on captcha.
We mock the page object so we don't need an actual browser.
"""
from scripts._yandex_reverse import reverse_via_playwright
page = MagicMock()
page.goto = AsyncMock()
page.wait_for_load_state = AsyncMock()
page.query_selector = AsyncMock(
side_effect=lambda sel: MagicMock() if sel == ".CheckboxCaptcha" else None
)
page.evaluate = AsyncMock(return_value=[None, None])
with pytest.raises(YandexBlockedError):
await reverse_via_playwright(56.838, 60.586, page)
# ---------------------------------------------------------------------------
# Mode resolver
# ---------------------------------------------------------------------------
def test_resolve_mode_auto_with_key():
assert _resolve_mode("auto", "abc") == "api"
def test_resolve_mode_auto_without_key():
assert _resolve_mode("auto", None) == "playwright"
assert _resolve_mode("auto", "") == "playwright"
def test_resolve_mode_explicit_passes_through():
assert _resolve_mode("api", None) == "api"
assert _resolve_mode("playwright", "abc") == "playwright"
# ---------------------------------------------------------------------------
# Resumability — main() twice with same batch
# ---------------------------------------------------------------------------
def _make_db_mock(initial_sample: list[dict], processed_ids: set[int]):
"""Build a MagicMock SQLAlchemy session that:
- returns `initial_sample` for the sampling SQL (text() with limit_per_district)
- returns `processed_ids` for the resume SQL (text() with batch only)
- records INSERTs so the test can count them
"""
inserted: list[dict] = []
db = MagicMock()
db.begin_nested.return_value.__enter__ = lambda self: self
db.begin_nested.return_value.__exit__ = lambda self, *a: False
def execute_side_effect(sql, params=None):
sql_str = str(sql)
result = MagicMock()
if "FROM houses h" in sql_str or "houses_in_districts" in sql_str:
result.mappings.return_value.all.return_value = initial_sample
elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str:
# Resume query — returns list of (house_id,) tuples.
result.all.return_value = [(hid,) for hid in processed_ids]
elif "INSERT INTO address_mismatch_audit" in sql_str:
inserted.append(dict(params))
# Simulate ON CONFLICT DO NOTHING — track id locally for re-run.
processed_ids.add(params["house_id"])
result = MagicMock()
elif "ST_Distance" in sql_str:
result.first.return_value = (42.0,)
else:
result = MagicMock()
return result
db.execute.side_effect = execute_side_effect
db.commit = MagicMock()
db.rollback = MagicMock()
db.close = MagicMock()
return db, inserted
async def test_audit_script_resumable(monkeypatch):
"""Run main() twice with the same batch — second pass inserts 0."""
sample = [
{
"id": 1,
"address": "ул Малышева 51",
"lat": 56.838,
"lon": 60.586,
"district": "Кировский",
},
{"id": 2, "address": "ул Ленина 5", "lat": 56.840, "lon": 60.600, "district": "Ленинский"},
]
processed_ids: set[int] = set()
db, inserted = _make_db_mock(sample, processed_ids)
# Force API mode without needing a real key.
monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
fake_result = YandexReverseResult(
address="Россия, Екатеринбург, улица Малышева, 51",
snapped_lat=56.838004,
snapped_lon=60.586155,
raw={"ok": True},
)
with (
patch("scripts.audit_address_mismatch.SessionLocal", return_value=db),
patch(
"scripts.audit_address_mismatch.reverse_via_api",
new=AsyncMock(return_value=fake_result),
),
):
# First run — both rows processed.
n1 = await main(["--batch", "test_batch_1", "--mode", "api"])
assert n1 == 2
assert len(inserted) == 2
# Second run with same batch — nothing left to do.
inserted.clear()
n2 = await main(["--batch", "test_batch_1", "--mode", "api"])
assert n2 == 0
assert inserted == []
async def test_audit_script_api_mode_marks_error(monkeypatch):
"""When the reverse call raises, the row is still inserted with status=error."""
sample = [
{
"id": 99,
"address": "ул Малышева 51",
"lat": 56.838,
"lon": 60.586,
"district": "Кировский",
},
]
processed_ids: set[int] = set()
db, inserted = _make_db_mock(sample, processed_ids)
monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
with (
patch("scripts.audit_address_mismatch.SessionLocal", return_value=db),
patch(
"scripts.audit_address_mismatch.reverse_via_api",
new=AsyncMock(side_effect=httpx.HTTPError("boom")),
),
):
n = await main(["--batch", "err_batch", "--mode", "api"])
assert n == 1
assert len(inserted) == 1
assert inserted[0]["audit_status"] == "error"
assert "boom" in (inserted[0]["error_message"] or "")
# ---------------------------------------------------------------------------
# Internal _run_api_mode no-match path
# ---------------------------------------------------------------------------
async def test_api_mode_no_match_path():
"""If Yandex returns address=None, row goes in with status=no_match."""
sample = [SampleRow(id=7, address="ул X 1", lat=56.0, lon=60.0, district="Кировский")]
processed_ids: set[int] = set()
db, inserted = _make_db_mock([], processed_ids)
res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True})
with patch(
"scripts.audit_address_mismatch.reverse_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_api_mode(db, sample, "b1", "key")
assert n == 1
assert inserted[0]["audit_status"] == "no_match"
assert inserted[0]["snapped_address"] is None

510
tradein-mvp/backend/tests/test_backfill_house_coords.py
View file

@ -1,510 +0,0 @@
"""Unit tests for the Phase 2-3 backfill/audit script (issue #582).
Coverage:
- `forward_via_api` request shape — verifies geocode/format/locality bias.
- `_parse_api_payload` precision + kind extraction.
- `_classify_backfill_status` precision filter rules.
- `_update_house_coords` — UPDATE shape + raw_payload merge.
- `_run_backfill_mode` — happy path UPDATE + audit row, plus imprecise-skip.
- `_run_audit_mode` — ok / mismatch / no_match distinction.
- `main()` resumability — second pass on same batch inserts 0.
No real Postgres in unit tests (same convention as test_audit_address_mismatch).
DB is a MagicMock that records INSERT/UPDATE calls and routes SELECT side-effects.
"""
from __future__ import annotations
import json
import os
from unittest.mock import AsyncMock, MagicMock, patch
# Same dance as test_audit_address_mismatch — settings needs a DSN at import.
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
import httpx
import pytest
from scripts._yandex_reverse import (
YandexReverseResult,
_parse_api_payload,
forward_via_api,
)
from scripts.backfill_house_coords import (
HouseRow,
_classify_backfill_status,
_run_audit_mode,
_run_backfill_mode,
_update_house_coords,
main,
)
# ---------------------------------------------------------------------------
# forward_via_api — request shape
# ---------------------------------------------------------------------------
async def test_forward_api_request_shape():
"""Verify the GET param dict — address as `geocode`, kind=house, EKB bias."""
fixture = {
"response": {
"GeoObjectCollection": {
"featureMember": [
{
"GeoObject": {
"metaDataProperty": {
"GeocoderMetaData": {
"text": "Россия, Свердловская область, Екатеринбург, "
"улица Малышева, 51",
"precision": "exact",
"kind": "house",
}
},
"name": "улица Малышева, 51",
"Point": {"pos": "60.586155 56.838004"},
}
}
]
}
}
}
captured: dict[str, httpx.Request] = {}
def handler(request: httpx.Request) -> httpx.Response:
captured["req"] = request
return httpx.Response(200, json=fixture)
transport = httpx.MockTransport(handler)
async with httpx.AsyncClient(transport=transport) as client:
res = await forward_via_api("ул Малышева 51", "DUMMY_KEY", client=client)
assert res.address is not None and "Малышева" in res.address
assert res.precision == "exact"
assert res.kind == "house"
assert res.snapped_lon == pytest.approx(60.586155, abs=1e-6)
assert res.snapped_lat == pytest.approx(56.838004, abs=1e-6)
qs = dict(httpx.QueryParams(captured["req"].url.query))
assert qs["apikey"] == "DUMMY_KEY"
assert qs["geocode"] == "ул Малышева 51"
assert qs["format"] == "json"
assert qs["kind"] == "house"
# EKB locality bias for forward geocode — important so addresses without
# the city resolve to the correct Малышева (there's one in Moscow too).
assert "ll" in qs
assert "spn" in qs
# ---------------------------------------------------------------------------
# Precision / kind passthrough in _parse_api_payload
# ---------------------------------------------------------------------------
def test_parse_api_payload_propagates_precision_and_kind():
data = {
"response": {
"GeoObjectCollection": {
"featureMember": [
{
"GeoObject": {
"metaDataProperty": {
"GeocoderMetaData": {
"text": "ул Ленина 5",
"precision": "exact",
"kind": "house",
}
},
"name": "ул Ленина 5",
"Point": {"pos": "60.6 56.8"},
}
}
]
}
}
}
res = _parse_api_payload(data)
assert res.precision == "exact"
assert res.kind == "house"
# ---------------------------------------------------------------------------
# _classify_backfill_status — precision filter rules
# ---------------------------------------------------------------------------
def test_classify_backfill_status_exact_match():
res = YandexReverseResult(
address="ул Малышева 51",
snapped_lat=56.838,
snapped_lon=60.586,
precision="exact",
kind="house",
)
assert _classify_backfill_status(res) == "backfill"
def test_classify_backfill_status_number_match():
res = YandexReverseResult(
address="ул Ленина 5",
snapped_lat=56.840,
snapped_lon=60.600,
precision="number",
kind="house",
)
assert _classify_backfill_status(res) == "backfill"
def test_classify_backfill_status_street_is_imprecise():
res = YandexReverseResult(
address="ул Ленина",
snapped_lat=56.840,
snapped_lon=60.600,
precision="street",
kind="street",
)
assert _classify_backfill_status(res) == "imprecise"
def test_classify_backfill_status_other_is_imprecise():
res = YandexReverseResult(
address="Свердловская область",
snapped_lat=56.8,
snapped_lon=60.6,
precision="other",
kind="locality",
)
assert _classify_backfill_status(res) == "imprecise"
def test_classify_backfill_status_no_match():
res = YandexReverseResult(address=None, snapped_lat=None, snapped_lon=None)
assert _classify_backfill_status(res) == "no_match"
def test_classify_backfill_status_none():
assert _classify_backfill_status(None) == "no_match"
def test_classify_backfill_status_precision_ok_but_no_coords():
"""Defensive: precision=exact but snapped point missing → no_match, not backfill."""
res = YandexReverseResult(
address="ул Малышева 51",
snapped_lat=None,
snapped_lon=None,
precision="exact",
kind="house",
)
assert _classify_backfill_status(res) == "no_match"
# ---------------------------------------------------------------------------
# _update_house_coords — UPDATE shape verification
# ---------------------------------------------------------------------------
def test_update_house_coords_passes_bindings():
db = MagicMock()
_update_house_coords(
db,
house_id=42,
lat=56.838,
lon=60.586,
payload={"address": "ул Малышева 51", "precision": "exact"},
)
args, _kw = db.execute.call_args
sql_str = str(args[0])
binds = args[1]
assert "UPDATE houses" in sql_str
assert "raw_payload" in sql_str
assert "yandex_geocode" in sql_str
assert binds["id"] == 42
assert binds["lat"] == 56.838
assert binds["lon"] == 60.586
# payload bound as JSON string for CAST(:payload AS jsonb)
decoded = json.loads(binds["payload"])
assert decoded["address"] == "ул Малышева 51"
# ---------------------------------------------------------------------------
# DB mock helper — same approach as test_audit_address_mismatch
# ---------------------------------------------------------------------------
def _make_db_mock(
backfill_sample: list[dict] | None = None,
audit_sample: list[dict] | None = None,
processed_ids: set[int] | None = None,
distance_value: float = 12.5,
):
"""MagicMock DB that:
- returns `backfill_sample` for `lat IS NULL OR lon IS NULL` SELECT
- returns `audit_sample` for `lat IS NOT NULL` SELECT
- returns `processed_ids` for the resume SELECT
- records INSERTs and UPDATEs
- returns `distance_value` for ST_Distance calls
"""
backfill_sample = backfill_sample or []
audit_sample = audit_sample or []
processed_ids = processed_ids if processed_ids is not None else set()
inserted: list[dict] = []
updated: list[dict] = []
db = MagicMock()
db.begin_nested.return_value.__enter__ = lambda self: self
db.begin_nested.return_value.__exit__ = lambda self, *a: False
def execute_side_effect(sql, params=None):
sql_str = str(sql)
result = MagicMock()
if "FROM houses" in sql_str and "lat IS NULL OR lon IS NULL" in sql_str:
result.mappings.return_value.all.return_value = backfill_sample
elif "FROM houses" in sql_str and "lat IS NOT NULL" in sql_str:
result.mappings.return_value.all.return_value = audit_sample
elif "FROM address_mismatch_audit" in sql_str and "house_id" in sql_str:
result.all.return_value = [(hid,) for hid in processed_ids]
elif "INSERT INTO address_mismatch_audit" in sql_str:
inserted.append(dict(params))
processed_ids.add(params["house_id"])
elif "UPDATE houses" in sql_str:
updated.append(dict(params))
elif "ST_Distance" in sql_str:
result.first.return_value = (distance_value,)
return result
db.execute.side_effect = execute_side_effect
db.commit = MagicMock()
db.rollback = MagicMock()
db.close = MagicMock()
return db, inserted, updated
# ---------------------------------------------------------------------------
# _run_backfill_mode — happy path + imprecise-skip
# ---------------------------------------------------------------------------
async def test_run_backfill_mode_writes_update_and_audit():
sample = [
HouseRow(id=1, address="ул Малышева 51", lat=None, lon=None),
]
db, inserted, updated = _make_db_mock()
res = YandexReverseResult(
address="Россия, Екатеринбург, улица Малышева, 51",
snapped_lat=56.838,
snapped_lon=60.586,
precision="exact",
kind="house",
raw={"ok": True},
)
with patch(
"scripts.backfill_house_coords.forward_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_backfill_mode(db, sample, "b1", "KEY")
assert n == 1
assert len(updated) == 1
assert updated[0]["id"] == 1
assert updated[0]["lat"] == 56.838
assert updated[0]["lon"] == 60.586
assert len(inserted) == 1
assert inserted[0]["audit_status"] == "backfill"
assert inserted[0]["snapped_address"] == "Россия, Екатеринбург, улица Малышева, 51"
async def test_run_backfill_mode_imprecise_skips_update():
"""precision='street' → audit row written with status=imprecise, no UPDATE."""
sample = [HouseRow(id=2, address="ул Ленина", lat=None, lon=None)]
db, inserted, updated = _make_db_mock()
res = YandexReverseResult(
address="ул Ленина",
snapped_lat=56.840,
snapped_lon=60.600,
precision="street",
kind="street",
raw={"oh_well": True},
)
with patch(
"scripts.backfill_house_coords.forward_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_backfill_mode(db, sample, "b2", "KEY")
assert n == 1
assert updated == []
assert len(inserted) == 1
assert inserted[0]["audit_status"] == "imprecise"
async def test_run_backfill_mode_no_match():
"""Yandex returns empty result → status=no_match, no UPDATE."""
sample = [HouseRow(id=3, address="несуществующая улица 99", lat=None, lon=None)]
db, inserted, updated = _make_db_mock()
res = YandexReverseResult(
address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True}
)
with patch(
"scripts.backfill_house_coords.forward_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_backfill_mode(db, sample, "b3", "KEY")
assert n == 1
assert updated == []
assert inserted[0]["audit_status"] == "no_match"
async def test_run_backfill_mode_http_error_marks_error():
sample = [HouseRow(id=4, address="ул X 1", lat=None, lon=None)]
db, inserted, updated = _make_db_mock()
with patch(
"scripts.backfill_house_coords.forward_via_api",
new=AsyncMock(side_effect=httpx.HTTPError("boom")),
):
n = await _run_backfill_mode(db, sample, "b4", "KEY")
assert n == 1
assert updated == []
assert inserted[0]["audit_status"] == "error"
assert "boom" in (inserted[0]["error_message"] or "")
# ---------------------------------------------------------------------------
# _run_audit_mode — ok / mismatch / no_match
# ---------------------------------------------------------------------------
async def test_run_audit_mode_ok_within_50m():
sample = [HouseRow(id=10, address="ул Малышева 51", lat=56.838, lon=60.586)]
db, inserted, _updated = _make_db_mock(distance_value=12.5)
res = YandexReverseResult(
address="Россия, Екатеринбург, улица Малышева, 51",
snapped_lat=56.838004,
snapped_lon=60.586155,
precision="exact",
kind="house",
raw={"r": 1},
)
with patch(
"scripts.backfill_house_coords.reverse_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_audit_mode(db, sample, "ba1", "KEY")
assert n == 1
assert inserted[0]["audit_status"] == "ok"
assert inserted[0]["distance_m"] == 12.5
async def test_run_audit_mode_mismatch_above_50m():
sample = [HouseRow(id=11, address="ул Ленина 5", lat=56.840, lon=60.600)]
db, inserted, _updated = _make_db_mock(distance_value=312.0)
res = YandexReverseResult(
address="Россия, Екатеринбург, улица Ленина, 7",
snapped_lat=56.841,
snapped_lon=60.601,
precision="exact",
kind="house",
raw={"r": 2},
)
with patch(
"scripts.backfill_house_coords.reverse_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_audit_mode(db, sample, "ba2", "KEY")
assert n == 1
assert inserted[0]["audit_status"] == "mismatch"
assert inserted[0]["distance_m"] == 312.0
async def test_run_audit_mode_no_match():
sample = [HouseRow(id=12, address="ул X 99", lat=56.0, lon=60.0)]
db, inserted, _updated = _make_db_mock()
res = YandexReverseResult(
address=None, snapped_lat=None, snapped_lon=None, raw={"empty": True}
)
with patch(
"scripts.backfill_house_coords.reverse_via_api",
new=AsyncMock(return_value=res),
):
n = await _run_audit_mode(db, sample, "ba3", "KEY")
assert n == 1
assert inserted[0]["audit_status"] == "no_match"
# ---------------------------------------------------------------------------
# Resumability — second pass on same batch inserts 0
# ---------------------------------------------------------------------------
async def test_main_resumable_skips_processed(monkeypatch):
"""Run main() twice with same batch — second pass processes nothing."""
backfill_sample = [
{"id": 1, "address": "ул Малышева 51", "lat": None, "lon": None},
{"id": 2, "address": "ул Ленина 5", "lat": None, "lon": None},
]
processed_ids: set[int] = set()
db, inserted, updated = _make_db_mock(
backfill_sample=backfill_sample, processed_ids=processed_ids
)
monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
fake = YandexReverseResult(
address="ул Малышева 51",
snapped_lat=56.838,
snapped_lon=60.586,
precision="exact",
kind="house",
raw={"ok": True},
)
with (
patch("scripts.backfill_house_coords.SessionLocal", return_value=db),
patch(
"scripts.backfill_house_coords.forward_via_api",
new=AsyncMock(return_value=fake),
),
):
n1 = await main(["--batch", "resume_test"])
assert n1 == 2
assert len(inserted) == 2
assert len(updated) == 2
inserted.clear()
updated.clear()
n2 = await main(["--batch", "resume_test"])
assert n2 == 0
assert inserted == []
assert updated == []
async def test_main_requires_api_key(monkeypatch):
"""Without YANDEX_GEOCODER_API_KEY the script exits cleanly."""
monkeypatch.delenv("YANDEX_GEOCODER_API_KEY", raising=False)
with pytest.raises(SystemExit):
await main(["--batch", "no_key"])
async def test_main_audit_only_flag_routes_to_audit_loop(monkeypatch):
"""--audit-only switches sample query + loop, no UPDATE expected."""
audit_sample = [
{"id": 50, "address": "ул Малышева 51", "lat": 56.838, "lon": 60.586},
]
db, inserted, updated = _make_db_mock(audit_sample=audit_sample, distance_value=8.0)
monkeypatch.setenv("YANDEX_GEOCODER_API_KEY", "TEST_KEY")
fake = YandexReverseResult(
address="Россия, Екатеринбург, улица Малышева, 51",
snapped_lat=56.838004,
snapped_lon=60.586155,
precision="exact",
kind="house",
raw={"r": 1},
)
with (
patch("scripts.backfill_house_coords.SessionLocal", return_value=db),
patch(
"scripts.backfill_house_coords.reverse_via_api",
new=AsyncMock(return_value=fake),
),
):
n = await main(["--batch", "audit_run", "--audit-only"])
assert n == 1
assert updated == [] # audit mode never updates houses
assert inserted[0]["audit_status"] == "ok"
assert inserted[0]["distance_m"] == 8.0

91
tradein-mvp/backend/tests/test_geocoder_nominatim_lookup.py Normal file
View file

@ -0,0 +1,91 @@
"""Тесты `_nominatim_lookup` — city реально доходит до исходящего HTTP-запроса.
#2593 (часть 3): Yandex Geocoder полностью удалён из проекта, вместе с ним ушли
`_yandex_reverse.py` + `tests/test_audit_address_mismatch.py` +
`tests/test_backfill_house_coords.py` — они были единственной проверкой, что
`city`/`city_hint` реально передаётся во внешний геокодер, а не только влияет на
cache-ключ (см. `tests/test_geocoder_city_hint.py`, который мокает
`_nominatim_lookup`/`_nominatim_suggest` целиком и потому не видит их внутренности).
Nominatim теперь единственный живой внешний провайдер (`_nominatim_lookup`
docstring, `app/services/geocoder.py`) — этот файл закрывает получившуюся дыру:
мокает HTTP-транспорт (`httpx.MockTransport`, паттерн из `test_geocoder_bbox.py` /
`tests/services/test_dadata.py`) и проверяет параметр `q` реального исходящего
GET-запроса к `nominatim.openstreetmap.org/search`.
"""
from __future__ import annotations
import os
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost:5432/test")
from unittest.mock import patch
import httpx
from app.services.geocoder import _nominatim_lookup
# EKB-центр (Плотинка) — внутри tight EKB bbox, `_nominatim_query` его примет
# без похода во второй (typo-variant) тир.
_SAMPLE_ITEM = {
"lat": "56.838",
"lon": "60.605",
"class": "building",
"display_name": "ул. Малышева, 30, Екатеринбург",
"address": {"state": "Свердловская область"},
}
# Snapshot реального httpx.AsyncClient ДО patch'а — фабрика ниже использует именно
# его с подменённым transport (паттерн tests/services/test_dadata.py: избегает
# recursion, если бы `httpx.AsyncClient` патчился поверх самого себя).
_REAL_ASYNC_CLIENT = httpx.AsyncClient
def _async_client_factory(transport: httpx.MockTransport):
def factory(*_: object, **__: object) -> httpx.AsyncClient:
return _REAL_ASYNC_CLIENT(transport=transport)
return factory
def _capturing_transport(captured_q: list[str]) -> httpx.MockTransport:
def handler(request: httpx.Request) -> httpx.Response:
captured_q.append(request.url.params.get("q", ""))
return httpx.Response(200, json=[_SAMPLE_ITEM])
return httpx.MockTransport(handler)
async def test_nominatim_lookup_sends_city_hint_in_query_param() -> None:
"""city_hint="Нижний Тагил" должен попасть в q= реального GET-запроса.
Регрессия, о которой явно предупреждает docstring `_nominatim_lookup` (#2580 C):
city_hint обязан влиять на сам запрос к провайдеру, не только на cache-ключ.
"""
captured_q: list[str] = []
transport = _capturing_transport(captured_q)
with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)):
result = await _nominatim_lookup("Ленина, 1", city_hint="Нижний Тагил")
assert captured_q, "запрос к Nominatim не был отправлен"
assert captured_q[0] == "Нижний Тагил, Ленина, 1"
assert result is not None
assert result.provider == "nominatim"
async def test_nominatim_lookup_no_city_sends_bare_address() -> None:
"""Без city_hint и без маркера города в тексте — q= остаётся bare-адресом.
Guard против регрессии в молчаливый дефолт на конкретный город (#2576/#2593)
— до фикса #2576 сюда молча подставлялся "Екатеринбург".
"""
captured_q: list[str] = []
transport = _capturing_transport(captured_q)
with patch("app.services.geocoder.httpx.AsyncClient", _async_client_factory(transport)):
result = await _nominatim_lookup("Малышева, 30")
assert captured_q == ["Малышева, 30"]
assert result is not None