feat(tradein): yandex address backfill — street → full address #554

Merged

lekss361 merged 1 commit from feat/tradein-yandex-address-backfill into main

2026-05-24 19:59:24 +00:00

lekss361 commented

2026-05-24 19:53:13 +00:00

Owner

Summary

Standalone script tradein-mvp/scripts/backfill-yandex-addresses-ekb.py enriches listings.address with house number for active EKB yandex listings whose address is currently street-only.

Why

3354 active yandex listings (87% of EKB pool) have addresses like улица Горького — Yandex valuation endpoint cant resolve them. Only 350/3704 (9.4%) had house numbers, limiting valuation sweep to ~123 distinct addresses.

Detail page <title> contains the full address:

<title>Купить квартиру ... — Екатеринбург, улица Горького, 36 — id 3990910150608393521</title>

Script fetches detail via curl_cffi chrome120 (plain httpx returns CSR shell), regex-extracts address from title, UPDATEs DB.

CLI

--limit N         cap for smoke (default: all)
--delay 3.0       sec between requests
--dry-run         no DB write

Smoke validated

5/5 enriched correctly in dry-run:
- улица Бориса Ельцина → Екатеринбург, улица Бориса Ельцина, 6
- улица Карла Маркса → Екатеринбург, улица Карла Маркса, 25
- улица Степана Разина → Екатеринбург, улица Степана Разина, 23
- etc.

Downstream

After full backfill (~3.5h at 3s delay × 3218 items), eligible pool for sweep-yandex-valuation-ekb.py --require-house-number grows from 123 → ~3000+ houses. Idempotent (sweep uses ON CONFLICT + skip-recent).

## Summary Standalone script `tradein-mvp/scripts/backfill-yandex-addresses-ekb.py` enriches `listings.address` with house number for active EKB yandex listings whose address is currently street-only. ## Why 3354 active yandex listings (87% of EKB pool) have addresses like `улица Горького` — Yandex valuation endpoint cant resolve them. Only 350/3704 (9.4%) had house numbers, limiting valuation sweep to ~123 distinct addresses. Detail page `<title>` contains the full address: ``` <title>Купить квартиру ... — Екатеринбург, улица Горького, 36 — id 3990910150608393521</title> ``` Script fetches detail via `curl_cffi chrome120` (plain httpx returns CSR shell), regex-extracts address from title, UPDATEs DB. ## CLI ``` --limit N cap for smoke (default: all) --delay 3.0 sec between requests --dry-run no DB write ``` ## Smoke validated - 5/5 enriched correctly in dry-run: - `улица Бориса Ельцина` → `Екатеринбург, улица Бориса Ельцина, 6` - `улица Карла Маркса` → `Екатеринбург, улица Карла Маркса, 25` - `улица Степана Разина` → `Екатеринбург, улица Степана Разина, 23` - etc. ## Downstream After full backfill (~3.5h at 3s delay × 3218 items), eligible pool for `sweep-yandex-valuation-ekb.py --require-house-number` grows from 123 → ~3000+ houses. Idempotent (sweep uses ON CONFLICT + skip-recent).

lekss361 added 1 commit 2026-05-24 19:53:14 +00:00

feat(tradein): backfill yandex listings.address with house number from detail page 7575999ad9

3354 active yandex listings on prod (region 66) had street-only addresses
like 'улица Горького' — Yandex valuation endpoint can't resolve them. Detail
page <title> contains the full address with house number (e.g. 'Екатеринбург,
улица Горького, 36'). Backfill script fetches /offer/<id>/ via curl_cffi
chrome120 (plain httpx returns CSR shell), parses <title>, updates DB.

Smoke verified: 5/5 enriched correctly. Idempotent (skip-recent + ON CONFLICT
in downstream sweep). Rate-limited 3s default. --dry-run for safe preview.

lekss361 commented

2026-05-24 19:58:32 +00:00

Author

Owner

Deep Code Review — verdict APPROVE

Summary

Status: APPROVE
Files: 1 (P2: 1 standalone ops script)
Lines: +173 / -0 · PR: #554
SHA verified: 7575999

Scope

Standalone one-shot CLI tradein-mvp/scripts/backfill-yandex-addresses-ekb.py that:

SELECTs active EKB yandex listings whose address lacks a house number (regex address ~ ',\s*\d+' negated).
Fetches each source_url via curl_cffi chrome120 AsyncSession.
Regex-extracts <title> full address (Екатеринбург + street + house).
UPDATEs listings.address if changed (skip-unchanged, skip-failed).

No backend/app code or DB schema changes. Self-contained ops script.

Correctness — clean

Idempotency: backfilled rows acquire comma+digit and are auto-skipped on re-run. Matches the same regex used in sweep-yandex-valuation-ekb.py (require_house_number=True), guaranteeing consistency between filters.
No regression on already-full addresses: filtered out at SELECT.
Regex sanity (tested locally):
- Екатеринбург, улица Горького, 36 → captured
- ЖК Корица, Екатеринбург, улица Бориса Ельцина, 6 → captures address skipping ЖК prefix (optional non-greedy group works)
- проспект Ленина, 50/1 → slash preserved
- Москва, улица Тверская, 1 → no match (Екатеринбург-anchored)
No match_or_create_house interaction (#527 signature change irrelevant — script only writes listings.address, no INSERT into houses).
No CAST issue: only :addr/:id params, no :x IS NOT NULL predicates → psycopg v3 trap N/A.
No SQL injection: parameterised UPDATE, regex literal is hard-coded.
Error handling: catches all fetch exceptions, counted as failed. HTTP non-200 reported. Logs flush=True for long-run monitor.

Performance

3218 items × ~3s delay × 0.85–1.3 jitter ≈ 3.5h, matches PR description. Single sequential session, acceptable for ops script.
One SessionLocal() per UPDATE (3354 connections) — slightly wasteful but trivial against 3s sleep per row.

Conventions

weasyprint stub block — same shape as sibling sweep-yandex-valuation-ekb.py.
sys.path injection identical to peer.
ruff-clean (line lengths ok, noqa: E402 where needed).

Minor observations (not blocking)

Theoretical edge: title ... — Екатеринбург, улица — id N (street only, missing house) would write back bare Екатеринбург, улица. Script doesn't validate that the extracted address itself contains house digits. Unlikely in practice (Yandex titles consistently include house) and smoke validated 5/5 clean. Could add a post-extract if not RE_HOUSE_NUM.search(addr): return None for safety, but acceptable as-is.
Double-em-dash titles (... — ЖК Foo — Екатеринбург, ...): the optional ЖК prefix group uses [^—,]*? so won't span an em-dash. If Yandex changes title format, script silently skips those rows (failed counter) rather than corrupting data — fail-safe.
Connection-per-update: could batch via single with SessionLocal() over the loop, but at 3s/req the overhead is invisible.

Test coverage

No unit tests. Acceptable for one-shot ops script with documented 5/5 smoke-dry-run validation in PR body.

Vault cross-check

Aligns with PR #533 sweep require_house_number=True policy (same regex).
Independent of #527 bootstrap-houses (no INSERT into houses).
Independent of #540 region prefix normalization (operates on full address string).

Verdict

APPROVE — focused, idempotent, conservative ops script. No security/correctness blockers. Auto-merging.

## Deep Code Review — verdict APPROVE ### Summary - Status: APPROVE - Files: 1 (P2: 1 standalone ops script) - Lines: +173 / -0 · PR: #554 - SHA verified: 7575999 ### Scope Standalone one-shot CLI `tradein-mvp/scripts/backfill-yandex-addresses-ekb.py` that: 1. SELECTs active EKB yandex listings whose `address` lacks a house number (regex `address ~ ',\s*\d+'` negated). 2. Fetches each `source_url` via `curl_cffi chrome120 AsyncSession`. 3. Regex-extracts `<title>` full address (Екатеринбург + street + house). 4. UPDATEs `listings.address` if changed (skip-unchanged, skip-failed). No backend/app code or DB schema changes. Self-contained ops script. ### Correctness — clean - **Idempotency**: backfilled rows acquire comma+digit and are auto-skipped on re-run. Matches the same regex used in `sweep-yandex-valuation-ekb.py` (`require_house_number=True`), guaranteeing consistency between filters. - **No regression on already-full addresses**: filtered out at SELECT. - **Regex sanity** (tested locally): - `Екатеринбург, улица Горького, 36` → captured - `ЖК Корица, Екатеринбург, улица Бориса Ельцина, 6` → captures address skipping ЖК prefix (optional non-greedy group works) - `проспект Ленина, 50/1` → slash preserved - `Москва, улица Тверская, 1` → no match (Екатеринбург-anchored) - **No `match_or_create_house` interaction** (#527 signature change irrelevant — script only writes `listings.address`, no INSERT into houses). - **No CAST issue**: only `:addr`/`:id` params, no `:x IS NOT NULL` predicates → psycopg v3 trap N/A. - **No SQL injection**: parameterised UPDATE, regex literal is hard-coded. - **Error handling**: catches all fetch exceptions, counted as `failed`. HTTP non-200 reported. Logs `flush=True` for long-run monitor. ### Performance - 3218 items × ~3s delay × 0.85–1.3 jitter ≈ 3.5h, matches PR description. Single sequential session, acceptable for ops script. - One `SessionLocal()` per UPDATE (3354 connections) — slightly wasteful but trivial against 3s sleep per row. ### Conventions - `weasyprint` stub block — same shape as sibling `sweep-yandex-valuation-ekb.py`. - `sys.path` injection identical to peer. - ruff-clean (line lengths ok, `noqa: E402` where needed). ### Minor observations (not blocking) 1. **Theoretical edge**: title `... — Екатеринбург, улица — id N` (street only, missing house) would write back bare `Екатеринбург, улица`. Script doesn't validate that the extracted address itself contains house digits. Unlikely in practice (Yandex titles consistently include house) and smoke validated 5/5 clean. Could add a post-extract `if not RE_HOUSE_NUM.search(addr): return None` for safety, but acceptable as-is. 2. **Double-em-dash titles** (`... — ЖК Foo — Екатеринбург, ...`): the optional ЖК prefix group uses `[^—,]*?` so won't span an em-dash. If Yandex changes title format, script silently skips those rows (failed counter) rather than corrupting data — fail-safe. 3. **Connection-per-update**: could batch via single `with SessionLocal()` over the loop, but at 3s/req the overhead is invisible. ### Test coverage No unit tests. Acceptable for one-shot ops script with documented 5/5 smoke-dry-run validation in PR body. ### Vault cross-check - Aligns with PR #533 sweep `require_house_number=True` policy (same regex). - Independent of #527 bootstrap-houses (no INSERT into houses). - Independent of #540 region prefix normalization (operates on full address string). ### Verdict APPROVE — focused, idempotent, conservative ops script. No security/correctness blockers. Auto-merging.