lekss361/gendesign

Fork 0

feat(tradein): estimator — Yandex Valuation as on-demand 6th source #473

Merged

lekss361 merged 1 commit from feat/tradein-estimator-yandex-integration into main

2026-05-23 14:14:50 +00:00

lekss361 commented

2026-05-23 14:10:59 +00:00

Owner

Motivation

Stage 8 of 9 for YandexRealtyScraper v1 (Wave 6). Adds Yandex Valuation tool (anonymous, no cookies — vs Cian Calculator's auth requirement) as on-demand enrichment source in estimate_quality. Mirrors the proven Avito IMV pattern: 24h cache via external_valuations, graceful degradation, idempotent history persistence.

Changes — `estimator.py`

Function	Purpose
`_yandex_valuation_cache_key(address, category, type)`	SHA256 cache key
`_get_or_fetch_yandex_valuation_cached(db, address, ...)`	Cache lookup via `external_valuations(source='yandex_valuation')`. Any error → `None` + log warning.
`_save_yandex_history_items(db, result)`	INSERT into `house_placement_history(source='yandex_valuation')` with synthesized `ext_item_id` (sha256 of address+date+area+floor+prices, first 32 chars). Idempotent via UNIQUE (source, ext_item_id). `house_id=NULL` — matching pipeline not wired into estimator yet.
`estimate_quality`	Calls Yandex Valuation after IMV block, before deals fetch. Adds `'yandex_valuation'` to `sources_used` on hit.

Constants

YANDEX_VALUATION_CACHE_TTL_HOURS = 24
YANDEX_VALUATION_DEFAULT_CATEGORY = "APARTMENT"
YANDEX_VALUATION_DEFAULT_TYPE = "SELL"

Why "6th source", not "7th"

The vault plan called this "7th source" assuming Cian Valuation was already wired into estimator as #6. Inspection of current estimator.py shows only Avito IMV (5th) integrated — Cian Valuation lives in the scraper layer but isn't called from estimate_quality yet. So Yandex slots in as the 6th source currently active in the estimator. Plan numbering is aspirational; the integration order is what matters.

Graceful degradation

Geocode failure → estimator returns _empty_estimate (Yandex block skipped — needs geo.full_address)
Yandex scrape failure → log warning + None, sources_used omits yandex_valuation
Cache DB error → log warning + proceeds to fetch
History save DB error → per-item try/except, partial success counted

Tests

tests/test_estimator_yandex_integration.py: 11 passed, 0 failed
tests/ -k "estimator and not slow": 15 passed (11 new + 4 IMV regression)
ruff: 0 new errors (5 pre-existing in estimator.py unrelated to this PR)

Coverage: cache key determinism + uniqueness, TTL constant, cache hit/miss paths, fetch error/empty handling, history item save / dedup / per-item-error continuation.

Files

tradein-mvp/backend/app/services/estimator.py
tradein-mvp/backend/tests/test_estimator_yandex_integration.py

Reviewer note

Reviewer: deep-code-reviewer — only merge after APPROVE. After merge → Wave 7 (Stage 9 qa-tester post-deploy smoke).

## Motivation Stage 8 of 9 for YandexRealtyScraper v1 (Wave 6). Adds Yandex Valuation tool (anonymous, no cookies — vs Cian Calculator's auth requirement) as on-demand enrichment source in `estimate_quality`. Mirrors the proven Avito IMV pattern: 24h cache via `external_valuations`, graceful degradation, idempotent history persistence. ## Changes — `estimator.py` | Function | Purpose | |---|---| | `_yandex_valuation_cache_key(address, category, type)` | SHA256 cache key | | `_get_or_fetch_yandex_valuation_cached(db, address, ...)` | Cache lookup via `external_valuations(source='yandex_valuation')`. Any error → `None` + log warning. | | `_save_yandex_history_items(db, result)` | INSERT into `house_placement_history(source='yandex_valuation')` with synthesized `ext_item_id` (sha256 of address+date+area+floor+prices, first 32 chars). Idempotent via UNIQUE (source, ext_item_id). `house_id=NULL` — matching pipeline not wired into estimator yet. | | `estimate_quality` | Calls Yandex Valuation after IMV block, before deals fetch. Adds `'yandex_valuation'` to `sources_used` on hit. | ## Constants ```python YANDEX_VALUATION_CACHE_TTL_HOURS = 24 YANDEX_VALUATION_DEFAULT_CATEGORY = "APARTMENT" YANDEX_VALUATION_DEFAULT_TYPE = "SELL" ``` ## Why "6th source", not "7th" The vault plan called this "7th source" assuming Cian Valuation was already wired into estimator as #6. Inspection of current `estimator.py` shows only Avito IMV (5th) integrated — Cian Valuation lives in the scraper layer but isn't called from `estimate_quality` yet. So Yandex slots in as the 6th source currently active in the estimator. Plan numbering is aspirational; the integration order is what matters. ## Graceful degradation - Geocode failure → estimator returns `_empty_estimate` (Yandex block skipped — needs `geo.full_address`) - Yandex scrape failure → log warning + `None`, sources_used omits `yandex_valuation` - Cache DB error → log warning + proceeds to fetch - History save DB error → per-item try/except, partial success counted ## Tests ``` tests/test_estimator_yandex_integration.py: 11 passed, 0 failed tests/ -k "estimator and not slow": 15 passed (11 new + 4 IMV regression) ruff: 0 new errors (5 pre-existing in estimator.py unrelated to this PR) ``` Coverage: cache key determinism + uniqueness, TTL constant, cache hit/miss paths, fetch error/empty handling, history item save / dedup / per-item-error continuation. ## Files - `tradein-mvp/backend/app/services/estimator.py` - `tradein-mvp/backend/tests/test_estimator_yandex_integration.py` ## Reviewer note Reviewer: **deep-code-reviewer** — only merge after APPROVE. After merge → Wave 7 (Stage 9 qa-tester post-deploy smoke).

lekss361 added 1 commit 2026-05-23 14:11:59 +00:00

feat(tradein): estimator — Yandex Valuation as on-demand 6th source 8e4dea74dd

Stage 8 of YandexRealtyScraper v1 (Wave 6).

Adds Yandex Valuation tool (anonymous, no cookies) as on-demand enrichment
source in estimate_quality. Mirrors the Avito IMV pattern with 24h cache,
graceful degradation, and idempotent history persistence.

estimator.py:
- _yandex_valuation_cache_key(address, category, type) -> sha256
- _get_or_fetch_yandex_valuation_cached(db, address, ...) -> cache via
  external_valuations(source='yandex_valuation'). Any error -> None +
  log warning + estimator continues.
- _save_yandex_history_items(db, result) -> INSERT into
  house_placement_history(source='yandex_valuation') with synthesized
  ext_item_id (sha256 of address+date+area+floor+prices, first 32 chars).
  Idempotent via UNIQUE (source, ext_item_id). house_id stays NULL -
  estimator doesn't compute target_house_id yet (matching pipeline TBD).
- estimate_quality: call after IMV block, BEFORE deals fetch. Adds
  'yandex_valuation' to sources_used if Yandex returned a result.

Tests: 11 unit tests (cache key determinism, hit/miss, fetch error,
empty result, history item save / dedup / partial error). All pass.
Ruff clean (no new errors introduced).

Constants: YANDEX_VALUATION_CACHE_TTL_HOURS=24, default category=APARTMENT,
default type=SELL.

lekss361 merged commit b26e0ff7c3 into main

2026-05-23 14:14:50 +00:00

lekss361 referenced this pull request from a commit

2026-05-23 14:14:51 +00:00

feat(tradein): estimator — Yandex Valuation as on-demand 6th source (#473)

lekss361 commented

2026-05-23 14:15:22 +00:00

Author

Owner

Merged via deep-code-reviewer — verdict APPROVE.

Verified blast radius:

external_valuations (source, cache_key) UNIQUE — data/sql/026_external_valuations.sql:34 ✅
house_placement_history (source, ext_item_id) UNIQUE — data/sql/017_house_placement_history.sql:50 ✅
Cache TTL constant 24h matches IMV ✅
source='yandex_valuation' enum value documented — data/sql/032_yandex_history.sql:20 ✅

Pattern fidelity vs Avito IMV:

Cache lookup: SELECT WHERE source + cache_key + expires_at > NOW() ✅
UPSERT on conflict (source, cache_key) DO UPDATE ✅
Graceful degradation (cache err → fetch; fetch err → None; per-item history err → continue) ✅
CAST(:payload AS jsonb) / CAST(:raw AS jsonb) — psycopg v3 compliant ✅
Named params only, no f-string SQL ✅
ON CONFLICT (source, ext_item_id) DO NOTHING on history — idempotent ✅

Cache key determinism: SHA256(address|category|type) — deterministic, defaults APARTMENT/SELL always passed by estimator caller (no None ambiguity, callsite line 537-539 doesn't pass kwargs).

Follow-ups (non-blocking, separate PRs):

🟡 Address normalization ("Ленина 1" vs "ленина 1" → different cache keys). Acceptable for v1 since geocoder produces canonical geo.full_address.
🟡 Stage 8.x: wire match_or_create_house (PR #470) to backfill house_id in house_placement_history rows. Currently house_id=NULL — confirmed schema allows NULL (017:22).
🟡 Cian Valuation wire-up (PR #465 scraper merged, not called from estimate_quality yet) — explicitly out-of-scope per PR description.
🟢 Performance: IMV (~5s) + Yandex (~3s) = ~8s sequential cache miss. Consider asyncio.gather if total estimate latency becomes UX concern.
🟢 Vault entry: code/modules/tradein/Estimator_Yandex_Integration.md for worker to create.

Verify steps post-deploy:

deploy-tradein.yml triggers on merge to main
CI pytest: 15/15 green (11 new + 4 IMV regression)
Live: POST /api/v1/trade-in/estimate для ЕКБ адреса → response sources_used содержит yandex_valuation
SQL: SELECT count(*) FROM house_placement_history WHERE source='yandex_valuation' > 0
SQL: SELECT count(*) FROM external_valuations WHERE source='yandex_valuation' AND expires_at > NOW() > 0
Re-POST same address within 24h → cache HIT (logs yandex_valuation: cache HIT key=...)

Wave 7 (Stage 9 qa-tester post-deploy smoke) unblocked.

Merged via deep-code-reviewer — verdict APPROVE. **Verified blast radius:** - `external_valuations (source, cache_key)` UNIQUE — `data/sql/026_external_valuations.sql:34` ✅ - `house_placement_history (source, ext_item_id)` UNIQUE — `data/sql/017_house_placement_history.sql:50` ✅ - Cache TTL constant 24h matches IMV ✅ - `source='yandex_valuation'` enum value documented — `data/sql/032_yandex_history.sql:20` ✅ **Pattern fidelity vs Avito IMV:** - Cache lookup: SELECT WHERE source + cache_key + expires_at > NOW() ✅ - UPSERT on conflict (source, cache_key) DO UPDATE ✅ - Graceful degradation (cache err → fetch; fetch err → None; per-item history err → continue) ✅ - `CAST(:payload AS jsonb)` / `CAST(:raw AS jsonb)` — psycopg v3 compliant ✅ - Named params only, no f-string SQL ✅ - `ON CONFLICT (source, ext_item_id) DO NOTHING` on history — idempotent ✅ **Cache key determinism:** `SHA256(address|category|type)` — deterministic, defaults `APARTMENT`/`SELL` always passed by estimator caller (no None ambiguity, callsite line 537-539 doesn't pass kwargs). **ext_item_id determinism:** `SHA256(address|publish_date|area|floor|start_price|last_price)[:32]` = 128 bits — collision-resistant. Stable across re-fetches (test `test_save_history_items_ext_id_stable_across_calls` proves it). **Follow-ups (non-blocking, separate PRs):** - 🟡 Address normalization (`"Ленина 1"` vs `"ленина 1"` → different cache keys). Acceptable for v1 since geocoder produces canonical `geo.full_address`. - 🟡 Stage 8.x: wire `match_or_create_house` (PR #470) to backfill `house_id` in `house_placement_history` rows. Currently `house_id=NULL` — confirmed schema allows NULL (017:22). - 🟡 Cian Valuation wire-up (PR #465 scraper merged, not called from `estimate_quality` yet) — explicitly out-of-scope per PR description. - 🟢 Performance: IMV (~5s) + Yandex (~3s) = ~8s sequential cache miss. Consider `asyncio.gather` if total estimate latency becomes UX concern. - 🟢 Vault entry: `code/modules/tradein/Estimator_Yandex_Integration.md` for worker to create. **Verify steps post-deploy:** 1. `deploy-tradein.yml` triggers on merge to main 2. CI pytest: 15/15 green (11 new + 4 IMV regression) 3. Live: `POST /api/v1/trade-in/estimate` для ЕКБ адреса → response `sources_used` содержит `yandex_valuation` 4. SQL: `SELECT count(*) FROM house_placement_history WHERE source='yandex_valuation'` > 0 5. SQL: `SELECT count(*) FROM external_valuations WHERE source='yandex_valuation' AND expires_at > NOW()` > 0 6. Re-POST same address within 24h → cache HIT (logs `yandex_valuation: cache HIT key=...`) Wave 7 (Stage 9 qa-tester post-deploy smoke) unblocked.

lekss361 referenced this pull request

2026-05-23 14:25:12 +00:00

feat(tradein): estimator — Cian Valuation as on-demand evaluation source (Stage 9) #476

lekss361 referenced this pull request

2026-05-23 14:30:18 +00:00

feat(tradein): estimator — Cian Valuation as on-demand evaluation source (Stage 9) #476

lekss361 referenced this pull request

2026-05-23 15:25:54 +00:00

feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints #484