feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro) #1725

Merged

lekss361 merged 1 commit from feat/yandex-rich-fields into main

2026-06-17 20:03:26 +00:00

lekss361 commented

2026-06-17 20:00:06 +00:00

Owner

Summary

Yandex gate-API entities expose ~90 fields; we mapped ~11. This adds the high-value ones — most notably Yandex's own per-offer valuation (predictedPrice). Verified field paths on live gate-API this session.

Mapping (yandex_realty.py `_entity_to_lot`)

creationDate → listing_date + days_on_market (clamped ≥0) + publish_date
description → description
author.category OWNER→is_homeowner, AGENCY/AGENT→is_pro_seller+agency_name(org/agentName)
predictedPrice.predictedPrice.{value,min,max} → new predicted_price_rub/_min/_max
price.trend → new price_trend; price.previous → new price_previous_rub
location.metroList[] → metro_stations jsonb; offerId → yandex_offer_id

base.py (shared — additive only, verified gaps)

save_listings was silently dropping fields even for existing columns: description & agency_name were in no INSERT/UPDATE at all; days_on_market missing from UPDATE; publish_date/yandex_offer_id not wired. All added via COALESCE(EXCLUDED.x, listings.x) so non-yandex scrapers leaving them NULL never wipe stored values. New columns added to ScrapedLot + INSERT/VALUES/UPDATE.

Migration

data/sql/121_yandex_rich_fields.sql — idempotent ADD COLUMN IF NOT EXISTS predicted_price_rub/_min/_max, price_trend, price_previous_rub. Auto-applies on deploy.

Test plan

pytest yandex+base: 67 passed; broad save_listings regression 39 passed; +13 (yandex_valuation/matching savepoint); ruff clean
Post-deploy: SELECT count(*) FILTER(WHERE predicted_price_rub IS NOT NULL), count(*) FILTER(WHERE description IS NOT NULL) FROM listings WHERE source='yandex' climb after a sweep

Refs #1658

## Summary Yandex gate-API entities expose ~90 fields; we mapped ~11. This adds the high-value ones — most notably **Yandex's own per-offer valuation** (`predictedPrice`). Verified field paths on live gate-API this session. ## Mapping (yandex_realty.py `_entity_to_lot`) - `creationDate` → `listing_date` + `days_on_market` (clamped ≥0) + `publish_date` - `description` → `description` - `author.category` OWNER→`is_homeowner`, AGENCY/AGENT→`is_pro_seller`+`agency_name`(org/agentName) - `predictedPrice.predictedPrice.{value,min,max}` → **new** `predicted_price_rub/_min/_max` - `price.trend` → **new** `price_trend`; `price.previous` → **new** `price_previous_rub` - `location.metroList[]` → `metro_stations` jsonb; `offerId` → `yandex_offer_id` ## base.py (shared — additive only, verified gaps) `save_listings` was silently dropping fields even for existing columns: **`description` & `agency_name` were in no INSERT/UPDATE at all; `days_on_market` missing from UPDATE; `publish_date`/`yandex_offer_id` not wired.** All added via `COALESCE(EXCLUDED.x, listings.x)` so non-yandex scrapers leaving them NULL never wipe stored values. New columns added to ScrapedLot + INSERT/VALUES/UPDATE. ## Migration `data/sql/121_yandex_rich_fields.sql` — idempotent `ADD COLUMN IF NOT EXISTS` predicted_price_rub/_min/_max, price_trend, price_previous_rub. Auto-applies on deploy. ## Test plan - [x] `pytest` yandex+base: 67 passed; broad save_listings regression 39 passed; +13 (yandex_valuation/matching savepoint); ruff clean - [ ] Post-deploy: `SELECT count(*) FILTER(WHERE predicted_price_rub IS NOT NULL), count(*) FILTER(WHERE description IS NOT NULL) FROM listings WHERE source='yandex'` climb after a sweep Refs #1658

lekss361 added 1 commit 2026-06-17 20:00:07 +00:00