feat(avito): capture newbuilding ЖК id/url from SERP card #1729

Merged

lekss361 merged 1 commit from feat/avito-newbuilding-capture into main

2026-06-17 20:29:40 +00:00

lekss361 commented

2026-06-17 20:24:20 +00:00

Owner

Summary

Captures avito ЖК (newbuilding) linkage. Novostroyka SERP cards (detected via item-development-name, merged in #1716) carry a zhk-<slug>.avito.ru anchor — now parsed into listings.newbuilding_id/newbuilding_url (columns existed, were unwired in base.py save).

Changes

avito.py _dom_card_to_lot: for novostroyka cards, find a[href*="zhk-"][href*=".avito.ru"] → newbuilding_url (scheme+host+path, tracking stripped via urlparse/urlunparse), newbuilding_id = slug (host minus zhk- prefix / .avito.ru suffix). None for vtorichka or novostroyka without anchor.
base.py: ScrapedLot += newbuilding_id/newbuilding_url; wired into save_listings INSERT/VALUES/UPDATE (COALESCE(EXCLUDED.x, listings.x), additive — edits are in the ScrapedLot+INSERT/UPDATE region, NOT _link_listing_to_house, so no overlap with #1727).

Note: cian stores newbuilding_name only in raw_payload (not on the lot), so this base wiring affects only avito for now. yandex_detail.py has its own UPDATE (unaffected).

Test plan

pytest -k "avito or save_listings or base" → 256 passed; ruff clean
card with zhk anchor → cleaned url + slug id; without → None; vtorichka → None
Post-deploy: SELECT count(*) FILTER(WHERE newbuilding_url IS NOT NULL) FROM listings WHERE source='avito' > 0 after a sweep that hits novostroyki

Refs #1188

## Summary Captures avito ЖК (newbuilding) linkage. Novostroyka SERP cards (detected via `item-development-name`, merged in #1716) carry a `zhk-<slug>.avito.ru` anchor — now parsed into `listings.newbuilding_id`/`newbuilding_url` (columns existed, were unwired in base.py save). ## Changes - `avito.py _dom_card_to_lot`: for novostroyka cards, find `a[href*="zhk-"][href*=".avito.ru"]` → `newbuilding_url` (scheme+host+path, tracking stripped via urlparse/urlunparse), `newbuilding_id` = slug (host minus `zhk-` prefix / `.avito.ru` suffix). None for vtorichka or novostroyka without anchor. - `base.py`: `ScrapedLot` += `newbuilding_id`/`newbuilding_url`; wired into save_listings INSERT/VALUES/UPDATE (`COALESCE(EXCLUDED.x, listings.x)`, additive — edits are in the ScrapedLot+INSERT/UPDATE region, NOT `_link_listing_to_house`, so no overlap with #1727). Note: cian stores newbuilding_name only in raw_payload (not on the lot), so this base wiring affects only avito for now. yandex_detail.py has its own UPDATE (unaffected). ## Test plan - [x] `pytest -k "avito or save_listings or base"` → 256 passed; ruff clean - [x] card with zhk anchor → cleaned url + slug id; without → None; vtorichka → None - [ ] Post-deploy: `SELECT count(*) FILTER(WHERE newbuilding_url IS NOT NULL) FROM listings WHERE source='avito'` > 0 after a sweep that hits novostroyki Refs #1188

lekss361 added 1 commit 2026-06-17 20:24:21 +00:00