feat(yandex): ingest novostroyki via gate-API newFlat=YES #1715

Merged

lekss361 merged 1 commit from feat/yandex-novostroyki-newflat into main

2026-06-17 19:34:33 +00:00

lekss361 commented

2026-06-17 19:25:58 +00:00

Owner

Summary

Closes the yandex novostroyki segment gap. The gate-API sweep previously fetched only newFlat=NO (vtorichka) and hardcoded listing_segment="vtorichka". Now runs BOTH passes (newFlat=NO + newFlat=YES) and tags segment per pass.

Verified on prod gate-API (home IP, this session)

...&newFlat=YES&roomsTotal=2&page=1... → pager.totalItems=11784, totalPages=25; every entity newFlat:true, flatType:"NEW_FLAT", dealStatus∈{PRIMARY_SALE,REASSIGNMENT}, building.siteName=ЖК. Same JSON shape as newFlat=NO → _parse_gate_json unchanged. We were dropping ~all novostroyki (prod: yandex novostroyki=0).

Changes

yandex_realty.py: _build_url(new_flat="NO") param feeds newFlat; threaded through _fetch_page_json/fetch_around/_parse_gate_json/_entity_to_lot; listing_segment=_segment_for(new_flat) (novostroyki for YES). fetch_around_multi_room(segments=["NO"]) loops outermost over segments sharing ONE dedup seen dict.
scrape_pipeline.py::run_yandex_city_sweep: segments param default ["NO","YES"]; watchdog scaled ×len(segments) (was 30 combos×3p≈2190s/#1659 → now ×2 segments; defaults 30×2×2≈2820s). No separate total-sweep deadline exists; per-anchor asyncio.wait_for is the only cap.
scheduler.py: pass-through segments=params.get("segments") for operator override via default_params.

Test plan

pytest tests/test_yandex_realty_serp.py → 58 passed; test_yandex_city_sweep.py+test_anchor_watchdog.py → 28 passed; ruff clean
_build_url newFlat=YES/NO; segment tagging per pass; both-pass run with shared dedup
Post-deploy smoke: yandex sweep counters show novostroyki lots inserted; SELECT source,listing_segment,count(*) FROM listings WHERE source='yandex' GROUP BY 2

Refs #1658

## Summary Closes the yandex novostroyki segment gap. The gate-API sweep previously fetched only `newFlat=NO` (vtorichka) and hardcoded `listing_segment="vtorichka"`. Now runs BOTH passes (newFlat=NO + newFlat=YES) and tags segment per pass. ## Verified on prod gate-API (home IP, this session) `...&newFlat=YES&roomsTotal=2&page=1...` → `pager.totalItems=11784, totalPages=25`; every entity `newFlat:true, flatType:"NEW_FLAT", dealStatus∈{PRIMARY_SALE,REASSIGNMENT}, building.siteName`=ЖК. Same JSON shape as newFlat=NO → `_parse_gate_json` unchanged. We were dropping ~all novostroyki (prod: yandex novostroyki=0). ## Changes - `yandex_realty.py`: `_build_url(new_flat="NO")` param feeds `newFlat`; threaded through `_fetch_page_json`/`fetch_around`/`_parse_gate_json`/`_entity_to_lot`; `listing_segment=_segment_for(new_flat)` (novostroyki for YES). `fetch_around_multi_room(segments=["NO"])` loops outermost over segments sharing ONE dedup `seen` dict. - `scrape_pipeline.py::run_yandex_city_sweep`: `segments` param default `["NO","YES"]`; **watchdog scaled ×len(segments)** (was 30 combos×3p≈2190s/#1659 → now ×2 segments; defaults 30×2×2≈2820s). No separate total-sweep deadline exists; per-anchor `asyncio.wait_for` is the only cap. - `scheduler.py`: pass-through `segments=params.get("segments")` for operator override via default_params. ## Test plan - [x] `pytest tests/test_yandex_realty_serp.py` → 58 passed; `test_yandex_city_sweep.py`+`test_anchor_watchdog.py` → 28 passed; ruff clean - [x] `_build_url` newFlat=YES/NO; segment tagging per pass; both-pass run with shared dedup - [ ] Post-deploy smoke: yandex sweep counters show novostroyki lots inserted; `SELECT source,listing_segment,count(*) FROM listings WHERE source='yandex' GROUP BY 2` Refs #1658

lekss361 added 1 commit 2026-06-17 19:25:59 +00:00