feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) #1665

Merged

lekss361 merged 1 commit from feat/yandex-gate-api-json into main

2026-06-17 08:20:47 +00:00

lekss361 commented

2026-06-17 08:14:24 +00:00

Owner

Summary

Rewrites the yandex realty scraper to fetch the JSON gate-API (/gate/react-page/get/) instead of scraping the HTML SERP — bypassing yandex's bandwidth-tarpit.

Why (verified on prod 2026-06-17)

yandex anti-bot tarpits the heavy HTML SERP pages (throttles the flagged mobile-proxy IP to ~77 B/s → 30s timeout → 0 lots). But the gate-API path is NOT tarpitted: robots.txt = 6491 B/s, and the offers API returned HTTP 200, 129KB JSON in 17.8s, 20 offers through the same proxy. Neither cookies (tested — connection-level tarpit ignores them) nor the camoufox browser layer (also times out on the heavy SERP) help; the JSON API is the real bypass.

Bonus: richer data

The gate-API returns fields the HTML SERP lacked (which previously needed detail-enrichment / geocoding):

location.point → lat/lon (was NULL in SERP), location.geocoderAddress → full address with house number
area.value, livingSpace, kitchenSpace, roomsTotal, floorsOffered/floorsTotal, building.builtYear/buildingType, ceilingHeight, price.value/valuePerPart
pager.totalPages → proper pagination (was capped at MAX_PAGES=3)

Changes

yandex_realty.py rewrite (1222→835 lines): _build_url → gate-API URL (rgid=559132, type=SELL, category=APARTMENT, newFlat=NO, roomsTotal, priceMin/Max, page, _pageType=search, _providers=react-search-results-data; no crc needed); new _parse_gate_json over response.search.offers.entities[]; pagination via pager.totalPages (capped). Keeps the system-curl SOCKS5 transport + rotate-on-tarpit retry (#1664) as safety.
Parser mapping: offerId→source_id, url→source_url (prepend https:), price.value→price_rub, area/living/kitchen, roomsTotal (None=studio→0), floors, building year/type, point→lat/lon, geocoderAddress→address, mainImages→photo_urls.
Tests: 48 gate-API JSON tests (fixtures from a real captured response).

Test plan

Full gate: 1841 passed, 0 failed; ruff clean
Parser validated against real captured 20-offer sample
Post-deploy prod smoke (MANDATORY): trigger yandex_city_sweep → confirm lots_fetched ≫ 0 with lat/lon + address populated, no tarpit timeout

Refs #759, #1658

## Summary Rewrites the yandex realty scraper to fetch the **JSON gate-API** (`/gate/react-page/get/`) instead of scraping the HTML SERP — bypassing yandex's bandwidth-tarpit. ### Why (verified on prod 2026-06-17) yandex anti-bot **tarpits the heavy HTML SERP pages** (throttles the flagged mobile-proxy IP to ~77 B/s → 30s timeout → 0 lots). But the **gate-API path is NOT tarpitted**: `robots.txt` = 6491 B/s, and the offers API returned **HTTP 200, 129KB JSON in 17.8s, 20 offers** through the same proxy. Neither cookies (tested — connection-level tarpit ignores them) nor the camoufox browser layer (also times out on the heavy SERP) help; the **JSON API is the real bypass**. ### Bonus: richer data The gate-API returns fields the HTML SERP lacked (which previously needed detail-enrichment / geocoding): - `location.point` → **lat/lon** (was NULL in SERP), `location.geocoderAddress` → **full address with house number** - `area.value`, `livingSpace`, `kitchenSpace`, `roomsTotal`, `floorsOffered`/`floorsTotal`, `building.builtYear`/`buildingType`, `ceilingHeight`, `price.value`/`valuePerPart` - `pager.totalPages` → proper pagination (was capped at MAX_PAGES=3) ### Changes - `yandex_realty.py` rewrite (1222→835 lines): `_build_url` → gate-API URL (`rgid=559132`, `type=SELL`, `category=APARTMENT`, `newFlat=NO`, `roomsTotal`, `priceMin/Max`, `page`, `_pageType=search`, `_providers=react-search-results-data`; no `crc` needed); new `_parse_gate_json` over `response.search.offers.entities[]`; pagination via `pager.totalPages` (capped). Keeps the system-curl SOCKS5 transport + rotate-on-tarpit retry (#1664) as safety. - Parser mapping: offerId→source_id, url→source_url (prepend https:), price.value→price_rub, area/living/kitchen, roomsTotal (None=studio→0), floors, building year/type, point→lat/lon, geocoderAddress→address, mainImages→photo_urls. - Tests: 48 gate-API JSON tests (fixtures from a real captured response). ## Test plan - [x] Full gate: **1841 passed, 0 failed**; ruff clean - [x] Parser validated against real captured 20-offer sample - [ ] **Post-deploy prod smoke (MANDATORY)**: trigger yandex_city_sweep → confirm lots_fetched ≫ 0 with lat/lon + address populated, no tarpit timeout Refs #759, #1658

lekss361 added 1 commit 2026-06-17 08:14:25 +00:00

feat(tradein): rewrite Yandex SERP to gate-API JSON transport

CI / changes (push) Successful in 7s

Details

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 5s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

b351d62f45

Replace selectolax HTML SERP parser with Yandex gate-API JSON endpoint
(GET realty.yandex.ru/gate/react-page/get/) which bypasses bandwidth-tarpit.

Key changes:
- _build_url: now builds gate-API URL (rgid=559132, newFlat=NO, page 1-based)
- Remove _parse_html / _card_to_lot / _extract_address / all DOM selectors
- Add _parse_gate_json / _entity_to_lot: extract ScrapedLot from JSON entities
- Lat/lon now directly available from location.point (gate-API quality)
- Address from location.geocoderAddress (full with house number)
- Studio: roomsTotal=STUDIO param; entities return roomsTotal=None -> rooms=0
- Keep system curl SOCKS5 subprocess + rotate-on-tarpit retry (status==0)
- Pagination driven by pager.totalPages (capped at _GATE_MAX_PAGES_CAP=50)
- _walk_price_range adapted: totalItems from pager, not DOM state extraction
- test_yandex_realty_serp.py: full rewrite with gate-API JSON fixtures (48 tests)