feat(tradein): rewrite Yandex SERP to gate-API JSON transport

Browse source

Replace selectolax HTML SERP parser with Yandex gate-API JSON endpoint
(GET realty.yandex.ru/gate/react-page/get/) which bypasses bandwidth-tarpit.

Key changes:
- _build_url: now builds gate-API URL (rgid=559132, newFlat=NO, page 1-based)
- Remove _parse_html / _card_to_lot / _extract_address / all DOM selectors
- Add _parse_gate_json / _entity_to_lot: extract ScrapedLot from JSON entities
- Lat/lon now directly available from location.point (gate-API quality)
- Address from location.geocoderAddress (full with house number)
- Studio: roomsTotal=STUDIO param; entities return roomsTotal=None -> rooms=0
- Keep system curl SOCKS5 subprocess + rotate-on-tarpit retry (status==0)
- Pagination driven by pager.totalPages (capped at _GATE_MAX_PAGES_CAP=50)
- _walk_price_range adapted: totalItems from pager, not DOM state extraction
- test_yandex_realty_serp.py: full rewrite with gate-API JSON fixtures (48 tests)

...

This commit is contained in:

bot-backend 2026-06-17 11:12:35 +03:00

parent edaadc7685

commit b351d62f45

2 changed files with 1175 additions and 1798 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Show stats Download patch file Download diff file Expand all files Collapse all files

1447

tradein-mvp/backend/app/services/scrapers/yandex_realty.py

View file

File diff suppressed because it is too large Load diff

1526

tradein-mvp/backend/tests/test_yandex_realty_serp.py

View file

File diff suppressed because it is too large Load diff