bot-backend
b351d62f45
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Replace selectolax HTML SERP parser with Yandex gate-API JSON endpoint (GET realty.yandex.ru/gate/react-page/get/) which bypasses bandwidth-tarpit. Key changes: - _build_url: now builds gate-API URL (rgid=559132, newFlat=NO, page 1-based) - Remove _parse_html / _card_to_lot / _extract_address / all DOM selectors - Add _parse_gate_json / _entity_to_lot: extract ScrapedLot from JSON entities - Lat/lon now directly available from location.point (gate-API quality) - Address from location.geocoderAddress (full with house number) - Studio: roomsTotal=STUDIO param; entities return roomsTotal=None -> rooms=0 - Keep system curl SOCKS5 subprocess + rotate-on-tarpit retry (status==0) - Pagination driven by pager.totalPages (capped at _GATE_MAX_PAGES_CAP=50) - _walk_price_range adapted: totalItems from pager, not DOM state extraction - test_yandex_realty_serp.py: full rewrite with gate-API JSON fixtures (48 tests) |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| avito.py | ||
| avito_detail.py | ||
| avito_exceptions.py | ||
| avito_houses.py | ||
| avito_imv.py | ||
| base.py | ||
| browser_fetcher.py | ||
| cian.py | ||
| cian_detail.py | ||
| cian_newbuilding.py | ||
| cian_state_parser.py | ||
| cian_valuation.py | ||
| domclick.py | ||
| domclick_exceptions.py | ||
| repair_state_normalizer.py | ||
| snapshot_writer.py | ||
| yandex_detail.py | ||
| yandex_helpers.py | ||
| yandex_newbuilding.py | ||
| yandex_realty.py | ||
| yandex_valuation.py | ||