f8779c0812
feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) ( #1665 )
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 51s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 42s
Deploy Trade-In / changes (push) Successful in 6s
2026-06-17 08:20:46 +00:00
b19c4a0b8c
fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience)
...
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.
2026-06-17 10:12:24 +03:00
a23e650fae
fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 ( #940 )
...
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 22:39:06 +00:00
d93e96fb0c
feat(tradein): checkpoint/resume for yandex full-load — survive restart ( #930 ) ( #937 )
...
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been cancelled
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 21:34:23 +00:00
2bcccdf619
feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency ( #931 )
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 21:10:26 +00:00
571be4233b
feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) ( #847 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 26s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 06:11:30 +00:00
30ae4841a4
fix(scrapers): yandex area из embedded-state JSON (Refs #775 ) ( #783 )
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 16:38:12 +00:00
aa70e9e155
fix(tradein): yandex_realty captures house number + parse_rub strips NBSP ( #593 )
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 55s
Deploy Trade-In / deploy (push) Successful in 42s
2026-05-27 07:23:36 +00:00
lekss361
ead2ac16c5
feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page)
...
Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state
post-hydration) with selectolax DOM parsing using helpers from PR #456 .
Changes:
- New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo)
- Selector: [data-test="OffersSerpItem"] x ~23 per page
- Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex),
price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date),
street address, photos (avatars.mds.yandex domain filter + size upgrade),
newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id)
- ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing
fills later), listing_segment="vtorichka"
- Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by
source_id. **kwargs swallow keeps old rooms/sort/pages callers compat.
- Preserved: BaseScraper interface, name="yandex" (matches existing rows +
avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed),
request_delay_sec=5.0, sleep_between_requests, tenacity retry.
Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio /
no-price / empty page / multi-card / URL builder / city override) — all pass.
Ruff clean.
2026-05-23 16:21:47 +03:00
