17 commits

Author	SHA1	Message	Date
bot-backend	33e3503f97	fix(tradein/data): yandex url-дубли — дедуп-миграция + инвариант + корень (#2235)	2026-07-03 09:39:29 +03:00
lekss361	440bb2ca51	feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007) ... house_type normalized via shared house_type_normalizer (canon = _IMV_HOUSE_TYPE_MAP) — fixes estimator soft-penalty false-mismatch on ~70% yandex analogs; backfill migration 140 (SCREAMING->canon + 'other'->NULL). kitchen/ceiling promoted to columns (dashboard/matching, not valuation — see #2012); ceiling capped 2.0-6.0m to avoid numeric(3,2) overflow. Refs #2007	2026-06-27 17:45:11 +00:00
bot-backend	37f12f6303	fix(tradein/yandex): инкрементальный per-combo save + bounded gate-timeout (анти-зомби city_sweep)	2026-06-20 20:57:41 +03:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
bot-backend	41f2402b25	fix(yandex): read predicted price from entity.predictions.predictedPrice (#1725) ... Gate-API nests Yandex's per-offer valuation under predictions.predictedPrice, not predictedPrice.predictedPrice (that key does not exist). _predicted_prices read the wrong outer key, so predicted_price_rub/_min/_max were None for every offer (prod: pp=0 across all 6189 yandex rows). Verified live against the gate-API 2026-06-18. The rich-field test fixture used the same wrong wrapper, so the test passed vacuously. Updated _ENTITY_RICH_OWNER to the real predictions.predictedPrice shape — with the old code the assertions now fail, confirming non-vacuity.	2026-06-18 08:03:05 +03:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00
bot-backend	bc4c6852a1	feat(yandex): ingest novostroyki via gate-API newFlat=YES	2026-06-17 22:22:17 +03:00
bot-backend	a8d7be68ad	feat(yandex): route gate-API fetch through camoufox BrowserFetcher ... Replace the system-curl SOCKS5 subprocess transport with the shared BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl got the proxy IP flagged. One BrowserFetcher is opened per scraper session in __aenter__ and reused across all page fetches. _http_get now fetches via the browser, extracts the gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so all existing callers and the tarpit retry/rotate logic keep working unchanged. URL building, _parse_gate_json, pagination, combos and field mapping are untouched. Removed the dead curl subprocess constants/method.	2026-06-17 21:16:48 +03:00
lekss361	f8779c0812	feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)	2026-06-17 08:20:46 +00:00
bot-backend	b19c4a0b8c	fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience) ... Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected, call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.) return [] immediately without rotation. Mirrors the existing exhaustive-path pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success, captcha→rotate→success, exhausted retries→empty, 404→no-rotate.	2026-06-17 10:12:24 +03:00
lekss361	a23e650fae	fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 (#940) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 22:39:06 +00:00
lekss361	d93e96fb0c	feat(tradein): checkpoint/resume for yandex full-load — survive restart (#930) (#937) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:34:23 +00:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	571be4233b	feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) (#847) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:11:30 +00:00
bot-backend	30ae4841a4	fix(scrapers): yandex area из embedded-state JSON (Refs #775) (#783) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:38:12 +00:00
Light1YT	aa70e9e155	fix(tradein): yandex_realty captures house number + parse_rub strips NBSP (#593)	2026-05-27 07:23:36 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00