9 commits

Author	SHA1	Message	Date
bot-backend	a07fe55db9	fix(tradein): route yandex/valuation/imv/newbuilding scrapers through scraper_proxy_url ... Yandex SERP was being banned (datacenter IP) and on-demand scrapers (cian_valuation, cian_newbuilding, avito_imv, avito_detail) silently returned None due to same cause. Mobile proxy was already wired for cian.py/cian_detail.py/avito.py but missing elsewhere. - yandex_realty: proxies= added to _CurlCffiSession in __aenter__ - yandex_valuation: proxies= added to _CurlCffiSession in __aenter__ - cian_valuation: proxies= added to AsyncSession (async with ... fetch block) - cian_newbuilding: proxies= added to both own-session paths (fetch_newbuilding + resolve_cian_zhk_url); shared-session callers unaffected - avito_imv: proxies= added to own-session path only (shared CffiAsyncSession from AvitoScraper already carries proxy) - avito_detail: proxies= added to own-session path only (scrape_pipeline passes shared session with proxy; admin endpoint creates own-session -> now proxied) All use pattern: _proxy = settings.scraper_proxy_url; proxies={"http":_proxy,"https":_proxy} if _proxy else None -- matching cian.py (#806 reference). proxy=None = direct connection (dev). Tests: 9 new proxy-wiring unit tests added to test_scraper_proxy.py (20 total, all pass).	2026-05-31 09:57:17 +03:00
bot-backend	571be4233b	feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) (#847) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:11:30 +00:00
bot-backend	30ae4841a4	fix(scrapers): yandex area из embedded-state JSON (Refs #775) (#783) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:38:12 +00:00
Light1YT	312df8536a	feat(tradein): infer repair_state from listing description text (#622) (#632)	2026-05-28 15:37:34 +00:00
Light1YT	aa70e9e155	fix(tradein): yandex_realty captures house number + parse_rub strips NBSP (#593)	2026-05-27 07:23:36 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	ead2ac16c5	feat(tradein): yandex_realty.py — DOM SERP refactor (vtorichka, 23 cards/page) ... Replace dead JSON-state extraction (Yandex moved to chunked/ephemeral state post-hydration) with selectolax DOM parsing using helpers from PR #456. Changes: - New URL: /{city}/kupit/kvartira/vtorichniy-rynok/?page=N (path-based, no geo) - Selector: [data-test="OffersSerpItem"] x ~23 per page - Per-card extraction: offer_id (URL regex), title (rooms/area/floor regex), price/ppm2 (regex + parse_rub), bargain flag, publish_date (parse_ru_date), street address, photos (avatars.mds.yandex domain filter + size upgrade), newbuilding linkage (house_source='yandex_realty_nb' + house_ext_id) - ScrapedLot mapping: lat/lon=None (no coords on SERP — geocode-missing fills later), listing_segment="vtorichka" - Pagination: fetch_around_multi_room -> up to MAX_PAGES=3 pages, dedup by source_id. **kwargs swallow keeps old rooms/sort/pages callers compat. - Preserved: BaseScraper interface, name="yandex" (matches existing rows + avoids breaking dedup_hash; Wave 5 conflict resolution can alias if needed), request_delay_sec=5.0, sleep_between_requests, tenacity retry. Tests: 10 unit tests against hand-crafted fixture HTML (single card / studio / no-price / empty page / multi-card / URL builder / city override) — all pass. Ruff clean.	2026-05-23 16:21:47 +03:00
lekss361	02267d0306	feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) ... - FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex) - Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in - WeasyPrint PDF (Брусника-style 4-page report) - Address autocomplete с typo-tolerance + 6 EKB presets - Изолированный docker stack gendesign-tradein (отдельная postgres БД) - Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/* - Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login) - Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml) - Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest Первый запуск на сервере (вручную, один раз): - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email) - docker network create gendesign_shared (если нет) - docker compose -p gendesign-tradein up -d - docker compose -p gendesign exec caddy caddy reload	2026-05-21 00:25:39 +03:00