9 commits

Author	SHA1	Message	Date
bot-backend	a9b87aa311	chore(tradein/scraper-kit): удалить 5 orphaned legacy-модулей (#2277 финальный шаг, частичный) ... Удалены (0 runtime importers в app/, scripts/, packages/; проверено grep'ом, включая intra-legacy импорты в оставшихся 21 legacy-модуле scrapers/): - backend/app/services/scrapers/avito_imv.py - backend/app/services/scrapers/domclick_detail.py - backend/app/services/scrapers/ekb_geoportal_client.py - backend/app/services/scrapers/yandex_detail.py - backend/app/services/scrapers/yandex_valuation.py Тесты — удалены (pure legacy-vs-kit parity, kit-only coverage уже есть в другом месте): - tests/scrapers/test_ekb_geoportal_client_kit_parity.py - tests/scrapers/test_domclick_detail_kit_parity.py Тесты — конвертированы в kit-only (импорт легаси заменён на scraper_kit.*, покрытие сохранено без потерь): - tests/scrapers/test_avito_imv_kit_parity.py (parity-тесты убраны, config-footgun regression на kit-стороне оставлен) - tests/scrapers/test_avito_imv_browser_transport.py - tests/scrapers/test_yandex_valuation_kit_migration.py (parity убрана, footgun- регрессия mandatory config/delay_provider оставлена) - tests/scrapers/test_domclick_detail.py (+ exceptions переведены на scraper_kit.domclick_exceptions, т.к. kit parse_detail_html поднимает их) - tests/test_avito_imv_parse.py - tests/test_ekb_geoportal_ingest.py (client-часть; ingest/geocoder-часть не трогали — не зависят от удалённого модуля) - tests/test_yandex_detail.py - tests/test_yandex_detail_structural.py - tests/test_yandex_valuation.py (YandexValuationScraper() → config=_KIT_CONFIG, kit конструктор требует обязательный config) - tests/test_yandex_valuation_save.py - tests/test_extval_house_id_write_path.py (только yandex_valuation-часть; cian_valuation остался нетронутым — живой легаси-модуль) - tests/test_yandex_history_area_filter.py Тесты — частично отредактированы (убраны только части про удалённые модули, живые легаси-модули/их parity не тронуты): - tests/scrapers/test_admin_domclick_ingest_kit_parity.py (base.py/ScrapedLot parity остался) - tests/scrapers/test_admin_yandex_kit_parity.py (yandex_realty parity остался) - tests/scrapers/test_admin_avito_kit_parity.py (avito.py/avito_houses.py parity остался; _parse_price parity убран) - tests/scrapers/test_avito_unix_date_tz_consistency.py (IMV-ветка переведена на kit avito.imv, легаси avito/avito_houses/avito_shared/avito_detail не тронуты) - tests/tasks/test_yandex_detail_backfill.py (subject — kit-задача, уже вызывает scraper_kit.providers.yandex.detail; save_detail_enrichment coverage-тесты внизу файла переведены на kit) - tests/test_scraper_kit_yandex_golden_parity.py (detail/valuation parity убраны; serp/newbuilding/house_type_normalizer/build_url для yandex_realty/ yandex_newbuilding не тронуты) - tests/test_scraper_proxy.py (YandexValuation proxy-тесты переведены на kit config= DI; test_avito_imv_own_session_receives_proxies удалён — дублирует test_avito_imv_kit_parity.py footgun-тесты) - tests/test_yandex_scrapers_delay_wiring.py (yandex_detail/yandex_valuation → kit delay_provider=; yandex_realty/yandex_newbuilding не тронуты) - tests/test_scraper_kit_group_c_backfill_kit_parity.py (docstring уточнение, без функциональных изменений) Полный backend-suite зелёный (3253 passed, 6 skipped) кроме известного pre-existing flake test_search_api.py::test_search_cache_hit (#2208, не регрессия этого PR). ruff check + ruff format — чисто на всех изменённых файлах.	2026-07-04 10:41:46 +03:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
bot-backend	a8d7be68ad	feat(yandex): route gate-API fetch through camoufox BrowserFetcher ... Replace the system-curl SOCKS5 subprocess transport with the shared BrowserFetcher (camoufox). camoufox's real-browser fingerprint is not tarpitted by yandex and does not burn the mobile-proxy IP; raw system-curl got the proxy IP flagged. One BrowserFetcher is opened per scraper session in __aenter__ and reused across all page fetches. _http_get now fetches via the browser, extracts the gate-API JSON from camoufox's HTML <pre> wrapper, and returns it as the body of a _CurlResponse (status 200 on success, 0 on fetch/extraction failure) so all existing callers and the tarpit retry/rotate logic keep working unchanged. URL building, _parse_gate_json, pagination, combos and field mapping are untouched. Removed the dead curl subprocess constants/method.	2026-06-17 21:16:48 +03:00
bot-backend	35bc792cf2	fix(tradein): обновить test_scraper_proxy под BrowserFetcher-транспорт fetch_newbuilding — зелёный CI-гейт (#972)	2026-06-15 19:02:13 +03:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	b9690d4d63	fix(tradein): add cian_proxy_url to test _mock_settings — unblock deploy (#923) ... PR #923 switched Cian scrapers to settings.cian_proxy_url but the shared _mock_settings helper in test_scraper_proxy.py still only exposed scraper_proxy_url, so pre-existing cian proxy-wiring tests broke with AttributeError → deploy pytest gate failed. Mirror scraper_proxy_url into cian_proxy_url in the mock.	2026-05-31 22:34:16 +03:00
bot-backend	02af421799	feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:30:14 +00:00
bot-backend	4494866574	fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:04:31 +00:00
bot-backend	01593be438	feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:10:36 +00:00