22 commits

Author	SHA1	Message	Date
bot-backend	8c8289cdad	feat(tradein/scraper-kit): port domclick_detail + ekb_geoportal_client to kit (#2307) ... Group D of the scraper_kit migration epic (#2277). Both legacy modules had no kit-side equivalent yet (per audit Scraper_Kit_Legacy_Dependency_Audit_0703), unlike the other 24/26 files which were already strangler-duplicated. - scraper_kit/providers/domclick/detail.py — Layer B detail-card enrichment (parse_detail_html/fetch_detail/save_detail_enrichment), no ScraperConfig injection needed (fetch_detail takes an already-open browser_fetcher). - scraper_kit/providers/ekb_geoportal/{__init__,client}.py — new subpackage, first non-listing-site provider in scraper_kit. Legacy module had zero app.* coupling (only httpx), so the port is near-verbatim. Parity proven via tests/support/parity.py (assert_parity) against the exact legacy functions on realistic fixtures (live-card SSR JSON, GeoJSON FeatureCollection). Callers of both legacy modules are unchanged (out of scope — ekb_geoportal_ingest switch is #2310; the only other domclick_detail caller is scripts/ingest_domclick_jsonl.py, also untouched).	2026-07-04 00:05:07 +03:00
bot-backend	6f170504ba	fix(tradein/pii): удалить телефоны продавцов (NULL) + PII клиента (DROP) (#2217, #1969) ... - listings.phones: миграция 166 обнуляет данные; cian.py + scraper-kit serp.py (golden-parity зеркало) перестают читать offer.get("phones") на входе — write-path выключен у источника, колонка/поле в модели остаются. Мёртвая запись "phones" в LISTING_FIELD_PRIORITY убрана. - trade_in_estimates.client_name/client_phone: миграция 167 дропает колонки; синхронно убраны из TradeInEstimateInput, обоих INSERT в estimator.py (основной путь + _empty_estimate), frontend TradeInEstimateInput и CRM-блока EstimateForm (поля "Имя клиента"/"Телефон"). - sentry_scrub.py и management_companies.phones не тронуты — вне scope.	2026-07-03 23:00:42 +03:00
bot-backend	8a10d943d7	fix(tradein/scraper-kit): scale cian anchor watchdog for proxy-pool browser fetches (#2160) ... Fixed ANCHOR_TIMEOUT_SEC=240 guillotined every cian anchor when running via USE_PROXY_POOL_BROWSER=true: each SERP fetch goes through camoufox with relaunch on proxy swap (13-45s/page), and an anchor = 4 room-buckets x pages_per_anchor SERP fetches + detail_top_n details + houses-enrich >> 240s. save_listings never ran, lots_fetched=0, and after 3 consecutive skips the run was mark_banned. Introduces _cian_anchor_timeout_s() mirroring the avito/yandex/domclick scaling pattern: max(ANCHOR_TIMEOUT_SEC, buckets*pages*(delay+per_serp) + details*per_detail + houses_budget). Defaults -> 1580s worst-case watchdog (hang guard, not expected duration). Applied to both the scraper-kit copy (strangler target) and the active app.services.scrape_pipeline copy that run 581 actually hit.	2026-07-03 10:05:06 +03:00
lekss361	bbe596747d	Merge pull request 'fix(tradein/data): yandex url-«дубли» — каноникализация URL (скрейпер + миграция 164) + manifest 162 (#2235, #2216)' (#2271) from fix/tradein-yandex-url-dedup into main	2026-07-03 06:53:27 +00:00
bot-backend	33e3503f97	fix(tradein/data): yandex url-дубли — дедуп-миграция + инвариант + корень (#2235)	2026-07-03 09:39:29 +03:00
lekss361	e1c5f2184c	fix(tradein/scraper-kit): wire browser proxy-pool into yandex/cian SERP + persist activation flags (#2160) (#2268)	2026-07-03 06:36:56 +00:00
lekss361	47f45fc3d2	fix(tradein/scrapers): bump scraped_at при ре-подтверждении живым + бэкфилл (#2206) (#2224)	2026-07-02 20:30:01 +00:00
lekss361	ca473a0012	feat(proxy-pool): P4 — browser dynamic pool-proxy (acquire→/fetch→relaunch) + orchestration wiring, ship-dark (#2164)	2026-07-02 19:20:29 +00:00
lekss361	0937da4260	feat(proxy-pool): P3 — kit curl-paths acquire/release from pool behind USE_PROXY_POOL_CURL, ship-dark (#2163)	2026-07-02 18:56:36 +00:00
bot-backend	e84b6e553a	fix(tradein/scraper-kit): mirror numberless None-guards in avito houses provider copy	2026-07-02 21:44:46 +03:00
bot-backend	0eb033d77d	fix(tradein/matching): P1 extended — never create a house from a numberless address (no cadastral)	2026-07-02 21:33:52 +03:00
lekss361	509cecfa6a	feat(scraper-kit): copy scheduler with registry-dispatch (25+ trigger_* → dict), strangler (#2136)	2026-07-02 17:44:03 +00:00
lekss361	abdb413a99	feat(scraper-kit): copy remaining sweep orchestrators (yandex/cian/domclick/full-load), strangler (#2135 F2)	2026-07-02 17:29:05 +00:00
lekss361	8a177df0e2	feat(scraper-kit): copy sweep pipeline orchestration (avito) with protocol injection, strangler (#2135)	2026-07-02 17:08:40 +00:00
lekss361	62fa1f732a	refactor(scraper-kit): unify price bisection engine across providers + dedup (#2134)	2026-07-02 16:41:04 +00:00
lekss361	3be8062062	feat(scraper-kit): copy yandex providers with protocol injection, strangler (#2133 yandex)	2026-07-02 16:13:32 +00:00
lekss361	19c0be5d5a	feat(scraper-kit): copy cian providers with protocol injection, strangler (#2133 cian)	2026-07-02 16:11:31 +00:00
lekss361	b7abaeb569	feat(scraper-kit): copy domclick provider with protocol injection, strangler (#2133 domclick)	2026-07-02 16:08:09 +00:00
lekss361	df5e9003df	feat(scraper-kit): copy avito providers with protocol injection, strangler (#2133 avito)	2026-07-02 15:58:00 +00:00
lekss361	5c17d21f6c	feat(scraper-kit): copy core layer (base/save_listings/transport) with protocol injection, strangler (#2132)	2026-07-02 15:43:15 +00:00
lekss361	6391587572	feat(scraper-kit): protocol boundaries (matcher/config/session/jobs) + backend adapters (#2131)	2026-07-02 15:25:38 +00:00
lekss361	ddf6cb7ce3	chore(scraper-kit): scaffold internal package + uv workspace, build-safe (#2128)	2026-07-02 14:06:07 +00:00