225 commits

Author	SHA1	Message	Date
bot-backend	587564153f	fix(tradein): cian _rotate_ip uses cian_proxy_rotate_url; page-scope idiom ... Review #926 fixups: - _rotate_ip: cian_proxy_rotate_url or avito_proxy_rotate_url (was avito-only → rotated wrong proxy IP on Cian ban, ignored CIAN_PROXY_ROTATE_URL). - pagination log: dir() → locals() for local-var existence check.	2026-05-31 22:51:52 +03:00
bot-backend	46269b527a	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning ... Обходит Cian SERP-cap (~54 стр): адаптивно бьёт цену на бакеты внутри комнатности (totalOffers-driven split, minprice/maxprice), пагинирует каждый полностью, дедуп по source_id. Region-wide один проход (anchor'ы для Cian избыточны) + IP-rotation (cian_proxy_rotate_url). Endpoint POST /admin/scrape/cian-full-load.	2026-05-31 22:43:05 +03:00
lekss361	4999c877ae	feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) (#919) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 18:48:22 +00:00
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
bot-backend	92f44ac968	feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) (#916) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:10:38 +00:00
bot-backend	639d3b0cea	feat(tradein): systemd timer scheduler — trigger script + units + SCHEDULER_ENABLE flag (#581) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:53:38 +00:00
bot-backend	441e48141b	feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:49:35 +00:00
bot-backend	f09d9737ce	fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) ... Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning a 274KB generic avito shell with 0 listings — NOT a block/captcha (no captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms wait the page.content() snapshot still had only the generic shell. At 5-6s it's the full SERP: 3.2MB, priceDetailed=50. Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works — this was the last blocker. NOT headless detection (headless=True returns real data); the abandoned xvfb/headless=virtual change is unnecessary. Refs #905, #883	2026-05-31 19:17:01 +03:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
lekss361	98b0632dae	Merge pull request 'feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)' (#904) from feat/796-domclick-scraper into main ... Reviewed-on: #904	2026-05-31 14:43:39 +00:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	4c0338ce1d	fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup)	2026-05-31 17:21:30 +03:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	53af9a8c55	feat(tradein): add EKB/MSK REF_AREA codes to sber_price_index pull (#794) (#900) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:56:17 +00:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	fa2bd20ac6	refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:43 +00:00
lekss361	2890199d07	Merge pull request 'feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884)' (#892) from feat/884-browserfetcher-camoufox into main ... Reviewed-on: #892	2026-05-31 12:47:49 +00:00
lekss361	8df536ca5f	Merge pull request 'refactor(tradein): unify kadastr_num → cadastral_number (#732)' (#891) from feat/732-cadastral-unify into main ... Reviewed-on: #891	2026-05-31 12:47:10 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	42de30cd5c	refactor(tradein): unify kadastr_num -> cadastral_number (#732) ... p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql (idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) + v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number, DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass. WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo. Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply (reconstructed from migration history 050/046, not live capture). Refs #732	2026-05-31 15:41:52 +03:00
bot-backend	f3ba8acd76	feat(tradein): per-anchor watchdog timeout in city sweeps (#880) (#881) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 10:40:41 +00:00
lekss361	d1034bbc70	Merge pull request 'feat(tradein): same-building anchor safe-guard (#755)' (#877) from feat/755-anchor-safeguard into main ... Reviewed-on: #877	2026-05-31 10:36:22 +00:00
bot-backend	ec82742456	feat(tradein): same-building anchor safe-guard (#755) ... CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building anchor replaced headline median/range/confidence with an over-confident premium on as few as 2 same-building asking comps. Human-chosen (Anton) 3-part safe-guard, all implemented: 1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps. 2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium. 3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation (k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire (fallback to radius-median). Guards against price typos inflating anchor. Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen). quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor tests pass, ruff clean. MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 / Заводская 44а). Backtest needs live DB — pre-merge human/qa step. Refs #755, #694	2026-05-31 12:19:50 +03:00
bot-backend	d9b8047a7d	fix(tradein): yandex address-enrich — city-agnostic title regex (спутники ЕКБ) (#875) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 09:08:29 +00:00
lekss361	997eda489c	fix(tradein): single-run guard на avito-city-sweep admin-endpoint (#874) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:55:33 +00:00
lekss361	70838c7dfe	fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:27:47 +00:00
bot-backend	65b9247369	fix(matching): Tier 2b alias sync + collapse double SAVEPOINT (#849) (#870) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 08:05:48 +00:00
bot-backend	10b4b02446	fix(tradein): cian sweep houses-phase — JOIN house_sources + graceful (не fatal) (#868) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:59:26 +00:00
bot-backend	60353beaaf	feat(tradein): yandex city-sweep — один прогон SERP+address-enrich (#867) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:43:13 +00:00
bot-backend	2dd2d8e57e	feat(tradein): schedule yandex address backfill task (#855) (#866) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:31:10 +00:00
bot-backend	f2a123e2aa	feat(tradein): cian city-sweep — один прогон SERP+detail(+price-history)+houses (#864) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:24:34 +00:00
bot-backend	3f31d1f409	fix(estimator): sanity-clamp quarter-index factor (#859) (#863) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:20:59 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	d78ae9f649	feat(tradein): avito city-sweep — IMV-фаза (один прогон = SERP+detail+houses+IMV) (#861) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:09:42 +00:00
bot-backend	4494866574	fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:04:31 +00:00
bot-backend	b16a45ef87	feat(tradein): per-cadastral-quarter price index in estimator (#764) (#858) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:53:21 +00:00
bot-backend	5373f5db63	feat(tradein): admin backfill endpoints — cian price-history (T8a) + yandex address (T10) + house IMV (T7) (#856) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:37:10 +00:00
bot-backend	da28a52067	feat(tradein): avito citywide + byrooms sweep mode (T6) (#848) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:13:19 +00:00
bot-backend	571be4233b	feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) (#847) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:11:30 +00:00
bot-backend	5d2af47b87	fix(tradein): cian SERP warm-up GET для seed cookies (_CIAN_GK/_yasc) (T2) (#846) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:01:30 +00:00
bot-backend	08509b1b78	fix(scrapers): normalize cian descriptionMinhash list[int]→str + silent-0 guard (Refs #836) (#840) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:28:56 +00:00
bot-backend	80a063b482	feat(observability): sentry alert on N consecutive scraper failures (Refs #827) (#833) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:10:29 +00:00
bot-backend	fd75f4d47a	perf(estimator): executemany batch insert in _save_yandex_history_items (Refs #826) (#832) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:07:06 +00:00
bot-backend	f7deed096b	fix(scrapers): avito per-card parse_failures counter + logging (Refs #823) (#830) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:01:50 +00:00
bot-backend	2739973285	fix(scrapers): COALESCE-backfill listing_date/area_m2 in save_listings ON CONFLICT (Refs #821) (#829) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:58:06 +00:00
bot-backend	01593be438	feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:10:36 +00:00
bot-backend	7bd5a834dc	fix(tradein): cian _cianConfig concat-формат + auth из header-frontend (#639) (#819) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:08:01 +00:00
bot-backend	b736d7b7e0	feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E) ... Закрывает последнюю часть #769 (A1/A2/B/C/D в #798/#804). Finding #17: bare-city адреса геокодились в city-центроид и сохранялись как точные lat/lon → mislocated листинги участвовали в radius ST_DWithin-аналогах. - 089: ALTER TABLE listings ADD COLUMN IF NOT EXISTS geo_precision text (idempot). - ScrapedLot.geo_precision + проброс в save_listings. - geocode_missing.py: geo_precision='city' через существующий _geocode_is_coarse. - estimator _fetch_analogs Tier H+W: + AND (geo_precision IS DISTINCT FROM 'city'). NULL проходит (консервативно, сдвига до backfill нет). Pricing не тронут. 7 тестов; 270 passed, ruff clean.	2026-05-30 21:22:21 +03:00