273 commits

Author	SHA1	Message	Date
lekss361	ea5c1e29f6	test(tradein): quality-gate anti-regression тесты на 5 P0-фиксов (#1098) ... 13 anti-regression тестов в test_781_quality_gate.py на 5 merged P0-фиксов (#755 anchor safe-guard, #753 stable dedup_hash, #773 expected_sold anchor-only, #754 avito block-page HTTP-200, #740 insufficient_data). Tests-only, прод-код не тронут. #755-scope verify: radius _filter_outliers n<5 bypass корректно не менялся (MAD-clip только в anchor). 325 passed. Closes #781. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:27:23 +00:00
lekss361	8815239e02	fix(tradein): enable deactivate_stale_avito schedule on prod (#1096) ... Миграция 100_enable_deactivate_stale_avito.sql — идемпотентный UPDATE scrape_schedules SET enabled=true, next_run_at=NOW() WHERE source='deactivate_stale_avito' AND enabled=false. Чинит QA-находку #759: 090 ON CONFLICT DO NOTHING оставил pre-seed disabled-строку → task ни разу не отработала. Non-destructive, guard на повторный прогон. +7 тестов. Closes #759. Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-06 19:08:22 +00:00
bot-backend	ed15559f80	fix(tradein): praktika wordmark logo + form button cutoff on short viewports ... Logo «поехало»: brands.logo_url хотлинкал квадратный знак logo-pink.svg (viewBox 69×69, preserveAspectRatio=none, без width/height) → CSS height:24/width:auto рендерил его 300×24 расплющенным. Self-host полного wordmark (символ #i-logo-new, viewBox 1498×194) в /trade-in/brands/praktika.svg; миграция 099 перенаправляет brands.logo_url; Topbar рендерит только wordmark (убран дублирующий текст-название); .brand-logo защищён object-fit:contain + max-width. Кнопка «Оценить» обрезалась @100% на 13" MacBook: .form-card max-height+overflow:hidden выдавливал .form-foot когда flex-shrink тела не срабатывал. Теперь скроллом владеет карточка (overflow-y:auto), body overflow-y:visible, foot flex-shrink:0. Проверено вживую на проде 1280×720.	2026-06-01 10:15:49 +03:00
lekss361	a23e650fae	fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 (#940) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 22:39:06 +00:00
bot-backend	567549c3aa	feat(tradein): cian full-load secondary-only — drop novostroyki at parse (#938) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:35:32 +00:00
lekss361	d93e96fb0c	feat(tradein): checkpoint/resume for yandex full-load — survive restart (#930) (#937) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:34:23 +00:00
bot-backend	3294bc2021	fix(tradein): exclude novostroyki from radius-median comp pool (#935) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:33:23 +00:00
lekss361	571980c56e	feat(tradein): ppm²-tier segmentation of asking→sold ratio (#928, flag OFF) (#934) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:22:39 +00:00
bot-backend	846c5014c6	feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) (#932) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:11:48 +00:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	5e94d558b5	perf(tradein): cian full-load incremental save + concurrent pagination (#929) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:27:27 +00:00
bot-backend	0883f6d0f1	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning (#926) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:54:19 +00:00
lekss361	26c0b94131	Merge pull request 'fix(tradein): add cian_proxy_url to test _mock_settings — unblock deploy (#923)' (#925) from fix/cian-proxy-test-mock into main ... Reviewed-on: #925	2026-05-31 19:34:41 +00:00
bot-backend	b9690d4d63	fix(tradein): add cian_proxy_url to test _mock_settings — unblock deploy (#923) ... PR #923 switched Cian scrapers to settings.cian_proxy_url but the shared _mock_settings helper in test_scraper_proxy.py still only exposed scraper_proxy_url, so pre-existing cian proxy-wiring tests broke with AttributeError → deploy pytest gate failed. Mirror scraper_proxy_url into cian_proxy_url in the mock.	2026-05-31 22:34:16 +03:00
bot-backend	ca0943689d	fix(tradein): skip TLS verify for sberindex.ru pull — server omits LE intermediate (#922)	2026-05-31 22:33:42 +03:00
bot-backend	02af421799	feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:30:14 +00:00
lekss361	3bd107f923	test(tradein): fix #794 test regressions on main (broken by #919 early squash-merge) (#921) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 19:05:10 +00:00
lekss361	4999c877ae	feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) (#919) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 18:48:22 +00:00
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
bot-backend	92f44ac968	feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) (#916) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:10:38 +00:00
bot-backend	639d3b0cea	feat(tradein): systemd timer scheduler — trigger script + units + SCHEDULER_ENABLE flag (#581) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:53:38 +00:00
bot-backend	441e48141b	feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:49:35 +00:00
bot-backend	f09d9737ce	fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) ... Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning a 274KB generic avito shell with 0 listings — NOT a block/captcha (no captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms wait the page.content() snapshot still had only the generic shell. At 5-6s it's the full SERP: 3.2MB, priceDetailed=50. Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works — this was the last blocker. NOT headless detection (headless=True returns real data); the abandoned xvfb/headless=virtual change is unnecessary. Refs #905, #883	2026-05-31 19:17:01 +03:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
lekss361	98b0632dae	Merge pull request 'feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)' (#904) from feat/796-domclick-scraper into main ... Reviewed-on: #904	2026-05-31 14:43:39 +00:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	4c0338ce1d	fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup)	2026-05-31 17:21:30 +03:00
lekss361	21069dea36	Merge pull request 'fix(tradein): 095 drop v_data_quality before dropping sale_type_text — RED deploy (#731)' (#903) from fix/731-sale-type-text-view-dep into main ... Reviewed-on: #903	2026-05-31 14:02:00 +00:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	5d6238e89a	fix(tradein): 095 drop v_data_quality before dropping sale_type_text (#731) ... Prod deploy 2026-05-31 13:50 failed on 095: 'cannot drop column sale_type_text of table listings because view v_data_quality depends on it'. The view (recreated in 094) has a CTE 'SELECT * FROM listings' which records a column-level dependency on EVERY listings column incl sale_type_text — so the bare DROP COLUMN aborts the whole migration (exit 1), blocking 095/096/097 from applying. Fix: DROP VIEW v_data_quality -> DROP COLUMN sale_type_text -> recreate the view with the identical 094 DDL (sale_type_text was never an output column, only pulled in implicitly via SELECT *; after the drop, * no longer includes it). Verified v_data_quality is the only object doing SELECT * FROM listings — listings_search_mv and v_price_divergence reference explicit columns, so no other dependency blocks. 095 never applied on prod (transaction rolled back atomically, not in _schema_migrations) → next deploy re-runs the corrected file. Editing the merged migration in place is the correct recovery (a new 098 would not help: the runner hits 095 first and keeps failing). Refs #731	2026-05-31 16:57:18 +03:00
bot-backend	53af9a8c55	feat(tradein): add EKB/MSK REF_AREA codes to sber_price_index pull (#794) (#900) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:56:17 +00:00
bot-backend	751059cb01	fix(tradein): camoufox fetch under app-user so runtime finds Firefox (#884) ... Build-verify on the deploy host (2026-05-31) showed the camoufox Firefox build landed in /root/.cache/camoufox (RUN python -m camoufox fetch ran as root), but the runtime app user (uid 1000) resolves the cache to /home/app/.cache/camoufox → 'browser not found' the moment scraper_fetch_mode flips to browser. Fix: run the fetch AFTER USER app + set explicit ENV HOME=/home/app (Docker does not auto-set HOME from USER, and camoufox/platformdirs keys the cache off $HOME). Now build-time fetch and runtime both use /home/app/.cache/camoufox. Verified: A) 'camoufox path' as app = /home/app/.cache/camoufox; B) that dir was empty pre-fix while /root/.cache was root-only (Permission denied to app). Dormant change — scraper_fetch_mode default stays curl_cffi (no runtime behavior change), unblocks Phase 1 (#883). Refs #884, #883	2026-05-31 16:44:27 +03:00
bot-backend	5b4531f4cc	perf(tradein): index hygiene — drop dup/redundant + de-partial is_active (#730) (#897) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:29:11 +00:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	fa2bd20ac6	refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:43 +00:00
lekss361	2890199d07	Merge pull request 'feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884)' (#892) from feat/884-browserfetcher-camoufox into main ... Reviewed-on: #892	2026-05-31 12:47:49 +00:00
lekss361	8df536ca5f	Merge pull request 'refactor(tradein): unify kadastr_num → cadastral_number (#732)' (#891) from feat/732-cadastral-unify into main ... Reviewed-on: #891	2026-05-31 12:47:10 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	42de30cd5c	refactor(tradein): unify kadastr_num -> cadastral_number (#732) ... p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql (idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) + v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number, DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass. WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo. Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply (reconstructed from migration history 050/046, not live capture). Refs #732	2026-05-31 15:41:52 +03:00
bot-backend	f3ba8acd76	feat(tradein): per-anchor watchdog timeout in city sweeps (#880) (#881) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 10:40:41 +00:00
lekss361	d1034bbc70	Merge pull request 'feat(tradein): same-building anchor safe-guard (#755)' (#877) from feat/755-anchor-safeguard into main ... Reviewed-on: #877	2026-05-31 10:36:22 +00:00
bot-backend	ec82742456	feat(tradein): same-building anchor safe-guard (#755) ... CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building anchor replaced headline median/range/confidence with an over-confident premium on as few as 2 same-building asking comps. Human-chosen (Anton) 3-part safe-guard, all implemented: 1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps. 2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium. 3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation (k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire (fallback to radius-median). Guards against price typos inflating anchor. Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen). quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor tests pass, ruff clean. MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 / Заводская 44а). Backtest needs live DB — pre-merge human/qa step. Refs #755, #694	2026-05-31 12:19:50 +03:00
bot-backend	d9b8047a7d	fix(tradein): yandex address-enrich — city-agnostic title regex (спутники ЕКБ) (#875) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 09:08:29 +00:00
lekss361	960905d47f	test(tradein): fix sweep-endpoint тесты после single-run guard (#874 follow-up) (#876) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 09:07:41 +00:00
lekss361	997eda489c	fix(tradein): single-run guard на avito-city-sweep admin-endpoint (#874) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:55:33 +00:00
lekss361	70838c7dfe	fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:27:47 +00:00
bot-backend	65b9247369	fix(matching): Tier 2b alias sync + collapse double SAVEPOINT (#849) (#870) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 08:05:48 +00:00
bot-backend	10b4b02446	fix(tradein): cian sweep houses-phase — JOIN house_sources + graceful (не fatal) (#868) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:59:26 +00:00