967 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	0566fed5a2	ci(tradein): build+push tradein-browser image in deploy workflow (#905) ... The new browser service references ghcr.io/lekss361/gendesign-tradein-browser; without a build job the deploy's compose-up would fail pulling a missing image. Adds build-browser job (mirrors build-backend, context ./tradein-mvp/browser), a browser path-filter (tradein-mvp/browser/**), IMAGE_BROWSER env, and wires build-browser into deploy needs + the no-failure gate. Refs #905	2026-05-31 17:29:13 +03:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	4c0338ce1d	fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup)	2026-05-31 17:21:30 +03:00
lekss361	91d51e60d6	Merge pull request 'chore(agents): reviewer back to Opus (revert model-part of aa3d012)' (#906) from chore/reviewer-opus-model into main ... Reviewed-on: #906	2026-05-31 14:17:32 +00:00
bot-backend	6e0039dd47	chore(agents): reviewer back to Opus (revert model-part of aa3d012) ... Per user decision: reviewer = Opus (merge-authority needs strong reasoning on verdict); остальные loop-роли остаются Sonnet. Реверсит только model-часть aa3d012 (reviewer→sonnet). - start-bot.ps1: $model = reviewer? opus : sonnet + --model $model - auto-code-reviewer.md: frontmatter model: opus + doc reviewer на Opus	2026-05-31 17:16:57 +03:00
lekss361	21069dea36	Merge pull request 'fix(tradein): 095 drop v_data_quality before dropping sale_type_text — RED deploy (#731)' (#903) from fix/731-sale-type-text-view-dep into main ... Reviewed-on: #903	2026-05-31 14:02:00 +00:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	5d6238e89a	fix(tradein): 095 drop v_data_quality before dropping sale_type_text (#731) ... Prod deploy 2026-05-31 13:50 failed on 095: 'cannot drop column sale_type_text of table listings because view v_data_quality depends on it'. The view (recreated in 094) has a CTE 'SELECT * FROM listings' which records a column-level dependency on EVERY listings column incl sale_type_text — so the bare DROP COLUMN aborts the whole migration (exit 1), blocking 095/096/097 from applying. Fix: DROP VIEW v_data_quality -> DROP COLUMN sale_type_text -> recreate the view with the identical 094 DDL (sale_type_text was never an output column, only pulled in implicitly via SELECT *; after the drop, * no longer includes it). Verified v_data_quality is the only object doing SELECT * FROM listings — listings_search_mv and v_price_divergence reference explicit columns, so no other dependency blocks. 095 never applied on prod (transaction rolled back atomically, not in _schema_migrations) → next deploy re-runs the corrected file. Editing the merged migration in place is the correct recovery (a new 098 would not help: the runner hits 095 first and keeps failing). Refs #731	2026-05-31 16:57:18 +03:00
bot-backend	53af9a8c55	feat(tradein): add EKB/MSK REF_AREA codes to sber_price_index pull (#794) (#900) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:56:17 +00:00
lekss361	f94608a5f3	Merge pull request 'fix(tradein): camoufox fetch under app-user so runtime finds Firefox (#884)' (#899) from fix/884-camoufox-cache-path into main ... Reviewed-on: #899	2026-05-31 13:46:08 +00:00
bot-backend	751059cb01	fix(tradein): camoufox fetch under app-user so runtime finds Firefox (#884) ... Build-verify on the deploy host (2026-05-31) showed the camoufox Firefox build landed in /root/.cache/camoufox (RUN python -m camoufox fetch ran as root), but the runtime app user (uid 1000) resolves the cache to /home/app/.cache/camoufox → 'browser not found' the moment scraper_fetch_mode flips to browser. Fix: run the fetch AFTER USER app + set explicit ENV HOME=/home/app (Docker does not auto-set HOME from USER, and camoufox/platformdirs keys the cache off $HOME). Now build-time fetch and runtime both use /home/app/.cache/camoufox. Verified: A) 'camoufox path' as app = /home/app/.cache/camoufox; B) that dir was empty pre-fix while /root/.cache was root-only (Permission denied to app). Dormant change — scraper_fetch_mode default stays curl_cffi (no runtime behavior change), unblocks Phase 1 (#883). Refs #884, #883	2026-05-31 16:44:27 +03:00
lekss361	07b64bc740	Merge pull request 'chore(agents): bot harness tuning — per-role model, forgejo MCP deferral, vault capture, #893 curl hardening' (#898) from chore/bot-harness-tuning into main ... Reviewed-on: #898	2026-05-31 13:30:47 +00:00
bot-backend	5b4531f4cc	perf(tradein): index hygiene — drop dup/redundant + de-partial is_active (#730) (#897) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:29:11 +00:00
bot-backend	5fa9588935	chore(agents): bot harness tuning — per-role model, forgejo MCP deferral, vault capture, #893 curl hardening ... - start-bot.ps1: --model per-role (sonnet; opus только reviewer); CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=60 - .claude/mcp/*.json: forgejo (~90 tools) deferred во всех ролях; postgres-gendesign deferred в backend/reviewer - auto-analyst.md: knowledge-capture step (status/done → inbox draft → vault-overlord) — закрывает 'волт не обновляется' - forgejo ops через mcp__forgejo__* вместо сырого curl: auto-code-reviewer/backend/qa workflow + _autonomous_pickup contract - #893 guard: бан curl→/tmp→python3 (Windows 404+FileNotFoundError); deferred→ToolSearch; stale tool-name cleanup (deep-reviewer phases)	2026-05-31 16:26:17 +03:00
bot-frontend	191ed03d3c	fix(tradein): RouteGuard prod 401 → session-expired screen, stop re-subscribe storm (#800) (#896) ... Co-authored-by: bot-frontend <bot-frontend@gendsgn.local> Co-committed-by: bot-frontend <bot-frontend@gendsgn.local>	2026-05-31 13:17:44 +00:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	fa2bd20ac6	refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:43 +00:00
lekss361	2890199d07	Merge pull request 'feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884)' (#892) from feat/884-browserfetcher-camoufox into main ... Reviewed-on: #892	2026-05-31 12:47:49 +00:00
lekss361	8df536ca5f	Merge pull request 'refactor(tradein): unify kadastr_num → cadastral_number (#732)' (#891) from feat/732-cadastral-unify into main ... Reviewed-on: #891	2026-05-31 12:47:10 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
lekss361	22c6d67877	Merge pull request 'docs(agents): usage-limit idle-backoff (weekly cap) + qa terminal для non-UI status/qa' (#889) from docs/agent-usage-throttle-qa-terminal into main ... Reviewed-on: #889	2026-05-31 12:43:36 +00:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	42de30cd5c	refactor(tradein): unify kadastr_num -> cadastral_number (#732) ... p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql (idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) + v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number, DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass. WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo. Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply (reconstructed from migration history 050/046, not live capture). Refs #732	2026-05-31 15:41:52 +03:00
bot-backend	aa3d012fdf	chore(agents): bot-окна на Sonnet (start-bot --model) + context-hygiene + reviewer/resolver→sonnet (weekly Opus-пул)	2026-05-31 15:39:42 +03:00
bot-backend	06cff308d1	docs(agents): usage-limit idle-backoff (weekly cap) + qa terminal для non-UI status/qa items	2026-05-31 15:35:20 +03:00
bot-frontend	3b38b3f004	feat(tradein-fe): уведомление при global_fallback asking→sold ratio (тонкий сегмент) (#882) ... Co-authored-by: bot-frontend <bot-frontend@gendsgn.local> Co-committed-by: bot-frontend <bot-frontend@gendsgn.local>	2026-05-31 10:41:54 +00:00
bot-backend	f3ba8acd76	feat(tradein): per-anchor watchdog timeout in city sweeps (#880) (#881) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 10:40:41 +00:00
lekss361	d1034bbc70	Merge pull request 'feat(tradein): same-building anchor safe-guard (#755)' (#877) from feat/755-anchor-safeguard into main ... Reviewed-on: #877	2026-05-31 10:36:22 +00:00
bot-frontend	ee8f72da39	feat(tradein): unify scraper admin pages (avito/cian/yandex) + /scrapers hub (#878) ... Co-authored-by: bot-frontend <bot-frontend@gendsgn.local> Co-committed-by: bot-frontend <bot-frontend@gendsgn.local>	2026-05-31 09:31:47 +00:00
bot-backend	ec82742456	feat(tradein): same-building anchor safe-guard (#755) ... CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building anchor replaced headline median/range/confidence with an over-confident premium on as few as 2 same-building asking comps. Human-chosen (Anton) 3-part safe-guard, all implemented: 1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps. 2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium. 3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation (k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire (fallback to radius-median). Guards against price typos inflating anchor. Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen). quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor tests pass, ruff clean. MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 / Заводская 44а). Backtest needs live DB — pre-merge human/qa step. Refs #755, #694	2026-05-31 12:19:50 +03:00
bot-backend	d9b8047a7d	fix(tradein): yandex address-enrich — city-agnostic title regex (спутники ЕКБ) (#875) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 09:08:29 +00:00
lekss361	960905d47f	test(tradein): fix sweep-endpoint тесты после single-run guard (#874 follow-up) (#876) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 09:07:41 +00:00
lekss361	997eda489c	fix(tradein): single-run guard на avito-city-sweep admin-endpoint (#874) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:55:33 +00:00
lekss361	70838c7dfe	fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:27:47 +00:00
bot-frontend	1684c6ed3e	fix(tradein-fe): scraper-страница падала — scraper-settings {settings:[]} + string delay (#872) ... Co-authored-by: bot-frontend <bot-frontend@gendsgn.local> Co-committed-by: bot-frontend <bot-frontend@gendsgn.local>	2026-05-31 08:21:17 +00:00
bot-backend	65b9247369	fix(matching): Tier 2b alias sync + collapse double SAVEPOINT (#849) (#870) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 08:05:48 +00:00
bot-frontend	e85d2980b6	fix(tradein): money-screen contrast AA (#834 fixup) (#869) ... Co-authored-by: bot-frontend <bot-frontend@gendsgn.local> Co-committed-by: bot-frontend <bot-frontend@gendsgn.local>	2026-05-31 08:00:18 +00:00
bot-backend	10b4b02446	fix(tradein): cian sweep houses-phase — JOIN house_sources + graceful (не fatal) (#868) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:59:26 +00:00
bot-backend	60353beaaf	feat(tradein): yandex city-sweep — один прогон SERP+address-enrich (#867) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:43:13 +00:00
bot-backend	2dd2d8e57e	feat(tradein): schedule yandex address backfill task (#855) (#866) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:31:10 +00:00
bot-backend	f2a123e2aa	feat(tradein): cian city-sweep — один прогон SERP+detail(+price-history)+houses (#864) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:24:34 +00:00
bot-backend	3f31d1f409	fix(estimator): sanity-clamp quarter-index factor (#859) (#863) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:20:59 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	d78ae9f649	feat(tradein): avito city-sweep — IMV-фаза (один прогон = SERP+detail+houses+IMV) (#861) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:09:42 +00:00
bot-backend	4494866574	fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:04:31 +00:00
bot-backend	b16a45ef87	feat(tradein): per-cadastral-quarter price index in estimator (#764) (#858) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:53:21 +00:00
lekss361	5cfab0b23e	fix(bots): подключить postgres-tradein MCP в analyst/backend bot-окна (#857)	2026-05-31 06:44:46 +00:00
bot-backend	5373f5db63	feat(tradein): admin backfill endpoints — cian price-history (T8a) + yandex address (T10) + house IMV (T7) (#856) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:37:10 +00:00
lekss361	a920d057da	Merge pull request 'chore(agents): analyst inbound queue (status/needs-analysis) + tradein DB access' (#852) from chore/analyst-inbound-queue-tradein-db into main ... Reviewed-on: #852	2026-05-31 06:36:37 +00:00
bot-backend	4795bcd3a3	chore(agents): analyst — разбирает всю inbound-очередь за тик + параллельные саб-агенты, без throttle ... - INBOUND PICKUP: забирает ВСЕ status/needs-analysis тикеты в одном тике (не один-за-тик), затем продолжает на proactive в том же тике. - Убран THROTTLE по размеру ready-очереди (gate ready≥10 + hard-rule) — декомпозирует всё найденное, дубли отсекает по pipeline-карте, не по счётчику. - Новая секция «Параллельный анализ через саб-агенты»: ≥2 непересекающихся work-item → параллельные read-only Explore/general-purpose на archeology, synthesize+CREATE делает analyst (single writer); hot-file overlap → sequential. - +Task в whitelist. Перенумерованы шаги (1-9) + cross-refs.	2026-05-31 09:34:04 +03:00