966 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Light1YT	f8daf50f5f	feat(site_finder): future-supply-pressure index (#950 Step 6) ... Add backend/app/services/site_finder/future_supply.py — deterministic, no-LLM §9.3 composite quantifying queued competing supply (hidden + future) vs market absorption, computed on-the-fly (not stored). - Reads VIEW v_supply_layers_latest (incl. non-derivable graddoc_stub/ insider_manual L3 rows), not a layer recompute. - Reuses market_metrics.compute_market_metrics for monthly absorption (available / months_of_supply == sold/mo; algebra verified). - months_of_pressure = (hidden + future_by_horizon) / absorption/mo; index = linear-clamp saturation at 18mo (1.5x EKB cycle); confidence=MIN (thin L2/L3 honestly drags composite to low). - L3 horizon-weighted (NULL expected_online_date -> neutral 0.5); open excluded from numerator (only hidden+future compete with a new object). - Graceful: empty/thin/no-absorption -> index None + low, div-by-zero guarded. - 54 unit tests (pure-helper math + mocked db/market_metrics).	2026-06-03 01:57:29 +05:00
Light1YT	ae85f19993	feat(workers): supply-layers refresh task + weekly beat (#950 Step 5) (#1005)	2026-06-02 20:47:05 +00:00
Light1YT	900802264a	feat(site-finder): supply_layers v2 compute service (#950 EPIC6 step3+4) (#1004)	2026-06-02 20:34:26 +00:00
Light1YT	33ecbda689	feat(db): supply_layers store + latest view (#950 PR B) (#1003)	2026-06-02 20:21:04 +00:00
Light1YT	420cda2831	feat(domrf): persist domrf_kn_objects.free_flats (#950 PR A) (#1002)	2026-06-02 20:06:29 +00:00
Light1YT	59be55f80e	feat(site-finder): per-competitor relevance_weight (#949 PR B) (#1000)	2026-06-02 19:45:38 +00:00
Light1YT	45d61ecff0	feat(site_finder): market-metrics service (#949 PR A, §9.2) (#997)	2026-06-02 19:26:26 +00:00
Light1YT	c78c6ec2d0	feat(site-finder): live CBR key-rate scraper → macro_indicator (#945 PR B) (#964)	2026-06-02 19:05:57 +00:00
Light1YT	dbae4b0bda	feat(site-finder): macro_indicator table + backfill + reader (#945 PR A) (#963)	2026-06-02 18:41:28 +00:00
lekss361	6e2bb80690	Merge pull request 'fix(tradein): collapse CRM block + density so input form fits laptops' (#943) from fix/tradein-form-fit-laptops into main ... Reviewed-on: #943	2026-06-01 08:02:47 +00:00
bot-backend	702f322b22	fix(tradein): collapse CRM block + density so form fits laptops ... Форма ввода (10 полей + CRM-блок 221px) не помещалась в .form-card (max-height 100vh-104px) на ноутбуках: переполнение 58px @1440×900, ~158px @13" MacBook. CRM «для менеджера» теперь сворачиваемый (по умолчанию закрыт, открывается по клику) + лёгкое уплотнение отступов (.field 12→9, form-body/foot padding). Высота формы 854→646px → влезает без скролла на 1366×768 / 1440×900 / 13" MacBook @100%. Sub-720 — fallback на скролл-фикс из этой же ветки. Проверено вживую (injection) на проде, tsc --noEmit clean.	2026-06-01 10:58:43 +03:00
lekss361	8751fe819b	Merge pull request 'fix(tradein): praktika wordmark logo + form button cutoff on short viewports' (#942) from fix/tradein-praktika-logo-form-scroll into main ... Reviewed-on: #942	2026-06-01 07:18:49 +00:00
bot-backend	ed15559f80	fix(tradein): praktika wordmark logo + form button cutoff on short viewports ... Logo «поехало»: brands.logo_url хотлинкал квадратный знак logo-pink.svg (viewBox 69×69, preserveAspectRatio=none, без width/height) → CSS height:24/width:auto рендерил его 300×24 расплющенным. Self-host полного wordmark (символ #i-logo-new, viewBox 1498×194) в /trade-in/brands/praktika.svg; миграция 099 перенаправляет brands.logo_url; Topbar рендерит только wordmark (убран дублирующий текст-название); .brand-logo защищён object-fit:contain + max-width. Кнопка «Оценить» обрезалась @100% на 13" MacBook: .form-card max-height+overflow:hidden выдавливал .form-foot когда flex-shrink тела не срабатывал. Теперь скроллом владеет карточка (overflow-y:auto), body overflow-y:visible, foot flex-shrink:0. Проверено вживую на проде 1280×720.	2026-06-01 10:15:49 +03:00
lekss361	a23e650fae	fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 (#940) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 22:39:06 +00:00
bot-backend	921c14e054	feat(trade-in): OfferCard — убрать редактируемую цену + ползунок торга, разбивка на дефолтах (#939) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 22:31:20 +00:00
bot-backend	567549c3aa	feat(tradein): cian full-load secondary-only — drop novostroyki at parse (#938) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:35:32 +00:00
lekss361	d93e96fb0c	feat(tradein): checkpoint/resume for yandex full-load — survive restart (#930) (#937) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:34:23 +00:00
bot-backend	3294bc2021	fix(tradein): exclude novostroyki from radius-median comp pool (#935) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:33:23 +00:00
lekss361	571980c56e	feat(tradein): ppm²-tier segmentation of asking→sold ratio (#928, flag OFF) (#934) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:22:39 +00:00
bot-backend	846c5014c6	feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) (#932) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:11:48 +00:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	5e94d558b5	perf(tradein): cian full-load incremental save + concurrent pagination (#929) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:27:27 +00:00
bot-backend	ecf8add2ce	fix(browser): pageError.location needs valid defaults, not just optional chaining (#918 follow-up) (#927) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:15:39 +00:00
bot-backend	0883f6d0f1	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning (#926) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:54:19 +00:00
lekss361	26c0b94131	Merge pull request 'fix(tradein): add cian_proxy_url to test _mock_settings — unblock deploy (#923)' (#925) from fix/cian-proxy-test-mock into main ... Reviewed-on: #925	2026-05-31 19:34:41 +00:00
lekss361	a5457af716	Merge pull request 'fix(tradein): skip TLS verify for sberindex.ru pull — server omits LE intermediate (#922)' (#924) from fix/922-sber-tls-verify into main ... Reviewed-on: #924	2026-05-31 19:34:24 +00:00
bot-backend	b9690d4d63	fix(tradein): add cian_proxy_url to test _mock_settings — unblock deploy (#923) ... PR #923 switched Cian scrapers to settings.cian_proxy_url but the shared _mock_settings helper in test_scraper_proxy.py still only exposed scraper_proxy_url, so pre-existing cian proxy-wiring tests broke with AttributeError → deploy pytest gate failed. Mirror scraper_proxy_url into cian_proxy_url in the mock.	2026-05-31 22:34:16 +03:00
bot-backend	ca0943689d	fix(tradein): skip TLS verify for sberindex.ru pull — server omits LE intermediate (#922)	2026-05-31 22:33:42 +03:00
bot-backend	02af421799	feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:30:14 +00:00
lekss361	3bd107f923	test(tradein): fix #794 test regressions on main (broken by #919 early squash-merge) (#921) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 19:05:10 +00:00
lekss361	4999c877ae	feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix (#794) (#919) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 18:48:22 +00:00
bot-backend	19cf1162a0	fix(browser): null-guard pageError.location in playwright bundle (#915) (#918) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:45:41 +00:00
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
bot-backend	92f44ac968	feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) (#916) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:10:38 +00:00
bot-backend	639d3b0cea	feat(tradein): systemd timer scheduler — trigger script + units + SCHEDULER_ENABLE flag (#581) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:53:38 +00:00
bot-backend	441e48141b	feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:49:35 +00:00
lekss361	9d357f0cc4	Merge pull request 'fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) — migration proven' (#912) from fix/905-browser-wait-ms into main ... Reviewed-on: #912	2026-05-31 16:20:42 +00:00
bot-backend	f09d9737ce	fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) ... Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning a 274KB generic avito shell with 0 listings — NOT a block/captcha (no captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms wait the page.content() snapshot still had only the generic shell. At 5-6s it's the full SERP: 3.2MB, priceDetailed=50. Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works — this was the last blocker. NOT headless detection (headless=True returns real data); the abandoned xvfb/headless=virtual change is unnecessary. Refs #905, #883	2026-05-31 19:17:01 +03:00
lekss361	c43a25ef42	Merge pull request 'fix(browser): camoufox reads AVITO_PROXY_URL → real avito data not soft-block stub (#905)' (#911) from fix/905-browser-proxy-env into main ... Reviewed-on: #911	2026-05-31 15:51:43 +00:00
bot-backend	3527a6bc8d	style(browser): fix pre-existing ruff lints in server.py (RUF100, E501) ... Drive-by: browser/server.py was never linted (pre-commit ruff is scoped to backend/), so #909 left an unused noqa + a 101-char line. Cleaned up while editing this file for the proxy fix.	2026-05-31 18:50:35 +03:00
bot-backend	57acec215f	fix(browser): read AVITO_PROXY_URL for camoufox proxy (#905) ... Prod smoke: tradein-browser fetched avito but got the ~400KB soft-block stub (preloadedState present but priceDetailed=0, firewall=0) — because camoufox ran with NO proxy (datacenter IP). Root cause: server.py read SCRAPER_PROXY_URL, but the proxy in backend/.env.runtime is named AVITO_PROXY_URL (mobileproxy, #623) — the same var curl_cffi-avito uses. So the var was empty in the browser container. Fix: read AVITO_PROXY_URL first, SCRAPER_PROXY_URL as a generic fallback. One-line env-name swap; the value is the working mobileproxy egress already on the host. After deploy, re-run tradein-mvp/scripts/browser-smoke.sh — expect real data (priceDetailed>0, ~1.4MB, firewall 0). IP rotation (AVITO_PROXY_ROTATE_URL) stays backend-side and keeps working (same proxy endpoint, changing egress IP). Refs #905, #883, #623	2026-05-31 18:49:45 +03:00
lekss361	71874310aa	Merge pull request 'chore(tradein): browser-smoke.sh — verify tradein-browser /health + /fetch (#905)' (#910) from chore/browser-smoke-script into main ... Reviewed-on: #910	2026-05-31 15:43:24 +00:00
bot-backend	71f05308e9	chore(tradein): browser-smoke.sh — verify tradein-browser /health + /fetch (#905) ... Reusable smoke test for the camoufox HTTP service. Run from the deploy host: docker exec -i tradein-backend sh < tradein-mvp/scripts/browser-smoke.sh Reports health, byte count, preloadedState presence, firewall-block detection + first 200 bytes of the response. Avoids long one-line curl commands that get mangled on terminal paste. Refs #905, #883	2026-05-31 18:42:17 +03:00
lekss361	4aeeef8c2d	Merge pull request 'fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905)' (#909) from fix/905-browser-http-wrapper into main ... Reviewed-on: #909	2026-05-31 15:27:10 +00:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	bb64fe247a	Merge pull request 'fix(browser): run camoufox fetch as root so geoip mmdb write succeeds — RED build-browser (#905)' (#908) from fix/905-camoufox-geoip-mmdb into main ... Reviewed-on: #908	2026-05-31 14:53:04 +00:00
bot-backend	393ecea458	fix(browser): run camoufox fetch as root so geoip mmdb write succeeds (#905) ... CI build-browser (and the deploy) went red on #907's browser image: PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.12/site-packages/camoufox/GeoLite2-City.mmdb' camoufox fetch with geoip=True downloads TWO things: the Firefox build (→ $HOME/.cache, correctly routed to /home/app via ENV HOME, the #899 fix) AND the GeoLite2 mmdb, which it writes INTO the camoufox package dir in site-packages. Since the browser image pip-installs camoufox as root, that dir is root-owned, so running fetch as the app user can't write the mmdb → build fails. Fix: run fetch as root (writes both the mmdb and — via ENV HOME — Firefox into /home/app/.cache), then chown the Firefox cache to app. The mmdb stays root-owned in site-packages but world-readable, which is all the runtime app user needs. Verified by a real local image build (camoufox fetch now completes, image exports). The deploy was gated off by the failed build-browser job (not partially applied), so prod was never broken. Refs #905, #883, #899	2026-05-31 17:51:43 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
lekss361	98b0632dae	Merge pull request 'feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)' (#904) from feat/796-domclick-scraper into main ... Reviewed-on: #904	2026-05-31 14:43:39 +00:00
bot-backend	0566fed5a2	ci(tradein): build+push tradein-browser image in deploy workflow (#905) ... The new browser service references ghcr.io/lekss361/gendesign-tradein-browser; without a build job the deploy's compose-up would fail pulling a missing image. Adds build-browser job (mirrors build-backend, context ./tradein-mvp/browser), a browser path-filter (tradein-mvp/browser/**), IMAGE_BROWSER env, and wires build-browser into deploy needs + the no-failure gate. Refs #905	2026-05-31 17:29:13 +03:00