37 commits

Author	SHA1	Message	Date
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
bot-backend	639d3b0cea	feat(tradein): systemd timer scheduler — trigger script + units + SCHEDULER_ENABLE flag (#581) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:53:38 +00:00
bot-backend	f09d9737ce	fix(browser): bump fetch wait 2500→6000ms so avito listings hydrate (#905) ... Prod debug (ssh, headless=True + mobileproxy): the camoufox fetch was returning a 274KB generic avito shell with 0 listings — NOT a block/captcha (no captcha/datadome/firewall markers, status 200, correct final_url). Avito hydrates the SERP listings into the DOM client-side AFTER domcontentloaded; at the 2500ms wait the page.content() snapshot still had only the generic shell. At 5-6s it's the full SERP: 3.2MB, priceDetailed=50. Fix: BROWSER_WAIT_MS default 2500→6000 (5s proven full + 1s margin), config.py browser_wait_ms 2500→6000 to match. The whole camoufox+proxy migration works — this was the last blocker. NOT headless detection (headless=True returns real data); the abandoned xvfb/headless=virtual change is unnecessary. Refs #905, #883	2026-05-31 19:17:01 +03:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	ec82742456	feat(tradein): same-building anchor safe-guard (#755) ... CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building anchor replaced headline median/range/confidence with an over-confident premium on as few as 2 same-building asking comps. Human-chosen (Anton) 3-part safe-guard, all implemented: 1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps. 2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium. 3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation (k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire (fallback to radius-median). Guards against price typos inflating anchor. Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen). quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor tests pass, ruff clean. MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 / Заводская 44а). Backtest needs live DB — pre-merge human/qa step. Refs #755, #694	2026-05-31 12:19:50 +03:00
bot-backend	3f31d1f409	fix(estimator): sanity-clamp quarter-index factor (#859) (#863) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:20:59 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	b16a45ef87	feat(tradein): per-cadastral-quarter price index in estimator (#764) (#858) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:53:21 +00:00
bot-backend	571be4233b	feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) (#847) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:11:30 +00:00
bot-backend	01593be438	feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:10:36 +00:00
bot-backend	3653f020b6	fix(ratio): ppm² потолок asking→sold в конфиг, default 1.2M (Refs #767) (#790) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:57:09 +00:00
bot-backend	96027357b5	feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726) (#788) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:52:35 +00:00
bot-backend	bcd035034a	fix(tradein): SSRF host-allowlist на admin scrape + удалён dead ADMIN_TOKEN (Refs #756) (#778) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:27:19 +00:00
bot-backend	f43d593cf6	feat(tradein): brand-by-account + result IA redesign (distinct asking/sale prices) (#683) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 06:45:14 +00:00
bot-backend	3361e434d4	feat(tradein): refine same-building anchor for within-building heterogeneity (#681) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 05:49:46 +00:00
bot-backend	03a3296332	feat(tradein): same-building anchor estimator (#651/#652 v2) behind flag (#677) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 21:01:21 +00:00
bot-backend	96f02d37f3	feat(tradein): blend Avito IMV anchor into estimate + ДКП corridor advisory (#651, #652) (#674) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 17:42:06 +00:00
bot-backend	29679a9545	fix(tradein): time-budget estimate enrichment + observable errors to stop opaque 502 (#654) (#673) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 17:22:02 +00:00
bot-backend	ed4c8b3f02	feat(tradein): configurable estimate quota + new exhausted-quota copy (#658) (#671) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 16:07:02 +00:00
bot-backend	6ddc9bc661	fix(tradein): bypass rate-limit for authenticated pilots + configurable limit (#655) (#669) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-29 15:56:30 +00:00
Light1YT	a7fa1a4ffa	feat(tradein): DaData on-demand enrichment в estimate flow (PR Q1) ... Добавляет app/services/dadata.py — async client для DaData /clean/address, обогащающий target адрес canonical-формой, kadastr_num, ФИАС, координатами и ближайшим метро. Migration 069 расширяет trade_in_estimates 6 nullable колонками; estimator.estimate_quality вызывает service после geocode и сохраняет результат в основной INSERT. AggregatedEstimate отдаёт canonical_address / house_cadnum / house_fias_id / metro_nearest наружу. Service graceful: пустые credentials / quota 429 / 5xx / network fail → None, estimator продолжает работу. Demo tier 100 req/день — хватит для low-traffic prod и тестов. ENV: DADATA_API_TOKEN, DADATA_API_SECRET. Tests: 20 unit-тестов через httpx.MockTransport, zero real DaData calls. Покрыто: happy path, отсутствие creds, короткий адрес, timeout/network, HTTP 429/401/403/500/502/503, пустой массив, malformed shape, partial payload, coerce строковых qc-кодов в int, проверка request body+headers. Закладывает базу для PR Q2 (frontend DaData Suggest, 10k/день free) и PR Q3 (backfill 339 NULL-coords houses) — обе используют тот же сервис.	2026-05-27 18:50:12 +05:00
Light1YT	1ce9e10012	feat(tradein): persistent infra для backfill scripts (PR F) (#592)	2026-05-27 06:59:54 +00:00
Light1YT	88cdfd6adb	feat(rbac): role-based access control via X-Authenticated-User middleware (#585) ... Backend RBAC + Caddy header_up. Closes part of #fixes-rbac-pra.	2026-05-26 06:18:40 +00:00
lekss361	94cf199276	feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) ... Replaces tradein.cad_buildings snapshot with live postgres_fdw foreign table reading gendesign.v_tradein_cad_buildings. Fixes /trade-in/api/v1/geocode/reverse 500 (Nominatim ban) and address_not_geocoded for cadastre addresses (e.g. Хохрякова 81). Security (deep-review fixes): - 100_tradein_fdw_role.sql: passwordless CREATE ROLE; password set by deploy.yml ALTER ROLE bootstrap reading GENDESIGN_FDW_PASSWORD from backend/.env.runtime (via psql :'pw' var → format %L — injection-safe). - core/fdw.py: regex whitelist [A-Za-z0-9_-]{32,256} on password, ValueError without echoing value, try/rollback on commit. - 060_postgres_fdw_extension.sql: connect_timeout='3' on FOREIGN SERVER + ALTER ADD/SET fallback. - geocoder.py: _cadastral_forward_sync / _cadastral_reverse_sync wrapped in asyncio.to_thread. - 100_*.sql: REVOKE ALL ON ALL TABLES/SEQUENCES/FUNCTIONS IN SCHEMA public; only GRANT SELECT on v_tradein_cad_buildings. - pg_user_mappings query handles PUBLIC mapping (usename IS NULL). Tests: 3 SQL-injection guards on ensure_fdw_user_mapping + rewritten cadastral suite.	2026-05-24 08:57:30 +00:00
lekss361	0efd2dca1d	feat(tradein): cadastral reverse geocode via main backend cad_buildings (#492)	2026-05-24 07:31:12 +00:00
lekss361	9643979d2c	feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) ... POST /api/v1/search reading listings_search_mv with ~20 cross-source filters, parameterized SQL (CAST(:x AS type), psycopg v3), whitelisted ORDER BY (Pydantic Literal), Redis 5min hot cache with graceful degradation (singleton pool via lru_cache). Verified vs data/sql/050_search_optimization.sql: matview column refs (total_area, lng, house_rating, sources[], has_avito/cian/yandex), SQL injection safety, cache failure swallow, router prefix. Deep-code-reviewer: APPROVE.	2026-05-23 14:44:22 +00:00
lekss361	dbad7a626b	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474)	2026-05-23 14:21:01 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
lekss361	5e3ba351a0	fix(tradein): security hotfix — C-3 config default + C-4 empty estimate persist + C-6 PDF URL allowlist (#435)	2026-05-23 09:38:23 +00:00
TradeIn Deploy	59ad41c183	feat(tradein): мониторинг ошибок через GlitchTip (#396) ... Подключаем tradein-backend к GlitchTip (Sentry-совместимый, уже на сервере). Необработанные исключения FastAPI улетают в дашборд — баг estimate-500 (#388/#389) был бы виден сразу с трейсбеком. - sentry-sdk в зависимостях. - main.py — sentry_sdk.init(), env-gated: пусто GLITCHTIP_DSN → off. - config.glitchtip_dsn + GLITCHTIP_DSN в docker-compose.prod.yml. - traces_sample_rate=0 (только ошибки), send_default_pii=False. Активация: создать проект в GlitchTip → DSN в .env.runtime.	2026-05-22 11:18:26 +05:00
TradeIn Deploy	63c96628e3	feat(tradein): прогноз срока продажи + rate-limit (#9,#16)	2026-05-21 20:12:49 +03:00
TradeIn Deploy	7adb7cff66	feat(tradein): auth на admin-эндпоинтах (#2) ... /api/v1/admin/scrape и geocode-missing закрыты X-Admin-Token. settings.admin_token (env ADMIN_TOKEN). Пусто = открыто (dev), задано = требуем заголовок. cron шлёт токен из .env.runtime.	2026-05-21 19:47:01 +03:00
lekss361	02267d0306	feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) ... - FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex) - Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in - WeasyPrint PDF (Брусника-style 4-page report) - Address autocomplete с typo-tolerance + 6 EKB presets - Изолированный docker stack gendesign-tradein (отдельная postgres БД) - Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/* - Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login) - Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml) - Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest Первый запуск на сервере (вручную, один раз): - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email) - docker network create gendesign_shared (если нет) - docker compose -p gendesign-tradein up -d - docker compose -p gendesign exec caddy caddy reload	2026-05-21 00:25:39 +03:00