1565 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	8bf3d24190	Merge pull request 'feat(tradein/scrapers): detail_backfill через curl_cffi+backconnect вместо браузера/auv (развязка прокси)' (#1838) from feat/tradein-detail-backfill-curl into main ... Reviewed-on: #1838	2026-06-20 12:59:56 +00:00
bot-backend	cc3ff7c3d2	feat(tradein/scrapers): detail_backfill через curl_cffi+backconnect вместо браузера/auv (развязка прокси) ... SERP (full_load/city_sweep) и detail_backfill делили один прокси-аккаунт auv (~5 коннектов). Browser-фетч в backfill открывал десятки CONNECT-туннелей → cap превышался → HTTP 500/краши. - config.py: avito_detail_backfill_use_curl=True (ENV AVITO_DETAIL_BACKFILL_USE_CURL) - avito_detail_backfill.py: browser_mode гейтится флагом; use_curl=True → session=None, browser_fetcher=None → fetch_detail строит _build_detail_session() через scraper_proxy_url (backconnect mproxy, авто-ротация, 1 коннект/запрос) - tests: 2 новых теста (use_curl=True → BrowserFetcher не создаётся; False → создаётся)	2026-06-20 15:55:24 +03:00
lekss361	cf032182ed	Merge pull request 'feat(ptica): 3D building-massing module (Three.js) [PR#5]' (#1835) from feat/ptica-cockpit-pr5-3d into main ... Reviewed-on: #1835	2026-06-20 12:33:50 +00:00
lekss361	f6922e5527	Merge pull request 'feat(ptica): detail drawer system + 17 drawers [PR#4]' (#1834) from feat/ptica-cockpit-pr4-drawers into main ... Reviewed-on: #1834	2026-06-20 12:33:37 +00:00
lekss361	708d8b5aed	Merge pull request 'feat(ptica): Отчёты + Сравнение cockpit tabs [PR#3]' (#1833) from feat/ptica-cockpit-pr3-reports-compare into main ... Reviewed-on: #1833	2026-06-20 12:33:18 +00:00
lekss361	76311cbfe8	Merge pull request 'feat(ptica): Сценарии tab (§22 forecast) + real hero invest-score [PR#2]' (#1832) from feat/ptica-cockpit-pr2-scenarios into main ... Reviewed-on: #1832	2026-06-20 12:33:08 +00:00
lekss361	47187fe4f3	Merge pull request 'feat(site-finder): ПТИЦА dark cockpit — increment 1 (shell + hero + Development Scan)' (#1831) from feat/ptica-cockpit-pr1 into main ... Reviewed-on: #1831	2026-06-20 12:32:55 +00:00
lekss361	7b69e5a675	Merge pull request 'feat(scrapers/ui): backfill+full-load триггеры, source-фильтр, pacing-слайдер, coverage-таблица' (#1830) from feat/scrapers-ui-completeness into main ... Reviewed-on: #1830	2026-06-20 12:32:04 +00:00
lekss361	93783602c5	Merge pull request 'feat(scrapers): live pacing-регулятор + data-quality coverage API' (#1829) from feat/scrapers-config-apis into main ... Reviewed-on: #1829	2026-06-20 12:31:47 +00:00
lekss361	56e1c6e62d	Merge pull request 'feat(trade-in/ui): transparency «Мера» — analog_tier, address_precision, expected_sold бар, ratio_basis' (#1837) from feat/estimator-ui-transparency into main ... Reviewed-on: #1837	2026-06-20 12:31:19 +00:00
lekss361	485a3eb253	Merge pull request 'fix(estimator): топ-5 фиксов аудита «Мера» (anchor gate, analog_tier, freshness, MAD-order, data-age)' (#1836) from feat/estimator-audit-top5 into main ... Reviewed-on: #1836	2026-06-20 12:27:13 +00:00
bot-backend	daa2edfac3	test(estimator): unit tests для audit fixes #audit-1..5 + empty-series guard ... 18 тестов покрывают все 5 фиксов: low-conf gate, analog_tier enum, price_trend freshness, sigma=0 guard, sber staleness, thin-market flag. Попутный fix: _load_sber_index_series — guard 'if not series: continue' предотвращает max() на пустом dict при MagicMock rows (truthy=True).	2026-06-20 15:06:59 +03:00
bot-backend	a6ef2c73e1	fix(estimator): data-age guards — sber staleness + IMV thin-market (#audit-5) ... 5a: _load_sber_index_series логирует warning если latest месяц серии старее sber_index_max_age_days (дефолт 35) — расчёт не блокируется. 5b: AvitoImvSummary.thin_market (bool, дефолт False) — True когда market_count < avito_imv_thin_market_threshold (дефолт 10). Все три точки конструирования AvitoImvSummary обновлены. Warning при thin.	2026-06-20 15:03:19 +03:00
bot-backend	5b2a39cd5e	fix(estimator): MAD-clip после similarity-weighting + sigma=0 guard (#audit-4) ... sigma=0 guard: safe_area_sigma2/safe_floor_sigma2 предотвращает div/0 при нулевом sigma из конфига. Post-weight MAD-clip (estimate_sb_clip_after_weight, дефолт True) выполняется ПОСЛЕ Gaussian weighting — видовые/топ-юниты с высоким ppm² больше не выкидываются до учёта similarity. За флагом.	2026-06-20 15:01:36 +03:00
bot-backend	17edac532b	fix(estimator): Yandex history freshness filter в price_trend (#audit-3) ... _fetch_price_trend(freshness_months) — новый параметр, дефолт из settings.estimate_price_trend_max_age_months (6 мес). WHERE scraped_at > now() - N months исключает устаревшие items (yandex_valuation 2024 и т.д.) из house_placement_history fallback. houses_price_dynamics (Source 1) не затронут. psycopg v3 CAST syntax.	2026-06-20 15:00:55 +03:00
bot-backend	7080cce40e	feat(trade-in/ui): структурный analog_tier, address_precision в деталях, expected_sold бар, ratio_basis всегда ... - types/trade-in.ts: добавлен AnalogTier тип и analog_tier? поле в AggregatedEstimate (опциональное, graceful при absent/null) - HeroTransparency: resolveCompsTier() предпочитает analog_tier (структурный, без парсинга строки), fallback на compsTier() эвристику при absent/null; добавлена строка «Точность адреса» в collapsible «Как рассчитано» из address_precision с предупреждением при low+approximate - HeroSummary: третий бар «Ожидаемая цена сделки» (expected_sold_price_rub/low/high) с отличным цветом (viz-5/6) — graceful при null; ratio_basis показывается всегда: per_rooms→«по сделкам той же планировки», global_fallback→прежний текст	2026-06-20 15:00:45 +03:00
bot-backend	6b99f7d238	fix(estimator): структурный analog_tier + address_precision в API (#audit-2) ... Добавляет AggregatedEstimate.analog_tier — стабильный enum для фронта: same_building (Tier A), micro_radius (Tier C), district (S/H/0), city (W), null (нет данных). Фронт не парсит tier из explanation. address_precision уже был в схеме (подтверждено: line 208). Explanation не удаляется — фронт fallback'ает на него.	2026-06-20 15:00:12 +03:00
bot-backend	55c7bc2c72	fix(estimator): anchor low-confidence gate (#audit-1) ... Якорь с confidence=low ИЛИ (n<gate_min_n И FSD>gate_max_fsd) не заменяет headline — fallback на radius-median. Дефолты (min_n=3, max_fsd=0.20) не режут здоровые якоря (n≥4, FSD<0.15 проходят). За флагом estimate_sb_low_conf_gate_enabled (дефолт True). Логируется с причиной.	2026-06-20 14:59:05 +03:00
Light1YT	5779e512be	feat(ptica): interactive 3D building-massing module (Three.js) [PR#5] ... Port the prototype massing.js to a client-only React component (MassingScene) behind a dynamic(ssr:false) wrapper (MassingViewer), wired into the future3d + insolation drawers, replacing the СКОРО SoonCards. Generative mass from parcel area (real geometry_suitability.area_ha) × КСИТ-цель 3.5 (labelled regulation-default · НСПД — honest, real max_far not in /analyze), OrbitControls (orbit/zoom/auto-rotate), time-of-day sun with shadows (insolation preview), live GFA/КСИТ-факт metrics (green ≤3.5 / amber over). Full renderer/ scene/RAF dispose on drawer close; graceful WebGL-unavailable fallback. Adds three@0.184.0 + @types/three (package.json + package-lock.json in sync). SSR-isolated (three imported only in MassingScene). TS strict, no any.	2026-06-20 16:27:36 +05:00
Light1YT	95f93da83b	feat(ptica): right-slider detail drawer system + 17 drawers wired (PR#4) ... PticaDrawer: controlled right slider (480px / full-screen <768), scrim+Esc close, focus-on-open, body-scroll-lock (restored on close/unmount), dialog a11y. Deep-link ?drawer= via useSearchParams + shallow router.replace (no render loop; unknown keys ignored). Enables the «Подробнее» buttons on scan + hero cards. 17 drawer contents (passport/buildability/urban/restrictions/engineering/ market[4 sub-tabs]/economy/risks/georisks/oks/potential/product/finance/legal/ environment/insolation/future3d) built from ParcelAnalysis + forecast product_tz. Many are REAL: environment (noise/air_quality/wind/weather), georisks (geology/ hydrology/geotech_risk), market (competitors/market_trend/velocity/pipeline), restrictions (zouit/red_lines/zoning), engineering, passport. Absent fields (КСИТ/ОКС/финмодель/insolation) show honest «—»/«скоро» via ptica-adapt. All field paths verified against types. Dark theme scoped. TS strict, no any. Follow-up: drawer focus-trap + focus-restore for full modal a11y.	2026-06-20 16:16:39 +05:00
Light1YT	e18bb06539	feat(ptica): build Отчёты + Сравнение cockpit tabs (PR#3) ... Reports: dark export cards reusing real infra — PDF snapshot (/snapshot.pdf via shared API_BASE_URL+triggerDownload), client-side CSV (formula-injection guard + UTF-8 BOM preserved), §22 forecast DOCX/PPTX gated on forecast-ready with honest 404; full-report DOCX/PPTX disabled «скоро» (no endpoint). Section checklist mirrors the snapshot TOC. Compare: reuses compare-shortlist (localStorage, COMPARE_MAX) + ParcelCompareLoader for real per-cad analyze metrics (shared cache, rules-of-hooks safe via keyed loader). Current cad auto-pinned + non-removable, validated кадастр input, removable chips. Dark comparison table with winner highlight + colour-coded verdict (Можно/С огранич./Нельзя). Dark styles scoped under .pticaRoot[data-theme=dark]; reuse-only (ExportButtons, compare page, loaders untouched); no new deps. TS strict, no any.	2026-06-20 15:58:50 +05:00
Light1YT	b0736e3a45	feat(ptica): build Сценарии tab (§22 forecast) + real hero invest-score ... Dark cockpit Scenarios tab wired to useParcelForecastQuery (ForecastEnvelope): scenario cards (base/aggr/conserv), demand-vs-supply chart + horizons table, scenario compare table, confidence panel, recommended product (квартирография + target price + success-score), scoring transparency. Calm polling skeleton while the async forecast is pending — never an error. Hero invest-score/buy-signal now surface real values once the forecast resolves (scoring.overall×10; buy-signal from exec_summary.key_numbers.deficit_index, derived independent of the overall score), keeping the "после прогноза" placeholders while pending. advisory · §22 caption shown in both states. All forecast field paths verified against types/forecast.ts. Dark theme stays scoped under .pticaRoot[data-theme=dark]. TS strict, no any.	2026-06-20 15:48:54 +05:00
Light1YT	05eb70eb62	feat(site-finder): ПТИЦА dark cockpit — increment 1 (shell + hero + Development Scan) ... New non-destructive route /site-finder/analysis/[cad]/ptica rendering the existing /analyze data in the «ПТИЦА» operator-terminal cockpit. The light analysis page, Section1-6, globals.css and package.json are untouched. PR#1 scope: app-shell (topbar + 4 tabs + left rail), hero (dark Leaflet map + parcel passport KV-grid + Buildability radial gauge + invest-score block), and the Development Scan card grid — wired to useParcelAnalyzeQuery / ParcelAnalysis. Data honesty: real fields render live (passport egrn.*, district, utilities.summary, median_price, pipeline_24mo); absent fields show muted placeholders with source captions (buildability/risk = derived proxy «предв.»; invest/buy-signal «после прогноза»; КСИТ/height/ОКС/economy «—») via typed ptica-adapt.ts — no fake numbers presented as live. Dark theme is a CSS-module scoped under .pticaRoot[data-theme=dark] (cannot leak to the light app); IBM Plex Mono added via next/font (no dep/lockfile change). Leaflet via dynamic(ssr:false). TS strict, no any. Deferred to follow-up PRs: Scenarios (forecast), Reports+Compare tabs, 16 detail drawers, Three.js 3D massing.	2026-06-20 15:29:53 +05:00
bot-backend	00cd30b765	feat(scrapers/ui): backfill+full-load триггеры, source-фильтр runs, pacing-слайдер, coverage-таблица ... Часть 1 — недостающие триггеры: - CianBackfillSection: cian-backfill-history, cian-price-history (T8a), cian-full-load - YandexBackfillSection: yandex-address-backfill (T10), yandex-full-load, yandex-newbuilding-sweep - CianAutoLoginButton: POST /scrape/cian/auto-login в секцию Cookies - HouseImvBackfillSection: house-imv-backfill (T7) в системный раздел - GeocodeSection: geocode-missing-listings trigger + status polling Часть 2 — source-фильтр в RunsTable: - dropdown «Все / Avito / Cian / Yandex» + aria-label - фильтр прокидывается в queryKey и query-param ?source= Часть 3 — PacingControl: - GET /api/v1/admin/scraper/pacing → PacingSection - number-input + range slider (0–60с, шаг 0.5) + PUT /pacing/{source} - env_default_s подсказка; graceful-degradation на ошибку/404 Часть 4 — DataQualitySection: - GET /api/v1/admin/scraper/data-quality → таблица source × поле (% fill) - <50% → красный акцент; th scope="col"/"row" a11y - блок houses enrichment метрик; graceful-degradation на ошибку/404	2026-06-20 12:50:04 +03:00
bot-backend	b6634ea40c	feat(scrapers): live pacing-регулятор (GET/PUT /scraper/pacing) + data-quality coverage endpoint ... - browser/server.py: GET /pacing и PUT /pacing — read/write _MIN_PAGE_INTERVAL_BY_PROVIDER in-memory; валидация source in PROVIDERS, interval_s >= 0; логирует изменения; сбрасывается к env-дефолту на рестарте by design - backend/admin.py: GET /scraper/pacing — прокси к browser /pacing, PacingResponse pydantic; PUT /scraper/pacing/{source} — прокси PUT с валидацией ge=0 le=120; 502/503 при недоступности - backend/admin.py: GET /scraper/data-quality — single-pass FILTER-агрегаты по listings WHERE is_active GROUP BY source (description/photo_urls/address/lat/lon/kitchen_area_m2/living_area_m2/ ceiling_height/ceiling_height_m/metro_stations); houses (total/avito_validated_at%/rating_score%/ house_type%); house_reviews count - browser/test_server_pacing.py: 10 новых тестов GET+PUT /pacing (16 total, все зелёные) - backend/tests/test_scraper_admin_apis.py: тесты pacing-прокси + data-quality shape/pct-range (21 total, все зелёные)	2026-06-20 12:47:03 +03:00
lekss361	a425f2f200	Merge pull request 'feat(tradein/scrapers): full_load пропускает листинги обновлённые сегодня при повторном прогоне' (#1828) from feat/tradein-skip-seen-today into main ... Reviewed-on: #1828	2026-06-20 09:16:33 +00:00
bot-backend	533299ea47	feat(tradein/scrapers): full_load пропускает листинги обновлённые сегодня при повторном прогоне (skip_seen_today)	2026-06-20 12:13:32 +03:00
lekss361	7d92411347	Merge pull request 'feat(tradein-browser): per-provider page-interval override (#1812 follow-up)' (#1826) from feat/browser-per-provider-pacing into main ... Reviewed-on: #1826	2026-06-20 08:47:06 +00:00
lekss361	89590424f5	Merge pull request 'fix(tradein/db): widen offer_price_history.diff_percent (5,2)→(8,2) + clamp в триггере (краш full_load)' (#1827) from fix/tradein-diff-percent-overflow into main ... Reviewed-on: #1827	2026-06-20 08:46:58 +00:00
bot-backend	89cbddf318	fix(tradein/db): widen offer_price_history.diff_percent numeric(5,2)→(8,2) + clamp в триггере (краш full_load)	2026-06-20 11:43:23 +03:00
bot-backend	3f30c1c924	feat(tradein-browser): per-provider page-interval override (BROWSER_MIN_PAGE_INTERVAL_S_{PROVIDER}) — #1812 follow-up	2026-06-20 10:44:00 +03:00
lekss361	68f6dad62f	Merge pull request 'fix(scrapers/ui): дедуп queryKey cian test-auth + a11y (Phase-1 polish)' (#1824) from fix/scrapers-phase1-frontend into main ... Reviewed-on: #1824	2026-06-20 07:21:30 +00:00
lekss361	298b43e399	Merge pull request 'fix(scrapers/backend): socks5 health-probe + status Literal + #1820 ext_id=0 dedup' (#1825) from fix/scrapers-phase1-backend into main ... Reviewed-on: #1825	2026-06-20 07:21:20 +00:00
lekss361	6176028237	Merge pull request 'feat(tradein/ui): две равноправные hero-цены (выставить/получить) + компактный ряд бенчмарков' (#1823) from fix/tradein-estimate-hero-restructure into main ... Reviewed-on: #1823	2026-06-20 07:18:12 +00:00
bot-backend	8361961a44	fix(scrapers/backend): socks5 health-probe (httpx[socks]) + status Literal whitelist + #1820 ext_id=0 dedup-гард ... - pyproject.toml: httpx[socks]>=0.27.0 — тянет socksio, socks5:// прокси в _probe_current_ip работает прозрачно - admin.py list_scrape_runs_unified: status Literal["done","running","banned","zombie","failed","cancelled"] вместо str|None — невалидные значения → 422 - avito_houses.py save_house_catalog_enrichment: гард ext_id=0 + нет адреса → skip without DB touch; два id-less novostroyka URL не схлопываются в bogus avito:0 в house_sources - тесты: test_unified_runs_invalid_status_422 + test_unified_runs_valid_statuses_200; test_save_house_catalog_enrichment_skips_zero_ext_id_no_address + *_with_address_persists	2026-06-20 10:17:37 +03:00
bot-backend	fa65c3f212	fix(scrapers/ui): дедуп queryKey cian test-auth + a11y (th scope, табы ARIA, статус-точки aria-label) ... - cian/page.tsx: отдельный queryKey ["cian-test-auth", "manual"] для ручного testAuthQuery в CianCookiesSection — устраняет кросс-контаминацию с автопуллингом useTestAuth() (refetchInterval:60s) и ручной кнопкой "Test Current Session" - RunsTable.tsx, ScraperPage.tsx (RunsLogSection), ProxyHealthCard.tsx: scope="col" на все <th> в thead — соответствие WCAG SC 1.3.1 (Info and Relationships) - Topbar.tsx: aria-current="page" на активный nav-link вместо tablist/tabpanel — корректный паттерн для link-based навигации между отдельными страницами - ProxyHealthCard.tsx: role="img" + aria-label на цветовые dot-индикаторы статуса browser-инстансов — цвет больше не единственный носитель смысла (WCAG 1.4.1)	2026-06-20 10:14:58 +03:00
bot-backend	44fdab2d5e	feat(tradein/ui): две равноправные hero-цены (выставить/получить) + компактный ряд бенчмарков, убрать дубли	2026-06-20 10:14:27 +03:00
lekss361	7639dfe5fd	Merge pull request 'fix(scrapers): avito pipeline — новостройки house graceful + AvitoBlockedError per-item не крашит прогон (#1820)' (#1822) from fix/avito-pipeline-novostroyki-blocks into main ... Reviewed-on: #1822	2026-06-20 06:57:55 +00:00
lekss361	dae2b5a5d9	Merge pull request 'fix(tradein/ui): убрать debug cache-key, гейт тонких графиков, разграничить динамику цен' (#1821) from fix/tradein-estimate-ui-cleanup-v2 into main ... Reviewed-on: #1821	2026-06-20 06:55:03 +00:00
bot-backend	cd91d5e279	fix(scrapers): avito pipeline — новостройки house graceful (без housePage), AvitoBlockedError per-item не крашит прогон (#1820)	2026-06-20 09:53:46 +03:00
bot-backend	237c18597f	fix(tradein/ui): убрать debug cache-key, гейт тонких графиков, дедуп динамики цен	2026-06-20 09:49:15 +03:00
lekss361	86f4602f1d	Merge pull request 'feat(tradein-frontend): единая scrapers-страница — табы, proxy-health+rotate, runs-история (scrapers UI Phase 1)' (#1819) from feat/scrapers-unified-page into main ... Reviewed-on: #1819	2026-06-20 06:40:55 +00:00
lekss361	3fa35be26e	Merge pull request 'feat(admin): unified scrape/runs + scraper/health + rotate-ip API (scrapers UI Phase 1)' (#1818) from feat/scraper-admin-apis into main ... Reviewed-on: #1818	2026-06-20 06:40:35 +00:00
bot-backend	dd339d3302	feat(tradein-frontend): единая scrapers-страница — табы Avito/Cian/Yandex, proxy-health+rotate, runs-история, консолидация триггеров, удалён cian-cookies редирект	2026-06-20 09:33:07 +03:00
bot-backend	2f88a71915	feat(admin): unified scrape/runs + scraper/health + rotate-ip API для единой scrapers-страницы	2026-06-20 09:25:57 +03:00
lekss361	faa1af9c56	Merge pull request 'fix(tradein/estimator): расширить corridor-clamp slack 0.25→0.40 для премиума без own-листингов' (#1817) from fix/tradein-corridor-slack-040 into main ... Reviewed-on: #1817	2026-06-20 06:19:13 +00:00
bot-backend	3cb11355e6	fix(tradein/estimator): расширить corridor-clamp slack 0.25→0.40 для премиума без own-листингов	2026-06-20 09:16:00 +03:00
lekss361	3ff4e067e2	Merge pull request 'fix(scrapers): avito_detail_backfill — снапшот ЕКБ+active, detail browser-only без curl-fallback на бан (#1814)' (#1815) from fix/avito-detail-backfill-ekb-browser-only into main ... Reviewed-on: #1815	2026-06-20 05:34:18 +00:00
bot-backend	37fc50ad08	fix(scrapers): avito_detail_backfill — снапшот ЕКБ+active, detail browser-only без curl-fallback на бан (#1814)	2026-06-19 21:58:35 +03:00
lekss361	2402287d64	Merge pull request 'feat(browser): per-provider page pacing + resource-block tuning — снизить proxy fan-out (#1812)' (#1813) from feat/browser-reduce-page-fanout into main ... Reviewed-on: #1813	2026-06-19 18:30:13 +00:00