1713 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
lekss361	65cec74cca	Merge pull request 'feat(scrapers): cian SERP на BrowserFetcher (camoufox), снят code-side rotate (#1806)' (#1810) from feat/cian-serp-browser into main ... Reviewed-on: #1810	2026-06-19 17:51:59 +00:00
lekss361	c88170b680	Merge pull request 'fix(tradein-frontend): Мера estimate — hydration #418 + photos 422 (#1803 #1804)' (#1808) from fix/mera-estimate-ui-artifacts into main ... Reviewed-on: #1808	2026-06-19 17:49:44 +00:00
bot-backend	cbc20cfe54	fix(tradein-frontend): Мера estimate — hydration #418 + photos 422 graceful (#1803 #1804) ... #1803 HeroTransparency: formatRuDateTime добавляет timeZone:"UTC" (SSR Node UTC vs клиент Europe/Moscow давали разный час → React minified error #418). Date.now() fallback (data_freshness_minutes без last_scraped_at) вынесен в useEffect/useState — не выполняется при SSR. #1804 PhotoUpload: guard isRealEstimateId() проверяет UUID4 regex и отклоняет preview-0000-... — GET/POST /photos с фейковым ID больше не слаются → нет 422. Счётчик показывает «— / 12» вместо «0 / 12», input disabled в preview-режиме.	2026-06-19 20:44:28 +03:00
bot-backend	93f7c9ae3f	feat(scrapers): avito SERP дефолт на BrowserFetcher (camoufox), curl_cffi fallback (#1805)	2026-06-19 20:42:04 +03:00
bot-backend	b1fb0c5711	feat(scrapers): cian SERP на BrowserFetcher (camoufox), снят code-side rotate (#1806)	2026-06-19 20:41:44 +03:00
lekss361	abbe0b871d	Merge pull request 'fix(tradein/estimator): clamp premium headline к коридору сделок + робастный anchor (#1795)' (#1807) from fix/tradein-estimator-premium-clamp-1795 into main ... Reviewed-on: #1807	2026-06-19 17:37:58 +00:00
bot-backend	494f2e00e0	fix(tradein/estimator): clamp premium headline к коридору сделок + робастный anchor (#1795)	2026-06-19 20:34:48 +03:00
lekss361	a534dece06	Merge pull request 'test(browser): удалить устаревшие pool-routing тесты (CI red после #1793)' (#1802) from fix/remove-obsolete-browser-pool-tests into main ... Reviewed-on: #1802	2026-06-19 17:13:50 +00:00
bot-backend	8467cc647f	test(browser): remove obsolete pool-routing tests (FEATURE_BROWSER_POOL_ENABLED/_build_proxy_map removed in #1793 per-provider rewrite; routing covered by browser/test_server.py)	2026-06-19 20:12:51 +03:00
lekss361	2e3291abc9	Merge pull request 'fix(scrapers): cian_detail — ceiling из offer.building.ceilingHeight, repair из decoration (#1791)' (#1800) from feat/cian-detail-field-mapping into main ... Reviewed-on: #1800	2026-06-19 17:07:14 +00:00
lekss361	95069a3daf	Merge pull request 'fix(scrapers): yandex_detail — area/ceiling/kitchen из INITIAL_STATE, покрытие area 63%→~95% (#1792)' (#1799) from feat/yandex-detail-field-coverage into main ... Reviewed-on: #1799	2026-06-19 17:07:02 +00:00
bot-backend	2d3ca78b0b	fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) ... ceiling_height на detail-странице лежит в offer.building.ceilingHeight, а не offer.ceilingHeight (всегда null) → было 0% покрытия. repairType часто пустой, структурное значение приходит как offer.decoration ('preFine'/...) — читаем оба + добавлен preFine->needs_repair и case-insensitive lookup в нормализаторе. kitchen/description пути уже верны (offer.kitchenArea/offer.description) — низкое покрытие = мёртвый CIAN_PROXY_URL, не маппинг. Регресс-тест на реальной карточке 330982715.	2026-06-19 19:57:18 +03:00
lekss361	7c9a3cf09d	Merge pull request 'fix(tradein-browser): camoufox hardening — block_images/webrtc + форс МСК tz (#1796)' (#1798) from fix/tradein-browser-camoufox-hardening into main ... Reviewed-on: #1798	2026-06-19 16:50:24 +00:00
bot-backend	b5f439a095	fix(tradein-browser): camoufox hardening — block_images/webrtc + forced MSK tz ... Со всеми знаниями из ресёрча camoufox (#1796 Avito anti-bot): - block_images=True — быстрее грузит (HTML/JSON, без рендера картинок). - block_webrtc=True — закрывает WebRTC-leak реального IP мимо прокси. - config timezone=Europe/Moscow — форсим +3 МСК: geoip+БэкКоннект (ротация IP на каждый запрос) давал геолик tz↔IP (Ульяновск UTC+4 при выходе МСК/СПб).	2026-06-19 19:49:14 +03:00
bot-backend	0084efc12a	fix(scrapers): yandex_detail — area/ceiling из структурного блока, покрытие area 63%->~95% (#1792)	2026-06-19 19:37:51 +03:00
lekss361	059fa83589	Merge pull request 'fix(scrapers): avito_houses — отзывы/история/тип ЖК долетают (виджет-payload вложен глубже) (#1789)' (#1797) from feat/avito-houses-mfe-state into main ... Reviewed-on: #1797	2026-06-19 16:27:03 +00:00
bot-backend	464e9faf7b	fix(scrapers): avito_houses читает MFE props-вложенность виджетов — отзывы/история/тип ЖК теперь долетают (#1789)	2026-06-19 19:21:49 +03:00
lekss361	4755f8bd13	Merge pull request 'feat(browser): per-provider concurrency — параллель между поставщиками, ≤1 коннект на поставщика (#1793)' (#1794) from feat/browser-per-provider-concurrency into main ... Reviewed-on: #1794	2026-06-19 16:02:13 +00:00
lekss361	fb7b3323e4	Merge pull request 'fix(tradein): verify=False для WFS геопортала ЕКБ (российский CA-серт)' (#1788) from fix/tradein-ekb-geoportal-ssl-verify into main ... Reviewed-on: #1788	2026-06-19 16:02:04 +00:00
bot-backend	1578dd5a5d	fixup(browser): source-fallback routing, lazy launch, test stability (#1793 review)	2026-06-19 19:00:10 +03:00
bot-backend	77ae4ca779	feat(browser): per-provider concurrency — параллель между поставщиками, ≤1 коннект на поставщика (#1793)	2026-06-19 18:52:55 +03:00
bot-backend	112e5cc2f1	fix(tradein): verify=False for EKB geoportal WFS (Russian-CA cert) ... The geoportal serves a Russian national-CA (Минцифры) TLS cert absent from httpx's default trust store → CERTIFICATE_VERIFY_FAILED, loader fetched 0. Public read-only data → verify=False (same as the curl -k used in recon).	2026-06-19 18:19:03 +03:00
lekss361	7494255180	Merge pull request 'fix(tradein): чинит main-red — патч _geoportal_house_match в cadastral-тесте (хвост #1786)' (#1787) from fix/tradein-geoportal-test-cadastral into main ... Reviewed-on: #1787	2026-06-19 15:07:31 +00:00
bot-backend	91b04a1851	test(tradein): patch _geoportal_house_match in cadastral-before-yandex test ... #1786 merged before the test fix landed → main red on test_geocode_uses_cadastral_before_yandex. The new geoportal house-match tier runs first in geocode(); this legacy cadastral test must mock it to None so it exercises the cadastral forward path (else the MagicMock db row leaks through the new tier as lat=1.0).	2026-06-19 18:06:28 +03:00
lekss361	7bf39e24df	Merge pull request 'fix(db): #1781 backfill listings.house_id_fk from house_sources (cian novostroyki)' (#1784) from fix/1781-cian-novostroyki-fk-backfill into main ... Reviewed-on: #1784	2026-06-19 14:56:09 +00:00
lekss361	84ceb9d300	Merge pull request 'feat(tradein): ингест зданий из геопортала ЕКБ + tier в геокодере (полный кадастр города)' (#1786) from feat/tradein-ekb-geoportal-buildings into main ... Reviewed-on: #1786	2026-06-19 14:55:43 +00:00
bot-backend	64d113d649	feat(tradein): EKB geoportal building ingest + geocoder tier (complete city buildings)	2026-06-19 17:51:07 +03:00
Light1YT	8adbbef88c	fix(db): #1781 backfill listings.house_id_fk from house_sources ... The realtime FK mirror (base.py:607-616) only landed 2026-06-17 (e63b21e). Listings matched into house_sources before that — notably 9205 cian novostroyki frozen at 2026-05-31 — never had house_id_fk mirrored back, breaking the canonical "same-building via house_id_fk" estimator path (93% of active cian novostroyki had NULL fk citywide). Add data/sql/130_backfill_listings_house_id_fk.sql: deterministic source-id backfill via house_sources (ext_source, ext_id → house_id), idempotent (WHERE house_id_fk IS NULL), no fuzzy/address fallback. Prod (read-only): 9235/9235 target rows resolve deterministically, zero unresolvable. One-time backlog drain; future rows linked in realtime by the e63b21e mirror.	2026-06-19 16:20:32 +05:00
lekss361	190accc4a7	Merge pull request 'feat(tradein): фронт передаёт координаты выбранной подсказки в /estimate (Variant A, frontend)' (#1783) from feat/tradein-estimate-send-coords into main ... Reviewed-on: #1783	2026-06-19 11:17:31 +00:00
lekss361	9701d3289d	Merge pull request 'feat(tradein): /estimate принимает координаты от автокомплита, минует geocode (Variant A, backend)' (#1782) from feat/tradein-estimate-accept-coords into main ... Reviewed-on: #1782	2026-06-19 11:16:51 +00:00
bot-backend	ce0aaa4fb1	feat(tradein): send picked autocomplete coords to /estimate (Variant A)	2026-06-19 14:16:08 +03:00
bot-backend	99e2a5d75c	feat(tradein): estimate accepts client-provided coords, skips geocode (Variant A)	2026-06-19 14:15:02 +03:00
lekss361	41b3f08303	Merge pull request 'fix(tradein): чинит кадастр-матчер геокодера — резолв адресов из своей базы (~60% упавших)' (#1780) from fix/tradein-cadastral-geocode-matcher into main ... Reviewed-on: #1780	2026-06-19 11:11:13 +00:00
bot-backend	c2a1dc8aeb	fix(tradein): parse street+house for cadastral geocode match (resolve ~60% of failing addresses from local DB)	2026-06-19 14:04:12 +03:00
lekss361	be3a5df8eb	Merge pull request 'fix(tradein): удалить white-label бренд «Практика» целиком (БД + маппинг + ассеты)' (#1779) from fix/tradein-remove-praktika-brand into main ... Reviewed-on: #1779	2026-06-19 10:09:38 +00:00
bot-backend	ea453c3ff2	fix(tradein): remove «Практика» white-label brand (DB row + account mapping + assets)	2026-06-19 12:53:37 +03:00
lekss361	374238daac	Merge pull request 'fix(tradein): budget Cian valuation call to stop 25s estimate stalls («не считает»)' (#1777) from fix/tradein-estimate-latency-budgets into main ... Reviewed-on: #1777	2026-06-19 09:48:41 +00:00
lekss361	54444cc4a5	Merge pull request 'fix(tradein): убрать хардкод «Практика» из offer-баннера' (#1778) from fix/tradein-offercard-praktika-leak into main ... Reviewed-on: #1778	2026-06-19 09:48:35 +00:00
bot-backend	80380dc32a	fix(tradein): budget Cian valuation call to stop 25s estimate stalls ... The Cian Valuation enrichment was the only ungated external call in estimate_quality() — its internal curl timeout (~25s) blocks the whole /estimate request on every cache-miss, intermittently pushing latency past the gateway/patience window (perceived as 'сервис не считает'). Wrap it in _with_budget() (mirrors the existing Yandex-valuation guard): on timeout it degrades to None — the same graceful path as a network error — instead of stalling. New setting estimate_cian_valuation_timeout_s (default 8.0s, env ESTIMATE_CIAN_VALUATION_TIMEOUT_S).	2026-06-19 12:43:28 +03:00
bot-backend	c5d469da36	fix(tradein): remove hardcoded «Практика» from offer banner + comments ... Visible warn-banner trailing sentence no longer names the brand. Code comments in the file docstring updated to be brand-neutral. No logic or behaviour changed.	2026-06-19 12:41:27 +03:00
lekss361	ee2703419b	Merge pull request 'fix(estimator): #1774 Tier A admits same-building novostroyki resales (secondary-dominant gate)' (#1776) from fix/1774-tier-a-same-building-primary into main ... Reviewed-on: #1776	2026-06-19 09:28:16 +00:00
lekss361	19f740f2e9	Merge pull request 'fix(avito): de-glue house number from district in _clean_address (#1773 part)' (#1775) from fix/1773-avito-address-deglue into main ... Reviewed-on: #1775	2026-06-19 09:26:26 +00:00
Light1YT	07d9cc8e8b	fix(estimator): #1774 Tier A admits same-building novostroyki resales when secondary-dominant ... Tier A (same building) dropped cian listings tagged listing_segment='novostroyki' even in delivered buildings where that tag marks owner resales/переуступки (sale_type=free). Gated relaxation of the #1186 guard, Tier A ONLY: include novostroyki rows iff the same house has secondary (vtorichka/NULL) listings AND secondary_count >= primary_count — secondary-dominant signal of a delivered house. Prevents a lone vtorichka + clustered developer-priced novostroyki from anchoring on застройщик prices after MAD-clip drops the rare secondary (~27% overvaluation). Pure-primary / primary-dominated houses keep the #1186 guard. Behind estimate_sb_tier_a_allow_primary_if_secondary_present (default True). Dedup duplicate active rows by (source, source_id) primary key (codebase canon, namespaced id/url tags, source_url fallback) — collapses rows whose source_url differs only by trailing slash / query-param (e.g. cian source_id=330047129 ×2). Tier C / radius / asking_to_sold_ratio paths untouched. 84 unit tests pass.	2026-06-19 14:00:13 +05:00
Light1YT	78e7705a5f	fix(avito): de-glue house number from district marker in _clean_address (#1773) ... Avito leaks addresses with house number glued to district marker without separator («…Савкова, 29р-н Академический»), breaking house-number extraction and same-house matching (41 unmatched avito listings on one street). _clean_address now inserts ", " between a house-number token (digits + optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following marker (р-н, мкр, район, г., пос, снт, …). - avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept) - tests: de-glue positive/negative/idempotency (24 passed) - data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows normalize.py НЕ тронут (общий fingerprint матчинга, #1534). Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).	2026-06-19 13:33:43 +05:00
lekss361	cf6dc59cb6	Merge pull request 'fix(avito): reconnect-retry on detail 429 under backconnect proxy' (#1771) from fix/avito-detail-429-reconnect into main ... Reviewed-on: #1771	2026-06-19 04:42:11 +00:00
bot-backend	d65a88f704	fix(avito): reconnect-retry on detail 429 under backconnect proxy ... Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched because the detail fetcher only had a same-session 429 short-retry (no reconnect). 429s are persistent: keep-alive reuses one backconnect exit IP across sequential detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769): widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP) on 429 exhaustion before raising. Caller session never touched (as with 403 path).	2026-06-18 23:29:47 +03:00
lekss361	59728a197d	Merge pull request 'feat(avito): incremental daily + exhaustive weekly schedule split (+ scheduler interval_days)' (#1770) from feat/avito-incremental-schedule-split into main ... Reviewed-on: #1770	2026-06-18 20:25:39 +00:00
lekss361	0fd1d63ab3	Merge pull request 'fix(avito): tolerate 429 per-bucket instead of banning whole sweep' (#1769) from fix/avito-429-bucket-skip into main ... Reviewed-on: #1769	2026-06-18 20:25:26 +00:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00