1486 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	64d113d649	feat(tradein): EKB geoportal building ingest + geocoder tier (complete city buildings)	2026-06-19 17:51:07 +03:00
lekss361	190accc4a7	Merge pull request 'feat(tradein): фронт передаёт координаты выбранной подсказки в /estimate (Variant A, frontend)' (#1783) from feat/tradein-estimate-send-coords into main ... Reviewed-on: #1783	2026-06-19 11:17:31 +00:00
lekss361	9701d3289d	Merge pull request 'feat(tradein): /estimate принимает координаты от автокомплита, минует geocode (Variant A, backend)' (#1782) from feat/tradein-estimate-accept-coords into main ... Reviewed-on: #1782	2026-06-19 11:16:51 +00:00
bot-backend	ce0aaa4fb1	feat(tradein): send picked autocomplete coords to /estimate (Variant A)	2026-06-19 14:16:08 +03:00
bot-backend	99e2a5d75c	feat(tradein): estimate accepts client-provided coords, skips geocode (Variant A)	2026-06-19 14:15:02 +03:00
lekss361	41b3f08303	Merge pull request 'fix(tradein): чинит кадастр-матчер геокодера — резолв адресов из своей базы (~60% упавших)' (#1780) from fix/tradein-cadastral-geocode-matcher into main ... Reviewed-on: #1780	2026-06-19 11:11:13 +00:00
bot-backend	c2a1dc8aeb	fix(tradein): parse street+house for cadastral geocode match (resolve ~60% of failing addresses from local DB)	2026-06-19 14:04:12 +03:00
lekss361	be3a5df8eb	Merge pull request 'fix(tradein): удалить white-label бренд «Практика» целиком (БД + маппинг + ассеты)' (#1779) from fix/tradein-remove-praktika-brand into main ... Reviewed-on: #1779	2026-06-19 10:09:38 +00:00
bot-backend	ea453c3ff2	fix(tradein): remove «Практика» white-label brand (DB row + account mapping + assets)	2026-06-19 12:53:37 +03:00
lekss361	374238daac	Merge pull request 'fix(tradein): budget Cian valuation call to stop 25s estimate stalls («не считает»)' (#1777) from fix/tradein-estimate-latency-budgets into main ... Reviewed-on: #1777	2026-06-19 09:48:41 +00:00
lekss361	54444cc4a5	Merge pull request 'fix(tradein): убрать хардкод «Практика» из offer-баннера' (#1778) from fix/tradein-offercard-praktika-leak into main ... Reviewed-on: #1778	2026-06-19 09:48:35 +00:00
bot-backend	80380dc32a	fix(tradein): budget Cian valuation call to stop 25s estimate stalls ... The Cian Valuation enrichment was the only ungated external call in estimate_quality() — its internal curl timeout (~25s) blocks the whole /estimate request on every cache-miss, intermittently pushing latency past the gateway/patience window (perceived as 'сервис не считает'). Wrap it in _with_budget() (mirrors the existing Yandex-valuation guard): on timeout it degrades to None — the same graceful path as a network error — instead of stalling. New setting estimate_cian_valuation_timeout_s (default 8.0s, env ESTIMATE_CIAN_VALUATION_TIMEOUT_S).	2026-06-19 12:43:28 +03:00
bot-backend	c5d469da36	fix(tradein): remove hardcoded «Практика» from offer banner + comments ... Visible warn-banner trailing sentence no longer names the brand. Code comments in the file docstring updated to be brand-neutral. No logic or behaviour changed.	2026-06-19 12:41:27 +03:00
lekss361	ee2703419b	Merge pull request 'fix(estimator): #1774 Tier A admits same-building novostroyki resales (secondary-dominant gate)' (#1776) from fix/1774-tier-a-same-building-primary into main ... Reviewed-on: #1776	2026-06-19 09:28:16 +00:00
lekss361	19f740f2e9	Merge pull request 'fix(avito): de-glue house number from district in _clean_address (#1773 part)' (#1775) from fix/1773-avito-address-deglue into main ... Reviewed-on: #1775	2026-06-19 09:26:26 +00:00
Light1YT	07d9cc8e8b	fix(estimator): #1774 Tier A admits same-building novostroyki resales when secondary-dominant ... Tier A (same building) dropped cian listings tagged listing_segment='novostroyki' even in delivered buildings where that tag marks owner resales/переуступки (sale_type=free). Gated relaxation of the #1186 guard, Tier A ONLY: include novostroyki rows iff the same house has secondary (vtorichka/NULL) listings AND secondary_count >= primary_count — secondary-dominant signal of a delivered house. Prevents a lone vtorichka + clustered developer-priced novostroyki from anchoring on застройщик prices after MAD-clip drops the rare secondary (~27% overvaluation). Pure-primary / primary-dominated houses keep the #1186 guard. Behind estimate_sb_tier_a_allow_primary_if_secondary_present (default True). Dedup duplicate active rows by (source, source_id) primary key (codebase canon, namespaced id/url tags, source_url fallback) — collapses rows whose source_url differs only by trailing slash / query-param (e.g. cian source_id=330047129 ×2). Tier C / radius / asking_to_sold_ratio paths untouched. 84 unit tests pass.	2026-06-19 14:00:13 +05:00
Light1YT	78e7705a5f	fix(avito): de-glue house number from district marker in _clean_address (#1773) ... Avito leaks addresses with house number glued to district marker without separator («…Савкова, 29р-н Академический»), breaking house-number extraction and same-house matching (41 unmatched avito listings on one street). _clean_address now inserts ", " between a house-number token (digits + optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following marker (р-н, мкр, район, г., пос, снт, …). - avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept) - tests: de-glue positive/negative/idempotency (24 passed) - data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows normalize.py НЕ тронут (общий fingerprint матчинга, #1534). Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).	2026-06-19 13:33:43 +05:00
lekss361	cf6dc59cb6	Merge pull request 'fix(avito): reconnect-retry on detail 429 under backconnect proxy' (#1771) from fix/avito-detail-429-reconnect into main ... Reviewed-on: #1771	2026-06-19 04:42:11 +00:00
bot-backend	d65a88f704	fix(avito): reconnect-retry on detail 429 under backconnect proxy ... Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched because the detail fetcher only had a same-session 429 short-retry (no reconnect). 429s are persistent: keep-alive reuses one backconnect exit IP across sequential detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769): widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP) on 429 exhaustion before raising. Caller session never touched (as with 403 path).	2026-06-18 23:29:47 +03:00
lekss361	59728a197d	Merge pull request 'feat(avito): incremental daily + exhaustive weekly schedule split (+ scheduler interval_days)' (#1770) from feat/avito-incremental-schedule-split into main ... Reviewed-on: #1770	2026-06-18 20:25:39 +00:00
lekss361	0fd1d63ab3	Merge pull request 'fix(avito): tolerate 429 per-bucket instead of banning whole sweep' (#1769) from fix/avito-429-bucket-skip into main ... Reviewed-on: #1769	2026-06-18 20:25:26 +00:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00
bot-backend	bee44ffd3d	fix(avito): tolerate 429 per-bucket instead of banning whole sweep ... Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36 (deep pagination, backconnect conn-limit). Widen the 429 retry budget + one reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip a blocked bucket and continue rather than aborting the run; only N consecutive blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.	2026-06-18 23:03:28 +03:00
lekss361	b7d43db6da	Merge pull request 'fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy' (#1768) from fix/avito-detail-403-reconnect-retry into main ... Reviewed-on: #1768	2026-06-18 19:14:39 +00:00
bot-backend	c993243bc7	fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy ... fetch_detail raised on the first 403/429, aborting the detail backfill (run 191 enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active. Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy, rebuild an ephemeral session (fresh exit IP) and retry up to N before raising; short-retry 429. Never close/rebuild the caller-provided shared session. Also add the missing HTTP-200 firewall-interstitial check on the curl detail path.	2026-06-18 22:12:03 +03:00
lekss361	bf2b1fca79	Merge pull request 'feat(avito): incremental SERP pagination with date early-stop (param-gated, default off)' (#1766) from feat/avito-incremental-pagination into main ... Reviewed-on: #1766	2026-06-18 19:03:37 +00:00
lekss361	73c4a90946	Merge pull request 'feat(listings): card_hash column + snapshot-dedup on unchanged cards' (#1767) from feat/listings-card-hash into main ... Reviewed-on: #1767	2026-06-18 19:03:28 +00:00
bot-backend	8436532bf4	feat(listings): card_hash column + snapshot-dedup on unchanged cards ... Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and stored on every scrape. Skips the redundant per-day listings_snapshots write when a listing's card is unchanged. listings.last_seen_at/is_active still bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh gating (C2).	2026-06-18 22:01:54 +03:00
bot-backend	bdedac3012	feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) ... Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate sequentially newest-first and stop once cards fall below the watermark, skipping deep pagination (and the 403 clusters it triggers). since=None keeps the exact exhaustive bisection walk. No scheduler/schedule change here — wiring follows in a separate PR.	2026-06-18 22:01:46 +03:00
lekss361	c58c5a5aab	Merge pull request 'fix(avito): gate 403 reconnect-retry on proxy presence only (was dead code in prod)' (#1765) from fix/avito-403-reconnect-gate into main ... Reviewed-on: #1765	2026-06-18 18:32:54 +00:00
bot-backend	44e316c514	fix(avito): gate 403 reconnect-retry on proxy presence only, not rotate_url absence ... PR #1764's reconnect-retry was dead code in prod: the gate excluded configs with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL (what curl_cffi egresses through) AND a leftover changeip URL for the separate auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403 still aborted the sweep (run 212). The curl session always uses scraper_proxy_url, so session-rebuild rotates the exit IP regardless of changeip config.	2026-06-18 21:31:20 +03:00
lekss361	69ef61beb8	Merge pull request 'fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy' (#1764) from fix/avito-403-backconnect-reconnect-retry into main ... Reviewed-on: #1764	2026-06-18 18:13:20 +00:00
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
lekss361	8d516d0781	Merge pull request 'fix(tradein): avito _extract_total_count returns 0 on no-results page — stop empty-bracket split/rotate waste' (#1763) from fix/avito-empty-page-total-zero into main ... Reviewed-on: #1763	2026-06-18 17:05:34 +00:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
lekss361	f82ee32f50	Merge pull request 'feat(tradein): avito curl_cffi 429 short-retry — enables backconnect curl SERP (×50 faster than browser)' (#1762) from feat/avito-curl-429-retry into main ... Reviewed-on: #1762	2026-06-18 16:26:28 +00:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
lekss361	8a7538ab13	Merge pull request 'perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)' (#1761) from perf/camoufox-block-resources into main ... Reviewed-on: #1761	2026-06-18 15:18:28 +00:00
bot-backend	1533baad97	perf(browser): block image/media/font resources in camoufox fetch (~10x smaller pages, faster via proxy)	2026-06-18 18:16:53 +03:00
lekss361	e05e81f654	Merge pull request 'perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)' (#1760) from perf/shared-price-brackets-cian-yandex into main ... Reviewed-on: #1760	2026-06-18 14:57:41 +00:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
lekss361	0538834c6b	Merge pull request 'perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)' (#1759) from perf/avito-seed-price-brackets into main ... Reviewed-on: #1759	2026-06-18 14:33:07 +00:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
lekss361	c8f6fd1535	Merge pull request 'fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)' (#1758) from fix/praktika-brand-only-praktika-user into main ... Reviewed-on: #1758	2026-06-18 13:36:20 +00:00
bot-backend	5c37fcbfad	fix(tradein): praktika brand only under praktika account — drop kopylov mapping (brand leak)	2026-06-18 16:29:18 +03:00
lekss361	add002e8d7	Merge pull request 'fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run' (#1757) from fix/avito-fullload-probe-resilience into main ... Reviewed-on: #1757	2026-06-18 13:11:46 +00:00
lekss361	5b14d88290	Merge pull request 'feat(site-finder): snapshot.pdf exclude_dev param — hide competitor by developer name' (#1756) from feat/snapshot-exclude-dev into main ... Reviewed-on: #1756	2026-06-18 13:11:29 +00:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
bot-backend	5408466905	chore(frontend): regen api-types for snapshot exclude_dev param	2026-06-18 15:51:56 +03:00