136 commits

Author	SHA1	Message	Date
lekss361	0793571c4a	fix(tradein/deploy): миграции ДО подъёма app-контейнеров + manifest-инвариант (#2216) (#2228)	2026-07-02 21:03:37 +00:00
lekss361	ac19c4f7b3	feat(tradein/monitor): алерт на staleness данных deals по max(deal_date) (#2212) (#2226)	2026-07-02 20:50:37 +00:00
lekss361	47f45fc3d2	fix(tradein/scrapers): bump scraped_at при ре-подтверждении живым + бэкфилл (#2206) (#2224)	2026-07-02 20:30:01 +00:00
lekss361	f5b0076e6f	fix(tradein): deactivate_stale для domklik/n1 + честная freshness по scraped_at (#2204) (#2221)	2026-07-02 20:10:34 +00:00
bot-backend	67a852caf1	feat(tradein/matching): fias_exact tier + estimate fias persist + suggest fias_id contract	2026-07-02 20:49:04 +03:00
lekss361	d7f4873a53	feat(proxy-pool): pool service (acquire/release/health) + healthcheck scheduler task (#2162)	2026-07-02 16:41:39 +00:00
lekss361	90e4a741f7	feat(proxy-pool): scrape_proxies table + admin bulk-load/list endpoints (#2161)	2026-07-02 16:31:21 +00:00
lekss361	abef3e6e02	feat(market): revoke gendesign_reader direct SELECT on raw tables (#2138 step 3/3)	2026-07-02 15:11:55 +00:00
lekss361	03b3cb9fe4	feat(market): grant gendesign_reader SELECT on contract views (#2138 step 1/3)	2026-07-02 14:47:49 +00:00
lekss361	e2bb5061f9	feat(market): contract views market.v_* over producer tables (#2130)	2026-07-02 14:02:59 +00:00
bot-backend	078dadc440	fix(tradein): cadastre ЖКХ-fallback + COALESCE merge-key + median-floors guard ... Серия sale-share data-quality (по находкам юзера): - house_dedup_merge: cluster_key → tradein_canon_addr(COALESCE(short_address, full_address, address)) — ловит дубли с район-noise в address (Вайнера 66). - zhkh_flats_loader: cadastre-fallback (searchByAddress по cadastreNumber, guid-независимо) — резолвит дома где gar_house_guid ≠ ЖКХ houseGuid (Переходный 6: guid→0, cadastre→165). CLI --cadastre. - мигр.153: плаузибилити-гард по медиане этажности листингов — denom >= GREATEST(total_floors, median(listings.total_floors), 8); отсекает gar-недосчёт башен без ЖКХ (gar=9 vs листинги 19-26 эт → NULL). ruff clean, pytest 36 green.	2026-06-29 01:32:48 +03:00
bot-backend	ebc3813648	fix(tradein/sale-share): гард по этажности — отсечь near-neighbor mis-match ... Гео-фильтр ≤300м (мигр.150) широк для плотной застройки: к 5-этажному дому (Мраморская 34к4) примагничены чужие листинги 9/16/19-этажных башен (Благодатская 15, Мраморская 6) на 229-282м — все <300м, фильтр не отсекал. - мигр.152: +houses.zhkh_floors (parse zhkh_floors_raw "10-30"→max); гард во view во всех 5 FILTER: считать листинг только если abs(l.total_floors - COALESCE(zhkh_floors, total_floors)) <= 3 (или этажность неизвестна). Гео-фильтр 300м сохранён. - buildings_query.build_listings_query: тот же гард в детальной выдаче. Прод-замер: отсекает 2754 листинга (12%) height-mismatched. Покрытие этажности 75% total_floors / 73% ЖКХ. ruff clean, pytest 26 green.	2026-06-29 00:31:09 +03:00
bot-backend	b105e0a500	fix(tradein/sale-share): дедуп карточки листингов + чистка голых алиасов ... #4 Карточка дома показывала cross-площадочные дубли (Искровцев 33: 2 квартиры = 4 строки avito/domklik/cian). build_listings_query теперь дедуплит по сигнатуре (rooms, round(area_m2), floor) — как v_building_sale_share мигр.148 — одна строка на квартиру, представитель = самый дешёвый, +колонка sources[] (площадки). Гео-фильтр ≤300м (мигр.150) сохранён. +BuildingListing.sources в схеме. P4a Мигр.151: DELETE 6338/21720 голых алиасов house_address_aliases (без номера дома) — сток, в который сваливались безномерные листинги (Tier-2b). Матчер уже захардненен (P1 #2082), голые не регенерируют. Алиасы — регенерируемый кэш, house_id_fk/house_sources не тронуты. ruff clean, pytest test_buildings_api 26 green.	2026-06-29 00:22:20 +03:00
bot-backend	f617c323bd	fix(tradein/sale-share): гео-фильтр числителя ≤300м — отсечь mis-bucketed листинги ... Листинги примагничиваются к НЕ ТОМУ дому (кривой house_id_fk): прод-замер 373 дома (5.5%) / 1685 листингов (7.2%) дальше 300м от своего дома. Пример — house 131237 «Мамина-Сибиряка 126» (ЖКХ=51 кв, знаменатель ВЕРНЫЙ): из 20 активных листингов реально его ~2, остальные — Свердлова 32 / Азина 31 (до 2.2км) → ложная доля 39% вместо ~4%. - мигр.150: v_building_sale_share — числители (active/45д) + медианы считают только листинги ≤300м от geom дома (ST_DistanceSphere; листинг/дом без geom — keep). Знаменатель не тронут. - buildings_query.build_listings_query: тот же гео-фильтр в детальной выдаче (карточка дома больше не показывает чужие листинги). Симуляция на проде: меняется 351 дом, обнуляется 164 (чистые «мусорные вёдра»), числитель −8.3%. Мамина-126: 20→2. Это митигация на уровне view; корневой фикс house_id_fk-матчера — отдельно. ruff clean, pytest 24 green.	2026-06-28 23:24:05 +03:00
bot-backend	77c01b6d3b	feat(tradein/zhkh): ЖКХ-приоритет знаменателя «квартир в доме» + loader ... ГАР дико недосчитывает квартиры в МКД (прод-замер: Крауля 89А ГАР=11 vs ЖКХ=429; 27 из 347 домов ГАР ≥2× недосчёт). ГИС ЖКХ точнее — резолвит ~90% вторички с gar_house_guid. - мигр.149: COALESCE знаменателя → ЖКХ ПЕРВЫЙ (zhkh_flat_count, gar_flat_count, total_units, flat_count) во всех 7 местах view v_building_sale_share; +houses.zhkh_year/zhkh_floors_raw; +transparency- колонки zhkh_flat_count/flat_count_source. - app/services/zhkh_flats_loader.py + app/tasks/zhkh_flats_load.py: фетч ГИС ЖКХ searchByAddress (батч ≤100 по gar_house_guid == ЖКХ houseGuid, pageIndex=1, cookieless), идемпотентный UPDATE houses.zhkh_* (CAST(:x AS type) + IS DISTINCT FROM gate, SAVEPOINT-чанки, dry-run). Безопасно к мержу: до ручного прогона loader'а zhkh_flat_count=NULL → COALESCE падает на ГАР → поведение идентично до-149. ruff clean, pytest 8 green.	2026-06-28 21:55:05 +03:00
lekss361	0641183ecd	fix(tradein): дедуп объявлений по квартире + over_100 NULL-фикс (500 на списке) (148) (#2075)	2026-06-28 18:04:32 +00:00
lekss361	b45a819cb6	fix(tradein): канон срезает гео-префиксы (Россия/Екатеринбург/мкр/пр-т) + REINDEX — покрытие 38%→~70% (147) (#2073)	2026-06-28 17:54:26 +00:00
lekss361	2a38561175	feat(tradein): второй % «за 45 дней» + zhkh-знаменатель slot (146) (#2071)	2026-06-28 17:51:38 +00:00
lekss361	57ae2dfe00	fix(tradein): guard ГАР-undercount знаменателя в v_building_sale_share (145) (#2064)	2026-06-28 16:05:33 +00:00
lekss361	579536b4ad	feat(tradein): канонический матч ГАР↔houses (0%→41% yield) + tradein_canon_addr + --match-only (#2063)	2026-06-28 15:50:07 +00:00
lekss361	9e2d9ddbea	feat(tradein): схема «доли квартир дома в продаже» — gar_house_flats + houses.gar_* + v_building_sale_share (143) (#2053)	2026-06-28 13:48:41 +00:00
lekss361	e39fb86a9b	feat(tradein): AI-curated premium-building overlay — class + false-positive demotion (#2002)	2026-06-27 18:43:05 +00:00
lekss361	c5227a164e	feat(tradein/cian): promote kitchen/mortgage/apartments + normalize house_type (#2008) ... cian house_type normalized via shared house_type_normalizer (monolithBrick->monolith_brick etc.) — fixes estimator soft-penalty false-mismatch (~1208 rows); backfill migration 141 + new columns is_apartments/is_rosreestr_checked. kitchen/mortgage/apartments promoted (dashboard/matching, not valuation — #2012). Refs #2008	2026-06-27 18:01:50 +00:00
lekss361	440bb2ca51	feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007) ... house_type normalized via shared house_type_normalizer (canon = _IMV_HOUSE_TYPE_MAP) — fixes estimator soft-penalty false-mismatch on ~70% yandex analogs; backfill migration 140 (SCREAMING->canon + 'other'->NULL). kitchen/ceiling promoted to columns (dashboard/matching, not valuation — see #2012); ceiling capped 2.0-6.0m to avoid numeric(3,2) overflow. Refs #2007	2026-06-27 17:45:11 +00:00
lekss361	840d64a4a0	feat(tradein): premium_houses MV + premium_building flag (#2002)	2026-06-27 16:40:39 +00:00
bot-backend	52fc920507	fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status (#1846, #1968) ... Old HTML scraper dead: /search?city_id retired in DomClick redesign + VPS IP QRATOR-banned (prod domclick_city_sweep reported done/0 every run). New path: - GET bff-search-web.domclick.ru/api/offers/v1 + count/v1 via BrowserFetcher(source="domclick") -> generic provider -> shared mobile proxy -> bypasses QRATOR (prod-verified live: 20 items returned, no block). - Sweeps 6 room buckets (st/1/2/3/4/5+); recursive binary price-split for oversized buckets (rooms=2 ~2215 > offset cap 2000); count/v1 sizes buckets. - Maps SERP JSON -> ScrapedLot with lat/lon at ~100% (closes avito 54% NULL-lat gap); per-card EKB geo-guard. - Honest status (#1968): QRATOR block OR fetch errors with 0 lots -> mark_failed, not mark_done. Parse failures are per-page resilient (break bucket, keep partial lots) so a transient bad page never aborts the whole sweep. - Schedule shipped DORMANT (enabled=false, pages=100) via 138_domclick_bff_rewrite_schedule.sql; operator enables after prod smoke. 42 scraper tests (price-split boundaries, parse-error resilience, honest status); full backend suite green; ruff clean. Layer B (detail renovation/wallType/ priceHistory) deferred to #1846 follow-up.	2026-06-27 13:57:07 +03:00
Light1YT	8d5c1a888d	perf(estimate): trigram expression index for same-building anchor Tier A (~94x) ... _fetch_anchor_comps Tier A runs on EVERY /estimate (flag estimate_same_building_anchor_enabled defaults True, not overridden in prod). Its address predicate wraps address in lower(translate(address,'ёЁ','ее')) for both the LIKE '%street%' and the ~house-number regex — non-sargable, and the existing listings_address_trgm_idx is on the RAW address, so the planner seq-scanned all ~66.7k listings (167ms, 15801 buffers) every estimate. Add a GIN trigram expression index on lower(translate(address,'ёЁ','ее')) matching the predicate exactly (gin_trgm_ops serves both LIKE and the regex). Prod EXPLAIN (BEGIN/ROLLBACK, "Хохрякова 48"): 167ms -> 1.77ms (~94x), buffers 15801 -> 97. Independently dry-run-verified: Bitmap Index Scan, 0.66ms. Result-identical (pure access-path; count/sum/min/max unchanged). No code change (expression matches the predicate). ~8MB, sub-second build. Other estimate-path queries confirmed already optimal (Tier W/H geom+rooms BitmapAnd, Tier S house_id_fk) — no other index warranted. tradein-mvp/backend/data/sql/137 (correct tradein migrations path).	2026-06-27 11:16:48 +05:00
Light1YT	6ce60a0d8c	fix(tradein): backfill house_id_fk via source-identity path (#1781) ... Migration 130 only filled house_id_fk for listings carrying house_source/house_ext_id. Prod (2026-06-27): 1884 active NULL-FK rows remain, 0 with house_source/ext_id (130 re-run = no-op). They were matched via the per-listing-identity path the realtime mirror uses when no house catalog id exists: match_or_create_house(ext_source=source, ext_id=source_id) -> Tier-1 house_sources lookup (confidence 1.0). 136 replays that exact lookup offline: UPDATE listings SET house_id_fk = house_sources.house_id WHERE (ext_source=source, ext_id=source_id) and house_id_fk IS NULL. Faithful to base.py:704-750; idempotent (NULL guard); deterministic (no fuzzy/geo); collision-safe (catalog ids live under distinct ext_source like 'cian_newbuilding'). Dry-run BEGIN/ROLLBACK fills 4608 NULL-FK rows (744 active), restoring estimator Tier-S "same-building" grouping. The remaining ~1140 active need re-scrape/ re-match (scraper coverage — #1781 Лёха). Refs #1781	2026-06-27 05:01:02 +05:00
bot-backend	63ac12dc23	feat(tradein): recurring house-dedup merge (schedule dormant) (#1772) (#1933)	2026-06-26 21:39:30 +00:00
bot-backend	d89210200b	perf(tradein): GIST index on (geom::geography) for estimator radius filter (#1189) (#1924)	2026-06-26 17:12:49 +00:00
bot-backend	e0cd378d19	fix(migration-133): dedup via cascade-delete, not child re-point (prod hotfix) ... Migration 133 failed on prod (offer_price_history UNIQUE (listing_id,change_time) + multi-loser collisions). Drop data-child re-points, rely on FK ON DELETE CASCADE; keep only merged_into. Dry-run on prod: 9840 groups, 9889 losers deleted, clean. Refs #1773	2026-06-26 08:16:27 +00:00
bot-backend	1d5b081cb4	fix(cian): recover new-build house via cadastral anchor + listings (source,source_id) dedup (#1773) (#1912) ... cian new-build cadastral anchor + listings (source,source_id) dedup (#1773): migration 133 (deep-reviewed data-loss-safe). Refs #1773	2026-06-26 07:45:22 +00:00
bot-backend	3f1925052b	feat(scheduler): wire house_imv_backfill as scheduled source (#854) (#1910) ... wire house_imv_backfill as scheduled source (#854): trigger fn + seed migration 132. Refs #854	2026-06-26 07:45:08 +00:00
bot-backend	89cbddf318	fix(tradein/db): widen offer_price_history.diff_percent numeric(5,2)→(8,2) + clamp в триггере (краш full_load)	2026-06-20 11:43:23 +03:00
lekss361	7bf39e24df	Merge pull request 'fix(db): #1781 backfill listings.house_id_fk from house_sources (cian novostroyki)' (#1784) from fix/1781-cian-novostroyki-fk-backfill into main ... Reviewed-on: #1784	2026-06-19 14:56:09 +00:00
bot-backend	64d113d649	feat(tradein): EKB geoportal building ingest + geocoder tier (complete city buildings)	2026-06-19 17:51:07 +03:00
Light1YT	8adbbef88c	fix(db): #1781 backfill listings.house_id_fk from house_sources ... The realtime FK mirror (base.py:607-616) only landed 2026-06-17 (e63b21e). Listings matched into house_sources before that — notably 9205 cian novostroyki frozen at 2026-05-31 — never had house_id_fk mirrored back, breaking the canonical "same-building via house_id_fk" estimator path (93% of active cian novostroyki had NULL fk citywide). Add data/sql/130_backfill_listings_house_id_fk.sql: deterministic source-id backfill via house_sources (ext_source, ext_id → house_id), idempotent (WHERE house_id_fk IS NULL), no fuzzy/address fallback. Prod (read-only): 9235/9235 target rows resolve deterministically, zero unresolvable. One-time backlog drain; future rows linked in realtime by the e63b21e mirror.	2026-06-19 16:20:32 +05:00
bot-backend	ea453c3ff2	fix(tradein): remove «Практика» white-label brand (DB row + account mapping + assets)	2026-06-19 12:53:37 +03:00
lekss361	19f740f2e9	Merge pull request 'fix(avito): de-glue house number from district in _clean_address (#1773 part)' (#1775) from fix/1773-avito-address-deglue into main ... Reviewed-on: #1775	2026-06-19 09:26:26 +00:00
Light1YT	78e7705a5f	fix(avito): de-glue house number from district marker in _clean_address (#1773) ... Avito leaks addresses with house number glued to district marker without separator («…Савкова, 29р-н Академический»), breaking house-number extraction and same-house matching (41 unmatched avito listings on one street). _clean_address now inserts ", " between a house-number token (digits + optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following marker (р-н, мкр, район, г., пос, снт, …). - avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept) - tests: de-glue positive/negative/idempotency (24 passed) - data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows normalize.py НЕ тронут (общий fingerprint матчинга, #1534). Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).	2026-06-19 13:33:43 +05:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00
bot-backend	8436532bf4	feat(listings): card_hash column + snapshot-dedup on unchanged cards ... Adds listings.card_hash (SHA-256 of volatile SERP-card fields) computed and stored on every scrape. Skips the redundant per-day listings_snapshots write when a listing's card is unchanged. listings.last_seen_at/is_active still bumped on every sighting (delisting TTL). Lays groundwork for detail-refresh gating (C2).	2026-06-18 22:01:54 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
bot-backend	1fe670ea6a	feat(yandex): map rich entity fields (predictedPrice, seller, listing_date, trend, metro)	2026-06-17 22:58:16 +03:00