36 commits

Author	SHA1	Message	Date
bot-backend	63ac12dc23	feat(tradein): recurring house-dedup merge (schedule dormant) (#1772) (#1933)	2026-06-26 21:39:30 +00:00
bot-backend	3f1925052b	feat(scheduler): wire house_imv_backfill as scheduled source (#854) (#1910) ... wire house_imv_backfill as scheduled source (#854): trigger fn + seed migration 132. Refs #854	2026-06-26 07:45:08 +00:00
bot-backend	1f7018d415	feat(scheduler): per-source interval_days cadence (weekly exhaustive avito) ... scrape_schedules had no interval column -> every source ran daily. Add interval_days from default_params (default 1, backward-compatible) so compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive to interval_days=7 (weekly full pass for last_seen/delisting + silent price edits); avito_full_load stays daily incremental.	2026-06-18 23:10:18 +03:00
bot-backend	bf63683833	feat(avito): incremental daily + exhaustive weekly schedule split ... run_avito_full_load gains incremental_days param -> passes since= to the (merged) incremental SERP engine. avito_full_load schedule flipped to incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429 bans). New avito_full_load_exhaustive source runs the full walk weekly to refresh last_seen (10-day delisting TTL) and catch silent price edits.	2026-06-18 23:04:35 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	158a24c25b	feat(tradein): schedule cian_full_load (exhaustive secondary) + anchor sweep NB-only ... Подключает prod-проверенный run_cian_full_load (exhaustive региональный сбор Cian ЕКБ вторички, room×price партиционирование, incremental on_bucket save) в in-app scheduler как recurring-источник cian_full_load (окно 20-22 UTC — отделено от cian_city_sweep/history_backfill на 2-5 чтобы не конкурировать на shared kf-прокси). run_cian_city_sweep теперь NB-only (newbuilding_only=True default): SERP-фаза сохраняет только novostroyki-лоты, вторичку отбрасывает — ею авторитетно владеет full_load. Убирает дубль secondary SERP-сейва. DETAIL/HOUSES-фазы не затронуты.	2026-06-18 13:25:23 +03:00
lekss361	48bedb5f65	Merge branch 'main' into feat/cadastral-geo-match	2026-06-18 08:44:06 +00:00
bot-backend	edd187b47f	feat(cadastral): geo-nearest building matcher via local cad mirror ... Backfill listings.building_cadastral_number (was 0%) from the nearest cadastral building. gendesign_cad_buildings is a postgres_fdw foreign table with no geom — a per-listing FDW nearest query is ~1.16s/row (~13h for 43k listings). Instead materialize the FDW once into a LOCAL cad_buildings_local table (Point geom + GIST), then run a fast local KNN nearest-neighbour join. Perf: the distance gate uses geometry-space ST_DWithin(geom, point, deg) (GIST index, no geography cast) + geom <-> KNN order + ST_DistanceSphere metric recheck on the single nearest row. A geography-cast ST_DWithin in the WHERE defeated the index (58s+, full update never finished); the geometry-gate design does the whole ~41.9k-listing UPDATE in ~5.4s (refresh+match ~7s end-to-end), matching 16027 listings at 50m across 2229 distinct buildings. The match is GEO-NEAREST (approximate): a street-level-geocoded listing matches the nearest building within threshold_m (default 50m), not necessarily its exact cadastral building. Exact cadastral + parcel-containment deferred (cad_parcels FDW not exposed). Threshold is logged. - 124: cad_buildings_local table (empty) + GIST. Migration does not read the FDW (deploy-independent); populated by the refresh job. - 125: scrape_schedules seed source=cadastral_geo_match, enabled, next_run_at tomorrow 09:00 UTC (after geocode_missing). - tasks/cadastral_geo_match.py: refresh_cad_buildings_local (bulk FDW scan), match_listings_to_buildings (chunked LATERAL KNN UPDATE), run_cadastral_geo_match run-lifecycle wrapper. - scheduler: trigger_cadastral_geo_match_run + dispatch (sync DB-only, executor).	2026-06-18 10:18:54 +03:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
bot-backend	2ed3ae81bc	feat(domclick): wire DomClick citywide sweep into pipeline+scheduler (seeded disabled)	2026-06-17 23:21:49 +03:00
bot-backend	bc4c6852a1	feat(yandex): ingest novostroyki via gate-API newFlat=YES	2026-06-17 22:22:17 +03:00
lekss361	91f9a95f93	feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)	2026-06-16 16:09:19 +00:00
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	fd4a0d643d	fix(tradein): review minors — scheduler comment + save-error log label ... - scheduler.py: исправлен stale-комментарий yandex_detail_backfill (был: «YandexDetailScraper httpx, no proxy layer»; теперь: fetch via curl_cffi chrome120 + scraper_proxy_url, parse via YandexDetailScraper.parse) - yandex_detail_backfill.py: outer except — сообщение лога изменено на «save/iteration error for listing_id=%d» чтобы отличать save-ошибку от fetch/parse-None путей (run_id убран — он менее важен в этом scope)	2026-06-16 14:14:07 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	227e82dc01	feat(tradein): avito detail-enrichment nightly backfill (#1551) ... - New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill() * Single snapshot SELECT at start (guarantees termination) * Same proxy/AsyncSession path as scrape_pipeline.py step 5 * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed) * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368) * start = time.monotonic() initialized before try so except can reference it - Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop() - Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800, budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5 - 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing, full CI suite 1809 passed	2026-06-16 13:25:29 +03:00
bot-backend	a56420bdfb	fix(tradein): wire nightly geocode backfill для listings в in-app scheduler ... Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address, але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings. Parts: - geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d (loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW(). Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m), update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs. - scheduler.py: trigger_geocode_missing_listings_run + dispatch elif. - migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC). - tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів. 1794 passed.	2026-06-16 10:01:49 +03:00
bot-backend	479f01ae1a	Merge remote-tracking branch 'forgejo/main' into HEAD ... # Conflicts: # backend/app/services/site_finder/gate_verdict.py	2026-06-15 21:07:05 +03:00
bot-backend	b0a4df48e1	feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant (#973)	2026-06-15 19:09:26 +03:00
bot-backend	feab585ad5	feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974) ... - SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready), houses.yandex_jk_slug, GRANT gendesign_reader на market-слой - SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC) - yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href) - tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug → fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable) - admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs - scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch - tests: 8 task tests + 3 scheduler dispatch tests (20 pass)	2026-06-15 18:28:58 +03:00
Light1YT	86e9ea2937	fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») ... Многоагентный аудит + имплементация: один воркер на файл, точечные правки. Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv). Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538 Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498 Требуют cross-file (3, не тронуты): #1338, #1363, #1421 Пропущено (1): #1539 Не входило в партию: 22 needs-Leha issue (нужны решения владельца). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:21:11 +05:00
Light1YT	e1442d3b1d	feat(tradein-scheduler): nightly newbuilding-enrichment schedule (#973) ... Регистрирует отдельный in-app scheduler-источник для cian_newbuilding enrichment-backfill (houses_price_dynamics / house_reliability_checks / house_reviews), который раньше бежал только инлайн в Cian full-sweep. - run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed), делегирует backfill_newbuilding_enrichment (#972, уже в main). - trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop (зеркалит trigger_sber_index_pull_run): _claim_run guard, async task, zombie-reap наследуется; idempotency наследуется (skip уже обогащённых, per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов). - seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC (03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра. Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления (UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе. Closes #973	2026-06-13 16:24:03 +05:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
bot-backend	60353beaaf	feat(tradein): yandex city-sweep — один прогон SERP+address-enrich (#867) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:43:13 +00:00
bot-backend	2dd2d8e57e	feat(tradein): schedule yandex address backfill task (#855) (#866) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:31:10 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	a9e4aecb54	fix(tradein): data-layer tail A2+B+C — trgm index, FDW options, matview refresh (Refs #769) (#804) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 17:40:54 +00:00
bot-backend	2226b16cd3	fix(scheduler): atomic _claim_run via advisory xact lock (Refs #750) (#752) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:04:33 +00:00
bot-backend	6762a532eb	feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) (#745) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 15:43:38 +00:00
Light1YT	a060651328	feat(tradein): daily asking→sold ratio refresh (#648 S4) (#663)	2026-05-29 13:49:36 +00:00
Light1YT	e925a0ba6a	feat(tradein): per-source listing price history — schema + daily snapshot (#570) (#646)	2026-05-29 12:18:04 +00:00
Light1YT	7ec19b3536	feat(scheduler): dormant Yandex Realty city-sweep (#561) (#642)	2026-05-29 08:50:10 +00:00
Light1YT	b6cddcd9a8	fix(tradein): plain natural-key dedup_hash for rosreestr deals (#576) (#638)	2026-05-29 06:56:35 +00:00
Light1YT	e72cf87e35	feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563) (#630)	2026-05-28 15:31:24 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00