38 commits

Author	SHA1	Message	Date
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	479f01ae1a	Merge remote-tracking branch 'forgejo/main' into HEAD ... # Conflicts: # backend/app/services/site_finder/gate_verdict.py	2026-06-15 21:07:05 +03:00
bot-backend	c9daf8e268	fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx (#974 review M1/M3) ... M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue). Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''. M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()` на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую. Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.	2026-06-15 18:41:08 +03:00
bot-backend	feab585ad5	feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974) ... - SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready), houses.yandex_jk_slug, GRANT gendesign_reader на market-слой - SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC) - yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href) - tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug → fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable) - admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs - scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch - tests: 8 task tests + 3 scheduler dispatch tests (20 pass)	2026-06-15 18:28:58 +03:00
Light1YT	86e9ea2937	fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») ... Многоагентный аудит + имплементация: один воркер на файл, точечные правки. Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv). Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538 Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498 Требуют cross-file (3, не тронуты): #1338, #1363, #1421 Пропущено (1): #1539 Не входило в партию: 22 needs-Leha issue (нужны решения владельца). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:21:11 +05:00
lekss361	d93e96fb0c	feat(tradein): checkpoint/resume for yandex full-load — survive restart (#930) (#937) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:34:23 +00:00
bot-backend	846c5014c6	feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) (#932) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:11:48 +00:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	5e94d558b5	perf(tradein): cian full-load incremental save + concurrent pagination (#929) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:27:27 +00:00
bot-backend	0883f6d0f1	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning (#926) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:54:19 +00:00
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
lekss361	997eda489c	fix(tradein): single-run guard на avito-city-sweep admin-endpoint (#874) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:55:33 +00:00
bot-backend	60353beaaf	feat(tradein): yandex city-sweep — один прогон SERP+address-enrich (#867) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:43:13 +00:00
bot-backend	f2a123e2aa	feat(tradein): cian city-sweep — один прогон SERP+detail(+price-history)+houses (#864) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:24:34 +00:00
bot-backend	d78ae9f649	feat(tradein): avito city-sweep — IMV-фаза (один прогон = SERP+detail+houses+IMV) (#861) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:09:42 +00:00
bot-backend	5373f5db63	feat(tradein): admin backfill endpoints — cian price-history (T8a) + yandex address (T10) + house IMV (T7) (#856) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:37:10 +00:00
bot-backend	bcd035034a	fix(tradein): SSRF host-allowlist на admin scrape + удалён dead ADMIN_TOKEN (Refs #756) (#778) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:27:19 +00:00
bot-backend	6762a532eb	feat(scrapers): wire N1.ru city sweep — pipeline + scheduler + admin endpoint (#575) (#745) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 15:43:38 +00:00
Light1YT	1731704ddc	feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627)	2026-05-28 13:57:43 +00:00
lekss361	ba9cc3aef1	feat(tradein/cian): valuation backfill + runner script для prod (#539)	2026-05-24 15:16:45 +00:00
lekss361	bc594e3f79	feat(tradein/cian): batch backfill task для offer_price_history + houses_price_dynamics (#535)	2026-05-24 14:55:47 +00:00
lekss361	591039493c	feat(tradein/cian): admin endpoints + UI for detail+newbuilding history scrape (#525)	2026-05-24 14:05:55 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	b21d7c7c85	feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484)	2026-05-23 15:28:34 +00:00
lekss361	0202720096	feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)	2026-05-23 14:54:25 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	dbad7a626b	fix(tradein): убрать require_admin полностью — Caddy basic_auth достаточен (#474)	2026-05-23 14:21:01 +00:00
lekss361	b5d206bfa8	feat(tradein): Stage 4a — API endpoints for Avito enrichment + IMV benchmark UI (#460)	2026-05-23 13:34:38 +00:00
lekss361	2dd76ea5e8	feat(tradein): cian_session cookies management + admin endpoints (Wave 4 Worker A) (#457)	2026-05-23 13:22:00 +00:00
Light1YT	0b14538f7d	fix(tradein): бюджет по времени в geocode-missing — cron не таймаутит ... cron geocode-missing-шаг дёргает эндпоинт через `curl -m 320`, а геокодинг 100 адресов упирается в Nominatim (1 req/sec + typo-тиры на провалах) и выходит за 320с. При таймауте curl cron-loop обрывался — объявления Cian/N1 оставались без координат и были невидимы для оценки (ST_DWithin по geom). Сейчас ~200 таких объявлений в БД. geocode_missing теперь выходит из цикла по бюджету 240с (заведомо меньше cron-таймаута) и возвращает remaining > 0 — cron штатно продолжает следующим чанком. Заодно убран мёртвый import asyncio.	2026-05-22 14:37:15 +05:00
TradeIn Deploy	4ea35d5740	feat(tradein): geocode-missing для deals (#4 завершение) ... geocode-missing получил target=listings|deals. deals.geocode_tried_at добавлен в миграцию 005. cron геокодит и сделки Росреестра по улицам.	2026-05-21 20:04:19 +03:00
TradeIn Deploy	7adb7cff66	feat(tradein): auth на admin-эндпоинтах (#2) ... /api/v1/admin/scrape и geocode-missing закрыты X-Admin-Token. settings.admin_token (env ADMIN_TOKEN). Пусто = открыто (dev), задано = требуем заголовок. cron шлёт токен из .env.runtime.	2026-05-21 19:47:01 +03:00
TradeIn Deploy	609744df91	fix(tradein): чанковый геокодинг — cron больше не таймаутится (#1) ... geocode_tried_at: после каждой попытки NOW(), повтор только через 7 дней. Эндпоинт чанкуется (limit=100), отдаёт remaining. cron вызывает в цикле.	2026-05-21 19:43:12 +03:00
TradeIn Deploy	d2a740779d	fix(tradein): реальные координаты Cian/N1 через геокодинг (был баг 0 аналогов) ... Cian/N1 ставили anchor+jitter координаты — фейковые, вокруг центрального якоря. Для адресов вдали от центра ST_DWithin не находил аналоги → 0. Теперь Cian/N1 оставляют lat/lon=None, geocode-missing проставляет реальные координаты по адресу (geom пересчитывается триггером). cron вызывает geocode-missing после скрейпа. Адрес чистится от суффикса · р-н.	2026-05-21 13:14:33 +03:00
TradeIn Deploy	42a0c31f27	feat(tradein): N1.ru scraper + cron для 4 источников ... Добавлен 4-й источник N1.ru (ekaterinburg.n1.ru) — региональный портал, хорошо представлен на Урале. DOM-парсинг карточек data-test=offers-list-item через curl_cffi. Multi-room сегментация 1/2/3/4к. cron-scrape.sh переписан под прод: запрос к backend через docker exec (без публичного admin/scrape), все 4 источника (avito/cian/yandex/n1).	2026-05-21 11:44:32 +03:00
lekss361	02267d0306	feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) ... - FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex) - Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in - WeasyPrint PDF (Брусника-style 4-page report) - Address autocomplete с typo-tolerance + 6 EKB presets - Изолированный docker stack gendesign-tradein (отдельная postgres БД) - Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/* - Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login) - Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml) - Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest Первый запуск на сервере (вручную, один раз): - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email) - docker network create gendesign_shared (если нет) - docker compose -p gendesign-tradein up -d - docker compose -p gendesign exec caddy caddy reload	2026-05-21 00:25:39 +03:00