108 commits

Author	SHA1	Message	Date
bot-backend	17c494a063	fix(scrapers): cian _walk_price_range stop silently truncating price-range tail (#1393)	2026-06-17 20:39:16 +03:00
lekss361	f8779c0812	feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)	2026-06-17 08:20:46 +00:00
bot-backend	b19c4a0b8c	fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience) ... Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected, call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.) return [] immediately without rotation. Mirrors the existing exhaustive-path pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success, captcha→rotate→success, exhausted retries→empty, 404→no-rotate.	2026-06-17 10:12:24 +03:00
lekss361	66433dfe1b	feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 19:47:16 +00:00
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	b390c8334c	fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea ... Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass, извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail, и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings внутри save_detail_enrichment. Имя поля kitchenArea — стандартное cian offer API (live-verify не делался, прокси недоступен в dev-окружении). Closes #1553	2026-06-16 14:37:49 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	5285a03c98	fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing ... STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200. Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков" ("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена. - RUS_FIELD_MAP: добавлены три записи. - DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing (в dataclass attrs и в locals-блоке parse_detail_html). - parse_detail_html: ветки elif для трёх полей; helper _parse_height_m(). - save_detail_enrichment: COALESCE-строки + params-dict для новых полей. - migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing (living_area_m2 уже существовала, migration не нужна). - Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m. Полный suite: 1795 passed, 2 deselected.	2026-06-16 10:19:09 +03:00
lekss361	8d1a13ee5e	Merge pull request 'fix(week-review): аудит код-ревью — 181 фиксов (label week ревью 1)' (#1543) from fix/week-review-audit into main ... Reviewed-on: #1543	2026-06-16 05:46:11 +00:00
bot-backend	bc504d9519	fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace)	2026-06-16 08:35:38 +03:00
bot-backend	b72af1cc49	fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108) ... Корень проблемы: avito_houses.py обходил канонический matching-путь через прямой INSERT ON CONFLICT(source, ext_house_id), создавая дубли для физически одного дома (~33% таблицы houses = ~4479 лишних строк). Forward fix: upsert_house/upsert_house_with_url → _persist_house(db, h, url), который вызывает match_or_create_house (3-tier: source_exact→fingerprint→geo→new, с pg_advisory_xact_lock) и затем UPDATE с enrichment-полями. avito-авторитетные поля (rating, developer, технические хар-ки) перезаписываются напрямую; address/lat/lon/year_built — через COALESCE чтобы не затирать canonical-данные. Merge-миграция 108: cluster_key по cadastral/address/geo → canonical (max не-NULL полей, geom присутствует, min(id)) → репойнт всех 11 зависимых таблиц с обработкой UNIQUE-коллизий → DELETE дублей → backfill house_sources/aliases.	2026-06-15 22:40:46 +03:00
bot-backend	278dafba76	fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела (#974) ... Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count / text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а RE_RATING требовал \s+ перед «из» → match=None. - RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален). \s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий разделитель перед «из». - test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы. Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+). Refs #974	2026-06-15 21:29:43 +03:00
bot-backend	891598a5d7	fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction (#972) ... Cian переименовал MFE ЖК-карточки и переместил reliability-контейнер, из-за чего fetch_newbuilding получал initialState=None → failed_fetch для всех домов (verified live 2026-06-15 на прод-браузере через мобильный egress): - MFE: 'newbuilding-card-desktop-fichering-frontend' → 'newbuilding-card-desktop-frontend' (старое имя на странице отсутствует → extract_state=None → backfill succeeded=0) - reliability: initialState.reliability → initialState.reliabilityState.reliability (inner dict сохранил форму {checkStatus, details[], banner, actions}) Проверено на ЖК «Светлый» (nb 17556): newbuilding.id, realtyValuation chart (27 точек), reliabilityState.reliability.checkStatus + 2 details — всё извлекается. test_fetch_newbuilding_parses_state: fixture обновлён под reliabilityState.reliability. Полный suite зелёный (1779 passed; 2 несвязанных env-only локальных падения — cian_valuation cache / search_api auth — в CI проходят). Refs #972	2026-06-15 21:12:05 +03:00
bot-backend	479f01ae1a	Merge remote-tracking branch 'forgejo/main' into HEAD ... # Conflicts: # backend/app/services/site_finder/gate_verdict.py	2026-06-15 21:07:05 +03:00
lekss361	b3e7a9f9fa	Merge pull request 'fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом (#972)' (#1540) from fix/cian-newbuilding-browser-fetch into main ... Reviewed-on: #1540	2026-06-15 15:44:42 +00:00
bot-backend	feab585ad5	feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974) ... - SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready), houses.yandex_jk_slug, GRANT gendesign_reader на market-слой - SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC) - yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href) - tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug → fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable) - admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs - scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch - tests: 8 task tests + 3 scheduler dispatch tests (20 pass)	2026-06-15 18:28:58 +03:00
Light1YT	86e9ea2937	fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») ... Многоагентный аудит + имплементация: один воркер на файл, точечные правки. Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv). Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538 Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498 Требуют cross-file (3, не тронуты): #1338, #1363, #1421 Пропущено (1): #1539 Не входило в партию: 22 needs-Leha issue (нужны решения владельца). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:21:11 +05:00
bot-backend	480fbba3ca	fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом (#972) ... curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch; session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search остаётся на curl_cffi SERP. Тесты: 21 passed.	2026-06-15 18:19:32 +03:00
Light1YT	b0fe292a63	fix(tradein): working cian ЖК-url resolver via cat.php SERP (#972) ... The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the 318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL) unfetchable -> the newbuilding-enrichment backfill matched 0 houses. Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the <h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries promo/recommendation zhk-* links before the title that would otherwise resolve the wrong ЖК and silently corrupt enrichment). Validated against the real 2.81MB prod SERP + an adversarial poisoned-recommendation test. Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching, rate-limited + resumable + idempotent. Keep the old resolver (deprecated). Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy (mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) + resolver hardened against wrong-ЖК. 19 tests green. Refs #972.	2026-06-08 11:13:06 +05:00
Light1YT	bcb1285341	feat(tradein): backfill task for cian newbuilding enrichment (#972) ... 950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing fetch_newbuilding + save_newbuilding_enrichment over each, populating houses_price_dynamics + house_reliability_checks (+ house_reviews when present). - Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe (price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC tiebreaker, reviews ON CONFLICT (source,ext_review_id)). - Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house (one captcha/failure logs + continues, never aborts the batch). - limit= for bounded proof runs; sizing counters (total/fetchable/pending). Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None (bool excluded before int). Fixes the live enrichment crash, not just backfill. house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК initialState — extending fetch_newbuilding for them is a documented follow-up. Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1 reliability row, idempotency proven across re-runs. Refs #972.	2026-06-07 14:09:24 +00:00
lekss361	a23e650fae	fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 (#940) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 22:39:06 +00:00
bot-backend	567549c3aa	feat(tradein): cian full-load secondary-only — drop novostroyki at parse (#938) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:35:32 +00:00
lekss361	d93e96fb0c	feat(tradein): checkpoint/resume for yandex full-load — survive restart (#930) (#937) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:34:23 +00:00
bot-backend	846c5014c6	feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) (#932) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 21:11:48 +00:00
lekss361	2bcccdf619	feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency (#931) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:10:26 +00:00
bot-backend	5e94d558b5	perf(tradein): cian full-load incremental save + concurrent pagination (#929) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 20:27:27 +00:00
bot-backend	0883f6d0f1	feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning (#926) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:54:19 +00:00
bot-backend	02af421799	feat(tradein): dedicated Cian mobile proxy — separate egress from Avito (#923) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 19:30:14 +00:00
bot-backend	62d989b8a3	feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login (#639) (#917) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:24:51 +00:00
bot-backend	92f44ac968	feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) (#916) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 18:10:38 +00:00
bot-backend	441e48141b	feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:49:35 +00:00
bot-backend	6044a1fe3c	fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server (#905) ... The playwright WS-server path (camoufox.server.launch_server) is incompatible with playwright >=1.45: camoufox 0.4.11's launchServer.js requires playwright/driver/package/lib/browserServerImpl.js, which no longer exists in any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND ('Server process terminated unexpectedly'). Verified by running the built image. Pivot (no playwright protocol between containers): - browser/server.py: aiohttp service holding a local AsyncCamoufox (headless, os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} → new_page/goto/content/close → {html}; GET /health. Page-recycle every N + crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock. - browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched). - backend BrowserFetcher: playwright.connect → httpx POST to the browser service (one retry on transport/HTTP error). All playwright imports dropped. - config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000). Verified by a REAL local image build + run: image builds, container stays Up (Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env, proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with network+proxy). 7 unit tests + ruff clean. Dormant: scraper_fetch_mode stays curl_cffi. Refs #905, #883, #884	2026-05-31 18:25:00 +03:00
lekss361	d6479c13b9	Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905)' (#907) from feat/884-browser-separate-container into main ... Reviewed-on: #907	2026-05-31 14:44:00 +00:00
bot-backend	5fb367295a	feat(browser): split camoufox into tradein-browser container, connect-mode (#905) ... Phase 0.5 — revise #884's in-backend camoufox into a dedicated browser service. camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/ proxy configured server-side); the backend connects remotely via playwright, so it no longer ships Firefox. - NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000; Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix) - BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint); recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry) - backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer - pyproject: remove camoufox[geoip] from backend, promote playwright to main deps - docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove mem_limit from backend, backend depends_on browser (service_started) - config: browser_ws_endpoint default ws://tradein-browser:3000/playwright - tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass) Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change. launch_server signature verified (**kwargs→launch_options, port passes through). Refs #905, #883, #884, #899	2026-05-31 17:24:35 +03:00
bot-backend	4c0338ce1d	fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup)	2026-05-31 17:21:30 +03:00
bot-backend	3c5e9ad368	feat(tradein): DomClick вторичка scraper via BrowserFetcher (#796)	2026-05-31 16:58:12 +03:00
bot-backend	fa2bd20ac6	refactor(tradein): drop dead schema — sale_type_text column + audit_log (#731) (#893) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:43 +00:00
lekss361	2890199d07	Merge pull request 'feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884)' (#892) from feat/884-browserfetcher-camoufox into main ... Reviewed-on: #892	2026-05-31 12:47:49 +00:00
bot-backend	e03f4e4534	feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884) ... Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox): async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT (no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile: +5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored -> re-resolved at build.) 9 mocked tests pass, ruff clean. WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image; merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify. Refs #884, #883	2026-05-31 15:41:56 +03:00
bot-backend	42de30cd5c	refactor(tradein): unify kadastr_num -> cadastral_number (#732) ... p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql (idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) + v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number, DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass. WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo. Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply (reconstructed from migration history 050/046, not live capture). Refs #732	2026-05-31 15:41:52 +03:00
lekss361	70838c7dfe	fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 08:27:47 +00:00
bot-backend	65b9247369	fix(matching): Tier 2b alias sync + collapse double SAVEPOINT (#849) (#870) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 08:05:48 +00:00
bot-backend	4494866574	fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:04:31 +00:00
bot-backend	da28a52067	feat(tradein): avito citywide + byrooms sweep mode (T6) (#848) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:13:19 +00:00
bot-backend	571be4233b	feat(tradein): yandex SERP curl_cffi+cookies transport + rooms×price combos (T4+T5) (#847) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:11:30 +00:00
bot-backend	5d2af47b87	fix(tradein): cian SERP warm-up GET для seed cookies (_CIAN_GK/_yasc) (T2) (#846) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:01:30 +00:00
bot-backend	08509b1b78	fix(scrapers): normalize cian descriptionMinhash list[int]→str + silent-0 guard (Refs #836) (#840) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:28:56 +00:00
bot-backend	f7deed096b	fix(scrapers): avito per-card parse_failures counter + logging (Refs #823) (#830) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:01:50 +00:00
bot-backend	2739973285	fix(scrapers): COALESCE-backfill listing_date/area_m2 in save_listings ON CONFLICT (Refs #821) (#829) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:58:06 +00:00
bot-backend	01593be438	feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:10:36 +00:00