39 commits

Author	SHA1	Message	Date
bot-backend	5ed88c41a4	fix(avito): post-merge polish — unify fetch_mode guard + empty-render guard (#1811) (#1909) ... avito SERP browser post-merge polish (#1811): unify fetch_mode guard, empty-render guard, debug logging. Refs #1811	2026-06-26 07:44:22 +00:00
bot-backend	4eaf184d0e	fix(avito): graceful sidecar HTTP 500/503 in browser SERP fetch (#1790) ... Classify tradein-browser sidecar HTTP 500/503 in _fetch_serp_html_browser and route into existing rotation/retry machinery: soft-ban → IP rotation, transient → bounded backoff, both exhausted → AvitoRateLimitedError (graceful mark_banned, partial preserved) instead of mark_failed. Fixes run-#356 class where one late 500 nuked a run that had saved 3186 listings. Refs #1790	2026-06-25 06:47:41 +00:00
bot-backend	f026423715	feat(tradein/avito): per-card ЕКБ гео-фильтр (отсечь padding по России) + сплит комнат 4/5+ ... FIX 1: в _parse_html добавлен гео-фильтр per-card source_url. При avito_serp_ekb_only=True (ENV AVITO_SERP_EKB_ONLY, дефолт True) карточки без /ekaterinburg/ в source_url отбрасываются — Авито добивает выдачу «по всей России» при малом числе ЕКБ-результатов (rooms=4 давал 96% не-ЕКБ). Карточки без распознанного city-slug пропускаются консервативно. Лог: «avito SERP: dropped N non-EKB cards (padding)». FIX 2 (анализ): ROOM_SLUGS уже содержит отдельные бакеты «4-комн.» и «5+комн.» (разбиты ранее). rooms-значение берётся из текста заголовка карточки (_extract_rooms_from_title: regex «(\d)-к. квартира»), а не из бакета — 4/5/6 различаются по факту парсинга. Сплит slug'ов был сделан ранее, регрессий нет. Тесты: 4 новых unit-теста в test_avito_ekb_geo_filter.py покрывают: EKB+non-EKB → только EKB; ekb_only=False → все проходят; без city-slug → консервативно оставляем; все EKB → ничего не дропается.	2026-06-20 16:59:12 +03:00
bot-backend	93f7c9ae3f	feat(scrapers): avito SERP дефолт на BrowserFetcher (camoufox), curl_cffi fallback (#1805)	2026-06-19 20:42:04 +03:00
lekss361	19f740f2e9	Merge pull request 'fix(avito): de-glue house number from district in _clean_address (#1773 part)' (#1775) from fix/1773-avito-address-deglue into main ... Reviewed-on: #1775	2026-06-19 09:26:26 +00:00
Light1YT	78e7705a5f	fix(avito): de-glue house number from district marker in _clean_address (#1773) ... Avito leaks addresses with house number glued to district marker without separator («…Савкова, 29р-н Академический»), breaking house-number extraction and same-house matching (41 unmatched avito listings on one street). _clean_address now inserts ", " between a house-number token (digits + optional Cyrillic letter + optional /N|сN|кN|корпус|стр) and a following marker (р-н, мкр, район, г., пос, снт, …). - avito.py: idempotent _deglue_house_marker step (CSS/tail-strip kept) - tests: de-glue positive/negative/idempotency (24 passed) - data/sql/124_deglue_avito_addresses.sql: idempotent backfill for stored rows normalize.py НЕ тронут (общий fingerprint матчинга, #1534). Закрывает avito-часть #1773 (cian-без-номера + дедуп — отдельно, зона cian).	2026-06-19 13:33:43 +05:00
bot-backend	bee44ffd3d	fix(avito): tolerate 429 per-bucket instead of banning whole sweep ... Run 216 recovered 403s (merged fix) but still banned on a 429 at page 36 (deep pagination, backconnect conn-limit). Widen the 429 retry budget + one reconnect on exhaustion in _fetch_serp_html, and make fetch_all_secondary skip a blocked bucket and continue rather than aborting the run; only N consecutive blocked buckets (real hard ban) re-raises. Preserves hard-ban detection.	2026-06-18 23:03:28 +03:00
bot-backend	bdedac3012	feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) ... Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate sequentially newest-first and stop once cards fall below the watermark, skipping deep pagination (and the 403 clusters it triggers). since=None keeps the exact exhaustive bisection walk. No scheduler/schedule change here — wiring follows in a separate PR.	2026-06-18 22:01:46 +03:00
bot-backend	44e316c514	fix(avito): gate 403 reconnect-retry on proxy presence only, not rotate_url absence ... PR #1764's reconnect-retry was dead code in prod: the gate excluded configs with avito_proxy_rotate_url set, but prod has BOTH the backconnect SCRAPER_PROXY_URL (what curl_cffi egresses through) AND a leftover changeip URL for the separate auv/browser proxy. With max_rotations=0 neither path fired -> a single SERP 403 still aborted the sweep (run 212). The curl session always uses scraper_proxy_url, so session-rebuild rotates the exit IP regardless of changeip config.	2026-06-18 21:31:20 +03:00
bot-backend	9dc3be56e3	fix(avito): reconnect-retry on SERP 403/firewall under backconnect proxy ... Under a backconnect rotating proxy (no changeip URL) a single HTTP 403 or firewall interstitial aborted the entire avito_full_load sweep via AvitoBlockedError -> mark_banned. Each new connection through backconnect gets a fresh exit IP, so retry by rebuilding the curl_cffi session up to _AVITO_403_MAX_RETRIES times before giving up. Legacy single-IP changeip rotation and browser-mode paths unchanged. Mirrors existing 429 short-retry.	2026-06-18 20:41:32 +03:00
bot-backend	6454aa312d	fix(tradein): avito _extract_total_count returns 0 on no-results page (stop empty-bracket split waste + fallback-listing scrape)	2026-06-18 20:01:41 +03:00
bot-backend	2c6d0e4c71	feat(tradein): avito curl_cffi 429 short-retry (backconnect conn-limit transient, no rotate/ban)	2026-06-18 19:24:05 +03:00
bot-backend	ffecc6962d	perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city)	2026-06-18 17:55:55 +03:00
bot-backend	3f85fa59ff	perf(tradein): avito seed price-brackets with open top bucket (≤2k blocks, no price ceiling)	2026-06-18 17:29:31 +03:00
bot-backend	cbeeb9c414	fix(tradein): avito full-load probe resilience — browser 500/timeout splits/skips instead of crashing run	2026-06-18 16:03:39 +03:00
bot-backend	6c132d7f0a	fix(tradein): avito sweep page-level retry — transient page error no longer aborts run	2026-06-18 14:29:01 +03:00
bot-backend	1ddbf867f8	feat(tradein): avito exhaustive full-load (room×price bisection) — mirror cian	2026-06-18 13:55:18 +03:00
bot-backend	1a658a65a6	feat(avito): dedicated newbuilding (novostroyka) citywide sweep	2026-06-18 09:45:54 +03:00
bot-backend	b9eb478c8f	feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) ... Route each scraper source (avito/cian/yandex/domclick) to its own camoufox browser+proxy so they no longer wedge each other through a single global egress. Feature-flagged (FEATURE_BROWSER_POOL_ENABLED, default OFF): with the flag off the /fetch and /login paths are byte-for-byte the existing single-browser behavior. When on, /fetch routes by body["source"] to a per-proxy browser via BROWSER_PROXY_MAP (BROWSER_PROXY_AVITO/CIAN/YANDEX/DOMCLICK with legacy fallbacks), each guarded by its own lazy-launched lock. /login stays single-browser in Phase 1. BrowserFetcher gains a source arg (default avito) and sends it in the /fetch body; all scraper callsites pass their source. No docker-compose/.env.runtime changes (Phase 2, owner-gated).	2026-06-18 09:01:50 +03:00
bot-backend	687c9e5e9d	fix(avito): proactive IP rotate at sweep start + raise max_rotations 2->4 (Datadome ban recovery)	2026-06-17 23:35:24 +03:00
bot-backend	dd59ef0516	feat(avito): capture newbuilding ЖК id/url from SERP card; persist newbuilding_id/url	2026-06-17 23:15:35 +03:00
bot-backend	22a7a3f086	fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type	2026-06-17 22:24:13 +03:00
Light1YT	86e9ea2937	fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») ... Многоагентный аудит + имплементация: один воркер на файл, точечные правки. Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv). Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538 Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498 Требуют cross-file (3, не тронуты): #1338, #1363, #1421 Пропущено (1): #1539 Не входило в партию: 22 needs-Leha issue (нужны решения владельца). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 20:21:11 +05:00
bot-backend	441e48141b	feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 16:49:35 +00:00
bot-backend	da28a52067	feat(tradein): avito citywide + byrooms sweep mode (T6) (#848) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 06:13:19 +00:00
bot-backend	f7deed096b	fix(scrapers): avito per-card parse_failures counter + logging (Refs #823) (#830) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 20:01:50 +00:00
bot-backend	01593be438	feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 19:10:36 +00:00
bot-backend	96027357b5	feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726) (#788) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:52:35 +00:00
bot-backend	489666c735	fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:28:23 +00:00
bot-backend	e94929159b	fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) (#729) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 14:13:38 +00:00
Light1YT	f5c87201f1	fix(tradein): Avito listing_date extraction (3 формата) (#601)	2026-05-27 12:55:27 +00:00
lekss361	d4cbd4bb9f	feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)	2026-05-24 11:17:13 +00:00
lekss361	8fc309d9ad	fix(tradein-avito): strip Emotion CSS from listings.address (#502)	2026-05-24 10:52:50 +00:00
lekss361	222389cba9	fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)	2026-05-23 20:08:40 +00:00
lekss361	e79c9cec88	feat(tradein): global scraper delay setting (applies across all scrapers) (#485) ... Add global request delay floor + per-source delay management. - 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5) using ON CONFLICT DO NOTHING; reuses existing 053 table from #484 - scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY, refactored _get_setting_cached(); preserved yandex umbrella alias logic - admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0); cache invalidation on update; CAST(:d AS numeric) per psycopg v3 - avito/cian/n1 scrapers wired (yandex was wired by #484) - 18 unit tests pass (10 original updated for max() + 8 new global delay tests) Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).	2026-05-23 15:43:28 +00:00
lekss361	2914ff2ef2	feat(tradein): city sweep — auto ЕКБ pipeline (#477) ... Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel). - avito.py fetch_around(pages=N) — pagination backward compat default=1 - scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check - scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent) - admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel - migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction) - deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern) - 16 offline tests Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.	2026-05-23 14:38:07 +00:00
lekss361	47f4cc7d58	refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)	2026-05-23 11:55:18 +00:00
Light1YT	1d788b27d0	fix(tradein): Avito-парсер тащит фото и реальный адрес из DOM-карточек ... DOM-путь (_dom_card_to_lot) — основной для Avito: mobile API отдаёт бан, формат __initialState__ устарел, так что почти все объявления идут через него. Но он не извлекал фото вообще, а адрес брал мёртвым селектором [class*="geo-address"] — поэтому объявления Avito оставались без фото (серый блок в Сводке) и с заглушкой «Екатеринбург (Avito)». Теперь: - фото — из <img itemprop="image"> карточки-слайдера (до 5 шт); - адрес — первый <p> внутри data-marker="item-location" (улица + дом). Селекторы проверены на живой выдаче Avito.	2026-05-22 13:49:09 +05:00
lekss361	02267d0306	feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) ... - FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex) - Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in - WeasyPrint PDF (Брусника-style 4-page report) - Address autocomplete с typo-tolerance + 6 EKB presets - Изолированный docker stack gendesign-tradein (отдельная postgres БД) - Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/* - Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login) - Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml) - Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest Первый запуск на сервере (вручную, один раз): - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email) - docker network create gendesign_shared (если нет) - docker compose -p gendesign-tradein up -d - docker compose -p gendesign exec caddy caddy reload	2026-05-21 00:25:39 +03:00