Commit graph

17 commits

Author SHA1 Message Date
86e9ea2937 fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1»)
Многоагентный аудит + имплементация: один воркер на файл, точечные правки.
Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты
не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv).

Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538

Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498
Требуют cross-file (3, не тронуты): #1338, #1363, #1421
Пропущено (1): #1539

Не входило в партию: 22 needs-Leha issue (нужны решения владельца).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:21:11 +05:00
441e48141b feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode (#901)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 16:49:35 +00:00
da28a52067 feat(tradein): avito citywide + byrooms sweep mode (T6) (#848)
Some checks failed
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 24s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 06:13:19 +00:00
f7deed096b fix(scrapers): avito per-card parse_failures counter + logging (Refs #823) (#830)
Some checks failed
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 25s
Deploy Trade-In / build-backend (push) Successful in 1m11s
Deploy Trade-In / deploy (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 20:01:50 +00:00
01593be438 feat(scrapers): route cian sessions through mobile proxy (Refs #806) (#818)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 25s
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 35s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 19:10:36 +00:00
96027357b5 feat(tradein): Avito mobile proxy egress + firewall detect + MSK date (#623, #726) (#788)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / deploy (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 39s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 16:52:35 +00:00
489666c735 fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779)
Some checks failed
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 16:28:23 +00:00
e94929159b fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) (#729)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 34s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 14:13:38 +00:00
f5c87201f1 fix(tradein): Avito listing_date extraction (3 формата) (#601)
All checks were successful
Deploy Trade-In / build-backend (push) Successful in 49s
Deploy Trade-In / deploy (push) Successful in 36s
Deploy Trade-In / changes (push) Successful in 1m28s
Deploy Trade-In / build-frontend (push) Has been skipped
2026-05-27 12:55:27 +00:00
d4cbd4bb9f feat(tradein-scrapers): parse listing_date from Avito + Yandex (#508)
Some checks failed
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Has been cancelled
2026-05-24 11:17:13 +00:00
8fc309d9ad fix(tradein-avito): strip Emotion CSS from listings.address (#502)
Some checks failed
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Has been cancelled
2026-05-24 10:52:50 +00:00
222389cba9 fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-23 20:08:40 +00:00
e79c9cec88 feat(tradein): global scraper delay setting (applies across all scrapers) (#485)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 23s
Deploy Trade-In / build-backend (push) Successful in 50s
Add global request delay floor + per-source delay management.

- 054: seed 'global'=0 + per-source rows (avito=7, cian=5, n1=5, domrf=5, rosreestr=5)
  using ON CONFLICT DO NOTHING; reuses existing 053 table from #484
- scraper_settings.py: get_scraper_delay() = max(per_source, global); _GLOBAL_KEY,
  refactored _get_setting_cached(); preserved yandex umbrella alias logic
- admin.py: GET /scraper-settings + PUT /scraper-settings/{source} (ge=0.0, le=60.0);
  cache invalidation on update; CAST(:d AS numeric) per psycopg v3
- avito/cian/n1 scrapers wired (yandex was wired by #484)
- 18 unit tests pass (10 original updated for max() + 8 new global delay tests)

Closes follow-up gap from #486 (Cian admin slider min=0 now accepted by API).
2026-05-23 15:43:28 +00:00
2914ff2ef2 feat(tradein): city sweep — auto ЕКБ pipeline (#477)
All checks were successful
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).

- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests

Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
47f4cc7d58 refactor(tradein): avito search — remove jitter (C-5) + mobile API dead code + parse houseLinks (#441)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 18s
2026-05-23 11:55:18 +00:00
1d788b27d0 fix(tradein): Avito-парсер тащит фото и реальный адрес из DOM-карточек
DOM-путь (_dom_card_to_lot) — основной для Avito: mobile API отдаёт
бан, формат __initialState__ устарел, так что почти все объявления
идут через него. Но он не извлекал фото вообще, а адрес брал мёртвым
селектором [class*="geo-address"] — поэтому объявления Avito оставались
без фото (серый блок в Сводке) и с заглушкой «Екатеринбург (Avito)».

Теперь:
- фото — из <img itemprop="image"> карточки-слайдера (до 5 шт);
- адрес — первый <p> внутри data-marker="item-location" (улица + дом).

Селекторы проверены на живой выдаче Avito.
2026-05-22 13:49:09 +05:00
lekss361
02267d0306 feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest

Первый запуск на сервере (вручную, один раз):
  - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
  - docker network create gendesign_shared (если нет)
  - docker compose -p gendesign-tradein up -d
  - docker compose -p gendesign exec caddy caddy reload
2026-05-21 00:25:39 +03:00