479f01ae1a
Merge remote-tracking branch 'forgejo/main' into HEAD
...
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (push) Successful in 1m7s
CI / openapi-codegen-check (push) Failing after 2m9s
CI / frontend-tests (pull_request) Successful in 1m4s
CI / openapi-codegen-check (pull_request) Failing after 1m37s
CI / backend-tests (pull_request) Failing after 9m7s
CI / backend-tests (push) Failing after 9m11s
# Conflicts:
# backend/app/services/site_finder/gate_verdict.py
2026-06-15 21:07:05 +03:00
b0a4df48e1
feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant ( #973 )
2026-06-15 19:09:26 +03:00
b3e7a9f9fa
Merge pull request 'fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )' ( #1540 ) from fix/cian-newbuilding-browser-fetch into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 40s
Deploy Trade-In / deploy (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Reviewed-on: #1540
2026-06-15 15:44:42 +00:00
8279b0ee1f
fix(week-review): cross-file добивка код-ревью ( #1338 , #1363 , #1419 , #1421 , #1424 , #1480 )
...
- #1338 concepts.py: geometry.generate вынесен в run_in_threadpool (не блокирует event loop)
- #1363 house_imv_backfill.py + scrape_pipeline.py: периодический heartbeat в долгих IMV-фазах (reap_zombies не помечает живой sweep zombie)
- #1419 site-finder-api.ts: placeholderData keepPreviousData для bbox-запроса (KPI не мигает «0/0» при пане/зуме)
- #1421 site-finder-api.ts: area_ha nullable + адаптер не подставляет 0; null-гейт потребителей EntryMap.tsx, ParcelDrawer.tsx
- #1424/#1480 EstimateResult.tsx: блок «Параметры объекта» скрыт при восстановлении по shared-ссылке (нет мусора «0 м²/Студия/Этаж 0 из 0»)
Верификация: tsc --noEmit 0 ошибок; py_compile OK.
2026-06-15 20:29:00 +05:00
feab585ad5
feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant ( #974 )
...
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 7s
- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
(SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)
2026-06-15 18:28:58 +03:00
86e9ea2937
fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1»)
...
Многоагентный аудит + имплементация: один воркер на файл, точечные правки.
Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты
не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv).
Полностью исправлено (169): #1336 , #1337 , #1339 , #1340 , #1341 , #1342 , #1343 , #1345 , #1346 , #1348 , #1349 , #1350 , #1351 , #1354 , #1356 , #1358 , #1359 , #1360 , #1362 , #1364 , #1365 , #1366 , #1367 , #1368 , #1369 , #1370 , #1371 , #1372 , #1373 , #1374 , #1375 , #1376 , #1377 , #1378 , #1379 , #1380 , #1381 , #1382 , #1384 , #1385 , #1386 , #1387 , #1388 , #1389 , #1390 , #1391 , #1392 , #1394 , #1395 , #1396 , #1397 , #1399 , #1400 , #1401 , #1402 , #1403 , #1404 , #1408 , #1409 , #1410 , #1411 , #1412 , #1413 , #1414 , #1415 , #1416 , #1417 , #1418 , #1420 , #1423 , #1425 , #1426 , #1427 , #1428 , #1429 , #1430 , #1431 , #1432 , #1433 , #1434 , #1435 , #1437 , #1438 , #1439 , #1440 , #1441 , #1442 , #1443 , #1444 , #1445 , #1446 , #1447 , #1448 , #1449 , #1450 , #1451 , #1452 , #1453 , #1454 , #1455 , #1456 , #1457 , #1458 , #1459 , #1460 , #1461 , #1462 , #1463 , #1464 , #1465 , #1466 , #1467 , #1468 , #1469 , #1471 , #1472 , #1473 , #1474 , #1476 , #1478 , #1479 , #1481 , #1482 , #1483 , #1484 , #1485 , #1487 , #1488 , #1489 , #1490 , #1491 , #1492 , #1493 , #1494 , #1495 , #1496 , #1497 , #1499 , #1500 , #1501 , #1502 , #1504 , #1505 , #1506 , #1507 , #1510 , #1514 , #1515 , #1516 , #1517 , #1518 , #1519 , #1521 , #1522 , #1523 , #1524 , #1525 , #1526 , #1527 , #1528 , #1529 , #1531 , #1532 , #1533 , #1534 , #1535 , #1536 , #1537 , #1538
Частично (9, in-file часть, остаток cross-file): #1361 , #1419 , #1422 , #1424 , #1470 , #1475 , #1477 , #1480 , #1498
Требуют cross-file (3, не тронуты): #1338 , #1363 , #1421
Пропущено (1): #1539
Не входило в партию: 22 needs-Leha issue (нужны решения владельца).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:21:11 +05:00
480fbba3ca
fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
2026-06-15 18:19:32 +03:00
e1442d3b1d
feat(tradein-scheduler): nightly newbuilding-enrichment schedule ( #973 )
...
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 7s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 1m3s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 40s
Регистрирует отдельный in-app scheduler-источник для cian_newbuilding
enrichment-backfill (houses_price_dynamics / house_reliability_checks /
house_reviews), который раньше бежал только инлайн в Cian full-sweep.
- run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed),
делегирует backfill_newbuilding_enrichment (#972 , уже в main).
- trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop
(зеркалит trigger_sber_index_pull_run): _claim_run guard, async task,
zombie-reap наследуется; idempotency наследуется (skip уже обогащённых,
per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов).
- seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC
(03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра.
Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно
на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления
(UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе.
Closes #973
2026-06-13 16:24:03 +05:00
a9a489bba5
refactor(tradein): убрать dead listing_segment param из _fetch_anchor_comps + негативные guard-asserts (review fixup, #1186 )
...
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
- Удалён параметр listing_segment: str | None = None из сигнатуры _fetch_anchor_comps
(guard хардкожен в SQL-блоках Tier A/C, параметр не использовался)
- Убраны seg_counts / target_segment в caller-блоке SAME-BUILDING ANCHOR (~L2174)
- Docstring Tier C обновлён: «вторичка-канон guard (#1186 ): NULL = legacy вторичка»
- test_segment_guard_1186.py: добавлены 4 негативных assert'а:
· no bare = 'vtorichka' без IS NULL OR
· no deprecated <> / != 'novostroyki' form
· _fetch_anchor_comps не содержит listing_segment в сигнатуре
· Tier C не использует CAST(:segment AS text) (параметрическую форму)
2026-06-12 12:01:00 +03:00
b70ae46885
fix(tradein): guard listing_segment во всех читателях listings — novostroyki вне comps ( #1186 )
...
Канон-предикат: (listing_segment IS NULL OR listing_segment = 'vtorichka')
NULL = legacy вторичка до миграции 011 (sweep'ы были secondary-only) — отбрасывать нельзя.
Затронуты все SELECT-пути, влияющие на оценку и корректирующие коэффициенты:
- estimator.py: _COMMON_WHERE (Tier S/H), Tier W inline, Tier A anchor (добавлен впервые),
Tier C anchor (заменён с параметрического CAST(:segment) на канон — прежний вариант
отбрасывал NULL-строки при segment='vtorichka').
- asking_to_sold_ratio.py: ask_side, ask_global (_REDERIVE_SQL); bounds (_REDERIVE_BOUNDS_SQL);
ask_tiered, ask_side_all, ask_global (_REDERIVE_TIERED_SQL) — 6 мест.
- sber_index.py: reconciliation-запрос (диагностический asking-median).
Не затронуты: скрейперы/scrape_pipeline, search matview (catalog display, не оценка),
admin stats, geocoding, cian price history, matching — обосновано в инвентаризации.
Тесты: новый test_segment_guard_1186.py (статика SQL + поведение mock); обновлён
test_asking_to_sold_ratio.py (subset-тест нормализует guard, добавленный в refresh но
отсутствующий в 080-seed).
2026-06-12 11:52:09 +03:00
b0fe292a63
fix(tradein): working cian ЖК-url resolver via cat.php SERP ( #972 )
...
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 5s
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 1m20s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 46s
The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the
318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL)
unfetchable -> the newbuilding-enrichment backfill matched 0 houses.
Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP
and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the
<h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries
promo/recommendation zhk-* links before the title that would otherwise resolve
the wrong ЖК and silently corrupt enrichment). Validated against the real
2.81MB prod SERP + an adversarial poisoned-recommendation test.
Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR
cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching,
rate-limited + resumable + idempotent. Keep the old resolver (deprecated).
Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully
enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode
anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy
(mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) +
resolver hardened against wrong-ЖК. 19 tests green.
Refs #972 .
2026-06-08 11:13:06 +05:00
bcb1285341
feat(tradein): backfill task for cian newbuilding enrichment ( #972 )
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 46s
Deploy Trade-In / test (push) Successful in 37s
Deploy Trade-In / build-backend (push) Successful in 45s
950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to
ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing
fetch_newbuilding + save_newbuilding_enrichment over each, populating
houses_price_dynamics + house_reliability_checks (+ house_reviews when present).
- Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe
(price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC
tiebreaker, reviews ON CONFLICT (source,ext_review_id)).
- Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house
(one captcha/failure logs + continues, never aborts the batch).
- limit= for bounded proof runs; sizing counters (total/fetchable/pending).
Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate
drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on
LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None
(bool excluded before int). Fixes the live enrichment crash, not just backfill.
house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК
initialState — extending fetch_newbuilding for them is a documented follow-up.
Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1
reliability row, idempotency proven across re-runs. Refs #972 .
2026-06-07 14:09:24 +00:00
a23e650fae
fix(tradein): system-curl subprocess transport for Yandex SERP over SOCKS5 ( #940 )
...
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 22:39:06 +00:00
567549c3aa
feat(tradein): cian full-load secondary-only — drop novostroyki at parse ( #938 )
...
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 35s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 21:35:32 +00:00
d93e96fb0c
feat(tradein): checkpoint/resume for yandex full-load — survive restart ( #930 ) ( #937 )
...
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been cancelled
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 21:34:23 +00:00
3294bc2021
fix(tradein): exclude novostroyki from radius-median comp pool ( #935 )
...
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 21:33:23 +00:00
571980c56e
feat(tradein): ppm²-tier segmentation of asking→sold ratio ( #928 , flag OFF) ( #934 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 21:22:39 +00:00
846c5014c6
feat(tradein): checkpoint/resume for cian full-load — survive restart ( #930 ) ( #932 )
...
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 38s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 21:11:48 +00:00
2bcccdf619
feat(tradein): exhaustive Yandex realty load — adaptive room×price + concurrency ( #931 )
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 21:10:26 +00:00
5e94d558b5
perf(tradein): cian full-load incremental save + concurrent pagination ( #929 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 43s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 20:27:27 +00:00
0883f6d0f1
feat(tradein): exhaustive Cian EKB secondary load — room×price partitioning ( #926 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 43s
Deploy Trade-In / deploy (push) Successful in 37s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 19:54:19 +00:00
ca0943689d
fix(tradein): skip TLS verify for sberindex.ru pull — server omits LE intermediate ( #922 )
2026-05-31 22:33:42 +03:00
02af421799
feat(tradein): dedicated Cian mobile proxy — separate egress from Avito ( #923 )
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Failing after 30s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 19:30:14 +00:00
4999c877ae
feat(tradein): СберИндекс time-adjust for frozen Rosreestr ДКП + per-dashboard filter fix ( #794 ) ( #919 )
...
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Failing after 33s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 18:48:22 +00:00
62d989b8a3
feat(tradein): Cian browser auto-login — POST /admin/scrape/cian/auto-login ( #639 ) ( #917 )
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / deploy (push) Successful in 1m17s
Deploy Trade-In / build-backend (push) Successful in 50s
Deploy Trade-In / build-browser (push) Successful in 2m8s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 18:24:51 +00:00
92f44ac968
feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher ( #915 ) ( #916 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 46s
Deploy Trade-In / deploy (push) Successful in 37s
Deploy Trade-In / build-browser (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 18:10:38 +00:00
441e48141b
feat(tradein): avito SERP via BrowserFetcher behind scraper_fetch_mode ( #901 )
...
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 31s
Deploy Trade-In / build-backend (push) Successful in 44s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 16:49:35 +00:00
6044a1fe3c
fix(browser): local camoufox + HTTP wrapper, drop broken playwright WS server ( #905 )
...
The playwright WS-server path (camoufox.server.launch_server) is incompatible
with playwright >=1.45: camoufox 0.4.11's launchServer.js requires
playwright/driver/package/lib/browserServerImpl.js, which no longer exists in
any playwright 1.45-1.60 → tradein-browser crash-looped with MODULE_NOT_FOUND
('Server process terminated unexpectedly'). Verified by running the built image.
Pivot (no playwright protocol between containers):
- browser/server.py: aiohttp service holding a local AsyncCamoufox (headless,
os/locale/geoip/humanize, proxy from SCRAPER_PROXY_URL). POST /fetch {url} →
new_page/goto/content/close → {html}; GET /health. Page-recycle every N +
crash-recovery (relaunch + 1 retry), serialized via asyncio.Lock.
- browser/Dockerfile: +aiohttp (camoufox fetch-as-root/#908 block untouched).
- backend BrowserFetcher: playwright.connect → httpx POST to the browser service
(one retry on transport/HTTP error). All playwright imports dropped.
- config: browser_ws_endpoint → browser_http_endpoint (http://tradein-browser:3000 ).
Verified by a REAL local image build + run: image builds, container stays Up
(Restarts=0, no crash-loop), GET /health=200, POST /fetch launches camoufox and
drives Firefox (only NS_ERROR_UNKNOWN_HOST = no DNS egress in the local build env,
proving the full HTTP→camoufox→Firefox chain; real fetch validates on prod with
network+proxy). 7 unit tests + ruff clean.
Dormant: scraper_fetch_mode stays curl_cffi. Refs #905 , #883 , #884
2026-05-31 18:25:00 +03:00
d6479c13b9
Merge pull request 'feat(browser): split camoufox into a separate tradein-browser container (Phase 0.5, #905 )' ( #907 ) from feat/884-browser-separate-container into main
...
Deploy Trade-In / build-frontend (push) Successful in 26s
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-browser (push) Failing after 1m16s
Deploy Trade-In / build-backend (push) Successful in 1m14s
Deploy Trade-In / deploy (push) Has been skipped
Reviewed-on: #907
2026-05-31 14:44:00 +00:00
98b0632dae
Merge pull request 'feat(tradein): DomClick вторичка scraper via BrowserFetcher ( #796 )' ( #904 ) from feat/796-domclick-scraper into main
...
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Has been cancelled
Reviewed-on: #904
2026-05-31 14:43:39 +00:00
5fb367295a
feat(browser): split camoufox into tradein-browser container, connect-mode ( #905 )
...
Phase 0.5 — revise #884 's in-backend camoufox into a dedicated browser service.
camoufox.server.launch_server exposes a Playwright WS endpoint (fingerprint/geoip/
proxy configured server-side); the backend connects remotely via playwright, so it
no longer ships Firefox.
- NEW tradein-mvp/browser/{server.py,Dockerfile}: camoufox WS server on :3000;
Firefox fetch under app-user + ENV HOME (carries the #899 cache-path fix)
- BrowserFetcher rewritten launch-in-process → playwright.firefox.connect(ws_endpoint);
recycle (close+reconnect every N) + crash-recovery (TargetClosed → reconnect + 1 retry)
- backend Dockerfile slimmed: drop 5 Firefox libs + camoufox fetch layer
- pyproject: remove camoufox[geoip] from backend, promote playwright to main deps
- docker-compose.prod.yml: add browser service (mem_limit 2.5g moves here), remove
mem_limit from backend, backend depends_on browser (service_started)
- config: browser_ws_endpoint default ws://tradein-browser:3000/playwright
- tests/scrapers/test_browser_fetcher.py rewritten for connect-mode (6 pass)
Dormant: scraper_fetch_mode stays curl_cffi → no prod behavior change.
launch_server signature verified (**kwargs→launch_options, port passes through).
Refs #905 , #883 , #884 , #899
2026-05-31 17:24:35 +03:00
4c0338ce1d
fix(tradein): domklik source name, scraper delay, DataDome block detect ( #796 fixup)
2026-05-31 17:21:30 +03:00
3c5e9ad368
feat(tradein): DomClick вторичка scraper via BrowserFetcher ( #796 )
2026-05-31 16:58:12 +03:00
53af9a8c55
feat(tradein): add EKB/MSK REF_AREA codes to sber_price_index pull ( #794 ) ( #900 )
...
Deploy Trade-In / test (push) Successful in 34s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 2m20s
Deploy Trade-In / deploy (push) Failing after 51s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 13:56:17 +00:00
c2888ac52d
feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert ( #888 ) ( #894 )
...
Deploy Trade-In / build-backend (push) Successful in 2m21s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / deploy (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 13:13:55 +00:00
fa2bd20ac6
refactor(tradein): drop dead schema — sale_type_text column + audit_log ( #731 ) ( #893 )
...
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 13:13:43 +00:00
2890199d07
Merge pull request 'feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884 )' ( #892 ) from feat/884-browserfetcher-camoufox into main
...
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-backend (push) Successful in 2m45s
Deploy Trade-In / build-frontend (push) Successful in 24s
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / deploy (push) Successful in 1m5s
Reviewed-on: #892
2026-05-31 12:47:49 +00:00
8df536ca5f
Merge pull request 'refactor(tradein): unify kadastr_num → cadastral_number ( #732 )' ( #891 ) from feat/732-cadastral-unify into main
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 28s
Deploy Trade-In / build-backend (push) Has been cancelled
Reviewed-on: #891
2026-05-31 12:47:10 +00:00
de92c54540
feat(tradein): СберИндекс monthly city-level price index pull ( #887 , data-layer) ( #890 )
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 41s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 12:44:56 +00:00
e03f4e4534
feat(tradein): BrowserFetcher camoufox wrapper + Docker/config infra (Phase 0, #884 )
...
Epic #883 foundation. Provider-agnostic BrowserFetcher (camoufox headless Firefox):
async ctx mgr + page-recycle + crash-recovery + proxy. camoufox/playwright lazy
imports (won't break app import). scraper_fetch_mode defaults curl_cffi -> DORMANT
(no parser/estimator/avito/cian change). config: 4 browser_* settings. Dockerfile:
+5 Firefox libs + camoufox fetch layer. compose: mem_limit 2.5g. (uv.lock gitignored
-> re-resolved at build.) 9 mocked tests pass, ruff clean.
WARN: Docker build + container fetch UNVERIFIED (no local Docker); ~960MB image;
merge auto-deploys to prod. MERGE-GATE post-demo + CI-build-verify.
Refs #884 , #883
2026-05-31 15:41:56 +03:00
42de30cd5c
refactor(tradein): unify kadastr_num -> cadastral_number ( #732 )
...
p3 naming-consistency (cadastral cols 0% filled). Migration 094_cadastral_unify.sql
(idempotent DO-block IF EXISTS): drop+recreate listings_search_mv (MV) +
v_data_quality (view), RENAME deals/house_metadata kadastr_num->cadastral_number,
DROP listings.kadastr_num + listings_kadastr_idx. Code SQL-string renames in
search_query/estimator/base (Lot dataclass field NOT renamed). 240 tests pass.
WARN: destructive DDL auto-applies to prod on deploy -> MERGE-GATE post-demo.
Reviewer: verify recreated MV/view DDL vs LIVE pg_get_viewdef before apply
(reconstructed from migration history 050/046, not live capture).
Refs #732
2026-05-31 15:41:52 +03:00
f3ba8acd76
feat(tradein): per-anchor watchdog timeout in city sweeps ( #880 ) ( #881 )
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 29s
Deploy Trade-In / build-backend (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 10:40:41 +00:00
d1034bbc70
Merge pull request 'feat(tradein): same-building anchor safe-guard ( #755 )' ( #877 ) from feat/755-anchor-safeguard into main
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 28s
Deploy Trade-In / build-backend (push) Successful in 41s
Deploy Trade-In / deploy (push) Successful in 36s
Reviewed-on: #877
2026-05-31 10:36:22 +00:00
ec82742456
feat(tradein): same-building anchor safe-guard ( #755 )
...
CLIENT-VISIBLE pricing (killer-factor, demo 2026-06-01). The same-building
anchor replaced headline median/range/confidence with an over-confident
premium on as few as 2 same-building asking comps. Human-chosen (Anton)
3-part safe-guard, all implemented:
1. min_comps 2 -> 4: anchor fires only with >=4 same-building comps.
2. confidence cap: at <5 anchor comps, confidence cannot be 'high' -> medium.
3. MAD-clip: drop price_per_m2 outliers among anchor comps before aggregation
(k=estimate_sb_mad_k=3.5); if <min_comps survive, anchor does NOT fire
(fallback to radius-median). Guards against price typos inflating anchor.
Anchor still REPLACES the headline (uplift-no-replace variant NOT chosen).
quarter-index/radius-tier/asking-sold untouched. 8 new unit tests; 61 anchor
tests pass, ruff clean.
MERGE-GATE (client-visible, like #764/#858): do NOT merge without a green
backtest #763 (held-out MAPE <= 18.7% + canonical A/B Малышева 125 /
Заводская 44а). Backtest needs live DB — pre-merge human/qa step.
Refs #755 , #694
2026-05-31 12:19:50 +03:00
d9b8047a7d
fix(tradein): yandex address-enrich — city-agnostic title regex (спутники ЕКБ) ( #875 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 29s
Deploy Trade-In / build-backend (push) Successful in 41s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 09:08:29 +00:00
70838c7dfe
fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only ( #871 ) ( #873 )
...
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 41s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 08:27:47 +00:00
65b9247369
fix(matching): Tier 2b alias sync + collapse double SAVEPOINT ( #849 ) ( #870 )
...
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / deploy (push) Successful in 46s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 08:05:48 +00:00
10b4b02446
fix(tradein): cian sweep houses-phase — JOIN house_sources + graceful (не fatal) ( #868 )
...
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Has been cancelled
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 07:59:26 +00:00
60353beaaf
feat(tradein): yandex city-sweep — один прогон SERP+address-enrich ( #867 )
...
Deploy Trade-In / deploy (push) Successful in 36s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 29s
Deploy Trade-In / build-backend (push) Successful in 39s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 07:43:13 +00:00
2dd2d8e57e
feat(tradein): schedule yandex address backfill task ( #855 ) ( #866 )
...
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 29s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 35s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 07:31:10 +00:00