4 commits

Author	SHA1	Message	Date
bot-backend	566b2f9617	fix(tradein/scraper): drain detached run-children on SIGTERM, not just coordinator (#1182 Phase 2) ... Pre-push review: _await_scheduler ждал только scheduler_loop COORDINATOR, но вся scrape-работа крутится в detached asyncio.create_task детях (каждый trigger_* делал `task = create_task(_run())` без join). На SIGTERM coordinator выходил из while True и завершался → asyncio.run() teardown хард-кансельил ещё бегущих детей mid-await = ровно #1182 failure mode. Кооперативный checkpoint спасал ребёнка лишь когда его residual-время случайно перекрывало drain — вероятностно, не гарантированно. Fix — coordinator теперь дренажит детей перед выходом: - _spawn_tracked(coro): централизованный detached-spawn, кладёт задачу в module-level registry _inflight_tasks (strong-ref = RUF006 keep-alive) + done-callback ретривит exception и убирает из set'а. Заменил 26 одинаковых `task = create_task(_run()); task.add_done_callback(...)` сайтов. - _drain_inflight(): один asyncio.wait по живым детям с бюджетом _CHILD_DRAIN_TIMEOUT_S=80s (< scheduler_main 100s < docker grace 120s). Кооперативные дети (avito_detail_backfill, rosreestr-executor) дочекивают карточку/батч + mark_done и резолвятся; некооперативные упираются в timeout и падают на внешний hard-cancel. - scheduler_loop по выходу из tick-loop (только по SIGTERM) зовёт await _drain_inflight(). NB: raw asyncio.all_tasks()-minus-self здесь НЕЛЬЗЯ — в нашей топологии он захватывает _run parent-task (блокирован на wait_for(coordinator)) и shutdown_waiter → циклическое ожидание coordinator↔_run, всегда упирающееся в timeout. Точный registry это исключает. Tests: tests/test_scheduler.py — detached cooperative child drained-not-cancelled, non-cooperative child timeout→left for hard-cancel, no-op без детей, scheduler_loop→drain wiring. Обновил 3 source-inspection теста под новый _spawn_tracked паттерн.	2026-06-28 20:13:53 +03:00
Light1YT	e1442d3b1d	feat(tradein-scheduler): nightly newbuilding-enrichment schedule (#973) ... Регистрирует отдельный in-app scheduler-источник для cian_newbuilding enrichment-backfill (houses_price_dynamics / house_reliability_checks / house_reviews), который раньше бежал только инлайн в Cian full-sweep. - run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed), делегирует backfill_newbuilding_enrichment (#972, уже в main). - trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop (зеркалит trigger_sber_index_pull_run): _claim_run guard, async task, zombie-reap наследуется; idempotency наследуется (skip уже обогащённых, per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов). - seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC (03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра. Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления (UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе. Closes #973	2026-06-13 16:24:03 +05:00
Light1YT	b0fe292a63	fix(tradein): working cian ЖК-url resolver via cat.php SERP (#972) ... The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the 318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL) unfetchable -> the newbuilding-enrichment backfill matched 0 houses. Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the <h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries promo/recommendation zhk-* links before the title that would otherwise resolve the wrong ЖК and silently corrupt enrichment). Validated against the real 2.81MB prod SERP + an adversarial poisoned-recommendation test. Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching, rate-limited + resumable + idempotent. Keep the old resolver (deprecated). Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy (mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) + resolver hardened against wrong-ЖК. 19 tests green. Refs #972.	2026-06-08 11:13:06 +05:00
Light1YT	bcb1285341	feat(tradein): backfill task for cian newbuilding enrichment (#972) ... 950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing fetch_newbuilding + save_newbuilding_enrichment over each, populating houses_price_dynamics + house_reliability_checks (+ house_reviews when present). - Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe (price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC tiebreaker, reviews ON CONFLICT (source,ext_review_id)). - Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house (one captcha/failure logs + continues, never aborts the batch). - limit= for bounded proof runs; sizing counters (total/fetchable/pending). Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None (bool excluded before int). Fixes the live enrichment crash, not just backfill. house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК initialState — extending fetch_newbuilding for them is a documented follow-up. Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1 reliability row, idempotency proven across re-runs. Refs #972.	2026-06-07 14:09:24 +00:00