4 commits

Author	SHA1	Message	Date
bot-backend	eb564cfc01	feat(tradein/scraper-kit): migrate backfill-task imports to kit, Group C (#2310) ... Migrates legacy app.services.scrapers.* imports to scraper_kit equivalents for house_imv_backfill.py, avito_detail_backfill.py, cian_history_backfill.py, ekb_geoportal_ingest.py, and yandex_detail_backfill.py, proving parity via tests/support/parity.assert_parity per the epic's gate (#2304). newbuilding_enrich_backfill.py and yandex_newbuilding_sweep.py are left fully on legacy imports: both call into scraper_kit.providers.{cian,yandex}.newbuilding, which construct BrowserFetcher(source=...) without the now-mandatory endpoint= kwarg (issue #2322, verified still open against the actual provider source, not just issue status) -- no caller-side fix is possible, matching Group A's (#2305) precedent for the same bug in admin.py. Config-gated kit function footguns found and fixed (Group B #2306 pattern): - avito fetch_detail's backconnect-on-403 retry silently drops when config= is omitted -- now passes config=RealScraperConfig() explicitly, with a regression test proving the gate. - kit AvitoScraper's constructor now requires ScraperConfig positionally -- wired via RealScraperConfig(), which _rotate_ip() reads for avito_proxy_rotate_url. - BrowserFetcher(source=...) call sites (house_imv_backfill, avito/cian detail backfills) now pass the mandatory endpoint=settings.browser_http_endpoint. New footgun discovered (NOT #2322, flagged for follow-up): kit's build_warmed_session() builds its curl_cffi session via _build_detail_session() with no config parameter at all, unlike fetch_detail -- migrating it would silently drop the sticky MGTS-proxy egress on avito_detail_backfill's warm-batch path (the prod default). Left build_warmed_session/ _AVITO_WARM_SEARCH_URL on legacy imports, documented inline. Refs #2310	2026-07-04 01:19:34 +03:00
Light1YT	e1442d3b1d	feat(tradein-scheduler): nightly newbuilding-enrichment schedule (#973) ... Регистрирует отдельный in-app scheduler-источник для cian_newbuilding enrichment-backfill (houses_price_dynamics / house_reliability_checks / house_reviews), который раньше бежал только инлайн в Cian full-sweep. - run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed), делегирует backfill_newbuilding_enrichment (#972, уже в main). - trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop (зеркалит trigger_sber_index_pull_run): _claim_run guard, async task, zombie-reap наследуется; idempotency наследуется (skip уже обогащённых, per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов). - seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC (03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра. Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления (UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе. Closes #973	2026-06-13 16:24:03 +05:00
Light1YT	b0fe292a63	fix(tradein): working cian ЖК-url resolver via cat.php SERP (#972) ... The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the 318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL) unfetchable -> the newbuilding-enrichment backfill matched 0 houses. Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the <h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries promo/recommendation zhk-* links before the title that would otherwise resolve the wrong ЖК and silently corrupt enrichment). Validated against the real 2.81MB prod SERP + an adversarial poisoned-recommendation test. Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching, rate-limited + resumable + idempotent. Keep the old resolver (deprecated). Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy (mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) + resolver hardened against wrong-ЖК. 19 tests green. Refs #972.	2026-06-08 11:13:06 +05:00
Light1YT	bcb1285341	feat(tradein): backfill task for cian newbuilding enrichment (#972) ... 950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing fetch_newbuilding + save_newbuilding_enrichment over each, populating houses_price_dynamics + house_reliability_checks (+ house_reviews when present). - Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe (price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC tiebreaker, reviews ON CONFLICT (source,ext_review_id)). - Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house (one captcha/failure logs + continues, never aborts the batch). - limit= for bounded proof runs; sizing counters (total/fetchable/pending). Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None (bool excluded before int). Fixes the live enrichment crash, not just backfill. house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК initialState — extending fetch_newbuilding for them is a documented follow-up. Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1 reliability row, idempotency proven across re-runs. Refs #972.	2026-06-07 14:09:24 +00:00