29 commits

Author	SHA1	Message	Date
lekss361	66433dfe1b	feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 19:47:16 +00:00
lekss361	91f9a95f93	feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)	2026-06-16 16:09:19 +00:00
bot-backend	fd4a0d643d	fix(tradein): review minors — scheduler comment + save-error log label ... - scheduler.py: исправлен stale-комментарий yandex_detail_backfill (был: «YandexDetailScraper httpx, no proxy layer»; теперь: fetch via curl_cffi chrome120 + scraper_proxy_url, parse via YandexDetailScraper.parse) - yandex_detail_backfill.py: outer except — сообщение лога изменено на «save/iteration error for listing_id=%d» чтобы отличать save-ошибку от fetch/parse-None путей (run_id убран — он менее важен в этом scope)	2026-06-16 14:14:07 +03:00
bot-backend	e2e0291a0a	fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy ... YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get) без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу / shell-HTML → parse всегда None → backfill 0% эффективен. Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 → YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks. Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse (не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.	2026-06-16 14:05:22 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	227e82dc01	feat(tradein): avito detail-enrichment nightly backfill (#1551) ... - New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill() * Single snapshot SELECT at start (guarantees termination) * Same proxy/AsyncSession path as scrape_pipeline.py step 5 * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed) * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368) * start = time.monotonic() initialized before try so except can reference it - Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop() - Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800, budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5 - 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing, full CI suite 1809 passed	2026-06-16 13:25:29 +03:00
bot-backend	17fa6544a2	fix(geocode): set geocode_tried_at on successful geocode UPDATE ... Add geocode_tried_at = NOW() to the success UPDATE SET clause so successfully-geocoded listings record their attempt time, matching the per-ID admin endpoint contract and the column spec.	2026-06-16 10:07:51 +03:00
bot-backend	a56420bdfb	fix(tradein): wire nightly geocode backfill для listings в in-app scheduler ... Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address, але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings. Parts: - geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d (loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW(). Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m), update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs. - scheduler.py: trigger_geocode_missing_listings_run + dispatch elif. - migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC). - tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів. 1794 passed.	2026-06-16 10:01:49 +03:00
bot-backend	c9daf8e268	fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx (#974 review M1/M3) ... M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue). Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''. M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()` на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую. Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.	2026-06-15 18:41:08 +03:00
bot-backend	feab585ad5	feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974) ... - SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready), houses.yandex_jk_slug, GRANT gendesign_reader на market-слой - SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC) - yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href) - tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug → fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable) - admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs - scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch - tests: 8 task tests + 3 scheduler dispatch tests (20 pass)	2026-06-15 18:28:58 +03:00
Light1YT	e1442d3b1d	feat(tradein-scheduler): nightly newbuilding-enrichment schedule (#973) ... Регистрирует отдельный in-app scheduler-источник для cian_newbuilding enrichment-backfill (houses_price_dynamics / house_reliability_checks / house_reviews), который раньше бежал только инлайн в Cian full-sweep. - run_newbuilding_enrich() wrapper (heartbeat → backfill → mark_done/failed), делегирует backfill_newbuilding_enrichment (#972, уже в main). - trigger_newbuilding_enrich_run() + dispatch-ветка в scheduler_loop (зеркалит trigger_sber_index_pull_run): _claim_run guard, async task, zombie-reap наследуется; idempotency наследуется (skip уже обогащённых, per-house SAVEPOINT, bounded per-fire limit дренирует backlog 306 домов). - seed-миграция 103: source='newbuilding_enrich', окно 00:00-01:00 UTC (03:00-04:00 МСК, до 01:00+ UTC sweep-блока), limit=25, next_run_at на завтра. Засеяно enabled=FALSE (dormant): на prod весь external-HTTP scraping намеренно на паузе. Расписание+триггер готовы, но не стартуют до намеренного возобновления (UPDATE ... SET enabled=true). Не активирую одиночный источник при общей паузе. Closes #973	2026-06-13 16:24:03 +05:00
bot-backend	b70ae46885	fix(tradein): guard listing_segment во всех читателях listings — novostroyki вне comps (#1186) ... Канон-предикат: (listing_segment IS NULL OR listing_segment = 'vtorichka') NULL = legacy вторичка до миграции 011 (sweep'ы были secondary-only) — отбрасывать нельзя. Затронуты все SELECT-пути, влияющие на оценку и корректирующие коэффициенты: - estimator.py: _COMMON_WHERE (Tier S/H), Tier W inline, Tier A anchor (добавлен впервые), Tier C anchor (заменён с параметрического CAST(:segment) на канон — прежний вариант отбрасывал NULL-строки при segment='vtorichka'). - asking_to_sold_ratio.py: ask_side, ask_global (_REDERIVE_SQL); bounds (_REDERIVE_BOUNDS_SQL); ask_tiered, ask_side_all, ask_global (_REDERIVE_TIERED_SQL) — 6 мест. - sber_index.py: reconciliation-запрос (диагностический asking-median). Не затронуты: скрейперы/scrape_pipeline, search matview (catalog display, не оценка), admin stats, geocoding, cian price history, matching — обосновано в инвентаризации. Тесты: новый test_segment_guard_1186.py (статика SQL + поведение mock); обновлён test_asking_to_sold_ratio.py (subset-тест нормализует guard, добавленный в refresh но отсутствующий в 080-seed).	2026-06-12 11:52:09 +03:00
Light1YT	b0fe292a63	fix(tradein): working cian ЖК-url resolver via cat.php SERP (#972) ... The legacy resolve_cian_zhk_url hit /zhk/<id>/ which now 404s, leaving the 318 geo-matched cian houses (house_sources.ext_id set, cian_zhk_url NULL) unfetchable -> the newbuilding-enrichment backfill matched 0 houses. Add resolve_cian_zhk_url_via_search(nb_id): fetch the cat.php newbuilding SERP and extract the canonical zhk-<slug>.cian.ru url, ANCHORED on the <h1 data-name="Title"> header anchor (NOT naive first-match — the SERP carries promo/recommendation zhk-* links before the title that would otherwise resolve the wrong ЖК and silently corrupt enrichment). Validated against the real 2.81MB prod SERP + an adversarial poisoned-recommendation test. Wire into newbuilding_enrich_backfill: broaden selection to "has ext_id OR cian_zhk_url", resolve+persist the url under a SAVEPOINT before enriching, rate-limited + resumable + idempotent. Keep the old resolver (deprecated). Bounded prod proof (5 houses, direct): 5/5 urls resolved+persisted, 3/5 fully enriched (+18 price_dynamics, +3 reliability); the 2 misses were direct-mode anti-bot on the 2nd fetch. Full 318-run gated on the cian mobile proxy (mproxy.site) being restored. code-reviewer APPROVE (SQL/idempotency) + resolver hardened against wrong-ЖК. 19 tests green. Refs #972.	2026-06-08 11:13:06 +05:00
Light1YT	bcb1285341	feat(tradein): backfill task for cian newbuilding enrichment (#972) ... 950-E1: app/tasks/newbuilding_enrich_backfill.py selects houses linked to ext_source='cian_newbuilding' with a fetchable cian_zhk_url and runs the existing fetch_newbuilding + save_newbuilding_enrichment over each, populating houses_price_dynamics + house_reliability_checks (+ house_reviews when present). - Idempotent + resumable: skips already-enriched houses; force= re-run dup-safe (price_dynamics UPSERT via dim_key, reliability dedup guard with id-DESC tiebreaker, reviews ON CONFLICT (source,ext_review_id)). - Anti-bot aware: get_scraper_delay('cian') + jitter, SAVEPOINT-per-house (one captcha/failure logs + continues, never aborts the batch). - limit= for bounded proof runs; sizing counters (total/fetchable/pending). Fix cian_newbuilding._extract_transport_rate: cian transportAccessibilityRate drifted to a nested dict and broke save_newbuilding_enrichment's CAST bind on LIVE data (cannot adapt type 'dict') — coerce dict/float/bad -> int|None (bool excluded before int). Fixes the live enrichment crash, not just backfill. house_reviews stays ~0 for now: cian reviews are in a separate XHR, not the ЖК initialState — extending fetch_newbuilding for them is a documented follow-up. Tested: 8 new unit tests; isolated-DB proof-run landed 6 price_dynamics + 1 reliability row, idempotency proven across re-runs. Refs #972.	2026-06-07 14:09:24 +00:00
lekss361	571980c56e	feat(tradein): ppm²-tier segmentation of asking→sold ratio (#928, flag OFF) (#934) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-05-31 21:22:39 +00:00
bot-backend	c2888ac52d	feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 13:13:55 +00:00
bot-backend	de92c54540	feat(tradein): СберИндекс monthly city-level price index pull (#887, data-layer) (#890) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 12:44:56 +00:00
bot-backend	2dd2d8e57e	feat(tradein): schedule yandex address backfill task (#855) (#866) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:31:10 +00:00
bot-backend	42007e9b04	feat(tradein): nightly task to deactivate stale avito listings (#759) (#862) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-31 07:19:37 +00:00
bot-backend	b736d7b7e0	feat(estimator): exclude city-centroid listings from radius analogs (Refs #769 Part E) ... Закрывает последнюю часть #769 (A1/A2/B/C/D в #798/#804). Finding #17: bare-city адреса геокодились в city-центроид и сохранялись как точные lat/lon → mislocated листинги участвовали в radius ST_DWithin-аналогах. - 089: ALTER TABLE listings ADD COLUMN IF NOT EXISTS geo_precision text (idempot). - ScrapedLot.geo_precision + проброс в save_listings. - geocode_missing.py: geo_precision='city' через существующий _geocode_is_coarse. - estimator _fetch_analogs Tier H+W: + AND (geo_precision IS DISTINCT FROM 'city'). NULL проходит (консервативно, сдвига до backfill нет). Pricing не тронут. 7 тестов; 270 passed, ruff clean.	2026-05-30 21:22:21 +03:00
bot-backend	3653f020b6	fix(ratio): ppm² потолок asking→sold в конфиг, default 1.2M (Refs #767) (#790) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-05-30 16:57:09 +00:00
Light1YT	a060651328	feat(tradein): daily asking→sold ratio refresh (#648 S4) (#663)	2026-05-29 13:49:36 +00:00
Light1YT	e925a0ba6a	feat(tradein): per-source listing price history — schema + daily snapshot (#570) (#646)	2026-05-29 12:18:04 +00:00
Light1YT	1731704ddc	feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627)	2026-05-28 13:57:43 +00:00
Light1YT	1ce9e10012	feat(tradein): persistent infra для backfill scripts (PR F) (#592)	2026-05-27 06:59:54 +00:00
lekss361	ba9cc3aef1	feat(tradein/cian): valuation backfill + runner script для prod (#539)	2026-05-24 15:16:45 +00:00
lekss361	bc594e3f79	feat(tradein/cian): batch backfill task для offer_price_history + houses_price_dynamics (#535)	2026-05-24 14:55:47 +00:00
lekss361	52716a46c6	fix(tradein): geocode backfill + remove Avito exclusion from estimator (#490)	2026-05-23 20:03:48 +00:00
lekss361	e46b7d778e	feat(tradein): search matview + indexes (Phase 3.1) (#469) ... Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings. SQL 050_search_optimization.sql: - CREATE EXTENSION IF NOT EXISTS pg_trgm - ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED - 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...) - CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources) - 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY) Celery task tradein.refresh_search_matview: - REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3) - Logged duration via logger.info - Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet) - Beat schedule snippet in docstring for future wiring Fixups (deep-code-reviewer 2026-05-23): - h.ratings_count → h.reviews_count (alias house_ratings_count preserved) - Strip +psycopg dialect prefix from DSN for libpq compat Refs Master Plan sec 5.1 + 6.5 + 10.1.	2026-05-23 14:01:18 +00:00