1277 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
bot-backend	afa7d80ecf	fix(forecasting): deal_count confidence note carries «за N мес» window (#1637) ... Add `deal_count_months: int | None = None` to `compute_report_confidence`. When provided, threads it as suffix into `_factor_from_count` so the deal_count ConfidenceFactor note reads «7 сделок за 6 мес — мало» instead of the windowless «7 сделок — мало». Existing callers unaffected (default None). Tests: two new cases in TestComputeReportConfidence — with/without window.	2026-06-17 20:38:00 +03:00
lekss361	18da92ccc7	Merge pull request 'fix(week-review): дожим — 4 фикса (#1569, #1590, #1642, #801)' (#1662) from fix/week-review-finish into main	2026-06-17 17:23:10 +00:00
bot-backend	20efc3bf64	merge: resolve conflicts with main (#1660) into week-review-finish ... 5 conflicts resolved: - redaction.py: merged INN checksum + _PHONE_LOCAL_RE + _Repl alias from main - recommendation.py: kept commercial_sell_through_pct rename (#1635) - report_pdf.py: kept _scenario_deficit_cell + int|None return type (#1590) - parcel.py: kept main's detailed docstring (functionally identical) - alembic/env.py: kept HEAD (app.models import in __init__ already covers all)	2026-06-17 20:21:06 +03:00
bot-backend	14f3ef2019	fix(week-review): backend-аудит v2 — 82 фиксов (#1660) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-17 17:13:38 +00:00
lekss361	50db2b82aa	Merge pull request 'fix(site-finder): nearest_top3 metro from osm_poi_ekb (#1667)' (#1670) from fix/1667-metro-block into main	2026-06-17 17:13:17 +00:00
lekss361	e4b80938a0	Merge pull request 'docs(site-finder): analyze confidence by-design distinct from §15 (#1668)' (#1671) from fix/1668-analyze-confidence into main	2026-06-17 17:13:07 +00:00
lekss361	184e429526	Merge pull request 'fix(workers): schedule mv_layout_velocity refresh (#1666)' (#1669) from fix/1666-layout-velocity-beat into main	2026-06-17 17:12:54 +00:00
bot-backend	829f046852	feat(scripts): Potrace floor-plan vectorization spike (#299) (#1663) ... Co-authored-by: bot-backend <bot-backend@gendsgn.local> Co-committed-by: bot-backend <bot-backend@gendsgn.local>	2026-06-17 17:12:48 +00:00
Light1YT	d3a53ae475	docs(site-finder): clarify analyze confidence is by-design distinct from §15 (#1668) ... _compute_confidence docstring claimed "stub до G1/G2/D1/D2" — те эшелоны уже отгружены, формулировка устарела. Это намеренно отдельная метрика: надёжность site-finder скоринга (coverage POI/район/рынок), НЕ форсайтный §15 confidence_engine (надёжность прогноза спроса/цены над форсайт-входами, которых в analyze hot-path нет). Поведение не меняется — резолв документацией. Closes #1668	2026-06-17 18:47:07 +05:00
Light1YT	7072d7803e	fix(site-finder): compute nearest_top3 metro from osm_poi_ekb (#1667) ... Replace bare metro_block placeholder ({"nearest_top3": None}, blocked on a never-merged 22h metro scraper) with a direct KNN query against osm_poi_ekb category='metro_stop' (loaded by poi_loader). Returns 3 nearest stations with name + rounded distance_m; empty list when none nearby (vs None on error), SAVEPOINT + try/except like adjacent SF-B5 blocks. No frontend consumer yet. Closes #1667	2026-06-17 18:44:29 +05:00
Light1YT	7276edc885	fix(workers): schedule mv_layout_velocity refresh in Celery beat (#1666) ... mv_layout_velocity (powers best_layouts, #113) refreshed only via manual refresh_layout_velocity() → prod data silently went stale. Add Celery task mirroring refresh_quarter_price_index, register in celery_app include, add weekly beat entry (Sun 03:00 MSK, outside the monday heavy-refresh cluster). Closes #1666	2026-06-17 18:40:01 +05:00
Light1YT	764db61734	fix(exporters): annotate scenario deficit horizon in docx/pptx (#1590 follow-up) ... report_docx и report_pptx рендерили bare _scenario_deficit_index под захардкоженной шапкой «(12 мес)» — при fallback на другой горизонт подпись лгала. Переключил оба на _scenario_deficit_cell (как в report_md): per-cell аннотация «(гор. N мес)» при N != 12. Закрывает code-review major по #1590.	2026-06-17 14:35:48 +05:00
Light1YT	c9ba595abe	refactor(forecasting): rename commercial_share_pct → commercial_sell_through_pct (#1635) ... Внутренний recommendation→product_scoring контракт-ключ был мислейблом: величина — темп распродажи нежилого (sell-through, прокси ликвидности/спроса), а НЕ доля нежилого в объёме застройки. Переименован ключ + исправлены reason/docstring/комментарии у потребителя _score_commercial. Числовая логика не изменена. Ключ внутренний (нет frontend/schema/openapi-потребителей) → rename контракт-безопасен. pytest 171 passed. Closes #1635	2026-06-17 13:32:03 +05:00
lekss361	f8779c0812	feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)	2026-06-17 08:20:46 +00:00
lekss361	edaadc7685	Merge pull request 'fix(yandex): rotate proxy IP + retry on tarpit-timeout/captcha in combos sweep' (#1664) from fix/yandex-rotate-on-tarpit into main ... Reviewed-on: #1664	2026-06-17 07:13:48 +00:00
bot-backend	b19c4a0b8c	fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience) ... Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected, call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.) return [] immediately without rotation. Mirrors the existing exhaustive-path pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success, captcha→rotate→success, exhausted retries→empty, 404→no-rotate.	2026-06-17 10:12:24 +03:00
Light1YT	249182b678	fix(week-review): дожим ревью — #1569, #1590, #1642 + feat #801 MSW-preview ... Финиш-волна по cross-file/partial остаткам аудита + a11y-харнесс. Полностью (4): #1569 alembic регистрирует все ORM-модели (+убран фантомный Parcel) · #1590 «Индекс дефицита» больше не лжёт на fallback-горизонте (report_pdf+report_md) · #1642 thumbs mtime-freshness (нет устаревшей миниатюры) · #801 /__preview/estimate mock-render для axe/lighthouse (env-gated bypass). Частично (остаток cross-file/домен, открыты): #1593 #1635 #1640. needs-leha (открыт): #1650 (нужна новая миграция вьюхи supply-слоёв). Verify: tsc --noEmit 0; py_compile OK. Closes #1569 Closes #1590 Closes #1642 Closes #801	2026-06-17 11:24:34 +05:00
lekss361	7a88964ef5	fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters (#1661)	2026-06-17 06:07:50 +00:00
lekss361	4811920d28	fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap (#1659) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 20:22:27 +00:00
lekss361	e7901bc1e8	feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) (#1658)	2026-06-16 19:57:37 +00:00
lekss361	66433dfe1b	feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657) ... Co-authored-by: lekss361 <lekss361@gendsgn.local> Co-committed-by: lekss361 <lekss361@gendsgn.local>	2026-06-16 19:47:16 +00:00
lekss361	b5829fa226	feat(scrapers): enable cian_city_sweep (proxy alive, sweep is cookieless) (#1568)	2026-06-16 17:08:47 +00:00
lekss361	e46867fa7f	feat(tradein/db): derived price-history trigger (avito/yandex) + last_seen_at deactivate index (#1559)	2026-06-16 16:48:58 +00:00
lekss361	91f9a95f93	feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)	2026-06-16 16:09:19 +00:00
lekss361	496af12c9c	Merge pull request 'fix(scrapers): deactivate pre-#753 ghost duplicate listings (dedup correctness)' (#1557) from fix/deactivate-ghost-duplicate-listings into main ... Reviewed-on: #1557	2026-06-16 12:59:46 +00:00
bot-backend	6b7391ba93	fix(db): deactivate pre-#753 ghost duplicate listings (167 rows) ... Before #753 dedup_hash included price, so each price change created a new active row. The canonical rows are now upserted correctly; ghost rows sit frozen with stale hashes. Deactivates only active non-canonical rows in groups that already have a canonical twin, excluding all-ghost groups. Dry-run on prod (2026-06-16): UPDATE 167 rows as expected (yandex ~164, cian ~1, n1 ~2, avito 0). Refs #753	2026-06-16 15:40:30 +03:00
bot-backend	a200025571	Merge pull request 'chore(scrapers): убрать устаревший n1-скраппер (n1.ru → cian), сохранить историч. данные' (#1556) from chore/remove-n1-scraper into main	2026-06-16 11:58:02 +00:00
bot-backend	c07b5d7c5d	chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian ... N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД сохраняются без изменений. Убрано: - app/services/scrapers/n1.py (удалён) - run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES из scrape_pipeline.py - trigger_n1_city_sweep_run + dispatch elif в scheduler.py - POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py - N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1 - "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE - tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены) Добавлено: - data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории) Сохранено (историческое): - schemas/search.py Literal[..."n1"] — API-фильтр исторических listings - config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru - exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей - geocoder.py и admin.py docstring N1-комментарии где не активный код 1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)	2026-06-16 14:50:51 +03:00
bot-backend	f160ad3859	Merge pull request 'fix(scrapers): cian detail — извлечение kitchen_area_m2 (part of #1553)' (#1555) from fix/cian-kitchen-area into main	2026-06-16 11:42:35 +00:00
bot-backend	b390c8334c	fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea ... Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass, извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail, и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings внутри save_detail_enrichment. Имя поля kitchenArea — стандартное cian offer API (live-verify не делался, прокси недоступен в dev-окружении). Closes #1553	2026-06-16 14:37:49 +03:00
bot-backend	d76df54ece	Merge pull request 'feat(scrapers): yandex detail-enrichment — save + ночной backfill через прокси (part of #1553)' (#1554) from feat/yandex-detail-backfill into main	2026-06-16 11:18:18 +00:00
bot-backend	fd4a0d643d	fix(tradein): review minors — scheduler comment + save-error log label ... - scheduler.py: исправлен stale-комментарий yandex_detail_backfill (был: «YandexDetailScraper httpx, no proxy layer»; теперь: fetch via curl_cffi chrome120 + scraper_proxy_url, parse via YandexDetailScraper.parse) - yandex_detail_backfill.py: outer except — сообщение лога изменено на «save/iteration error for listing_id=%d» чтобы отличать save-ошибку от fetch/parse-None путей (run_id убран — он менее важен в этом scope)	2026-06-16 14:14:07 +03:00
bot-backend	e2e0291a0a	fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy ... YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get) без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу / shell-HTML → parse всегда None → backfill 0% эффективен. Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 → YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks. Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse (не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.	2026-06-16 14:05:22 +03:00
bot-backend	03e43f12b5	feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553) ... - yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool; UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/ description/repair_state/publish_date/views_total_yandex/publish_date_relative/ agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/ newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW() - app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill; snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard; run lifecycle update_heartbeat/mark_done/mark_failed - scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill' в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC) - migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text, newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true - tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed Closes #1553	2026-06-16 13:58:20 +03:00
bot-backend	1fc375c2cc	Merge pull request 'feat(scrapers): ночной avito detail-enrichment backfill для legacy listings (Closes #1551)' (#1552) from feat/avito-detail-backfill into main	2026-06-16 10:32:57 +00:00
bot-backend	227e82dc01	feat(tradein): avito detail-enrichment nightly backfill (#1551) ... - New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill() * Single snapshot SELECT at start (guarantees termination) * Same proxy/AsyncSession path as scrape_pipeline.py step 5 * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed) * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368) * start = time.monotonic() initialized before try so except can reference it - Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop() - Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800, budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5 - 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing, full CI suite 1809 passed	2026-06-16 13:25:29 +03:00
lekss361	b968b811cb	Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' (#1549) from fix/geocode-missing-listings-scheduler into main ... Reviewed-on: #1549	2026-06-16 08:32:22 +00:00
lekss361	6747ce941b	Merge pull request 'fix(scrapers): avito detail parser — захват Жилая площадь / Высота потолков / Отделка' (#1550) from fix/avito-detail-apartment-fields into main ... Reviewed-on: #1550	2026-06-16 08:32:14 +00:00
bot-backend	e76a8dcaef	fix(tradein): review minors — ceiling_height_m numeric(5,2) + strengthen finishing test ... - Migration 111: ceiling_height_m numeric → numeric(5,2) (schema consistency with living_area_m2/kitchen_area_m2) - test_finishing_stored_lowercase_stripped: fixture now uses padded uppercase ' Чистовая ' to actually exercise .strip().lower(), asserts finishing == 'чистовая'	2026-06-16 10:24:20 +03:00
bot-backend	5285a03c98	fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing ... STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200. Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков" ("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена. - RUS_FIELD_MAP: добавлены три записи. - DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing (в dataclass attrs и в locals-блоке parse_detail_html). - parse_detail_html: ветки elif для трёх полей; helper _parse_height_m(). - save_detail_enrichment: COALESCE-строки + params-dict для новых полей. - migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing (living_area_m2 уже существовала, migration не нужна). - Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m. Полный suite: 1795 passed, 2 deselected.	2026-06-16 10:19:09 +03:00
bot-backend	17fa6544a2	fix(geocode): set geocode_tried_at on successful geocode UPDATE ... Add geocode_tried_at = NOW() to the success UPDATE SET clause so successfully-geocoded listings record their attempt time, matching the per-ID admin endpoint contract and the column spec.	2026-06-16 10:07:51 +03:00
bot-backend	a56420bdfb	fix(tradein): wire nightly geocode backfill для listings в in-app scheduler ... Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address, але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings. Parts: - geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d (loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW(). Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m), update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs. - scheduler.py: trigger_geocode_missing_listings_run + dispatch elif. - migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC). - tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів. 1794 passed.	2026-06-16 10:01:49 +03:00
lekss361	aab07845f6	Merge pull request 'fix(tradein): починить 3 теста-регрессии из #1543 — разблокировать deploy gate' (#1548) from fix/week-review-test-regressions into main ... Reviewed-on: #1548	2026-06-16 06:08:12 +00:00
bot-backend	55049046b1	fix(tradein): починить 3 теста-регрессии из #1543 (sber rollback mock, matching tier0, cian price history) — deploy gate green	2026-06-16 09:03:27 +03:00
lekss361	7d39aebbd4	Merge pull request 'fix(tradein): hotfix миграции 108 — deploy red (typo + collision-safe репойнт)' (#1547) from fix/migration-108-collision-safe into main ... Reviewed-on: #1547	2026-06-16 05:48:38 +00:00
bot-backend	dca3de10e6	fix(tradein): миграция 108 — typo cl.cluster_key + collision-safe репойнт (dup-vs-dup) ... Деплой упал на 108: (1) c.cluster_key → cl.cluster_key (несуществующий алиас); (2) дедуп-перед-репойнтом сравнивал дубль только с каноном, но два разных дубля одного кластера с одинаковым unique-ключом коллизили при репойнте на канон. Переписаны 3c/houses_price_dynamics, 3d/house_imv_evaluations, 3e/house_suggestions, 3i/address_mismatch_audit на collision-safe паттерн (дедуп по целевому ключу, канон-строка выживает). address_mismatch_audit имеет UNIQUE(house_id,audit_batch) — заголовок ошибочно помечал «нет UNIQUE». Валидировано dry-run против прод-данных (4478 реальных дублей): houses 13456 → 8978, clusters_merged=3878, чисто end-to-end.	2026-06-16 08:46:44 +03:00
lekss361	8d1a13ee5e	Merge pull request 'fix(week-review): аудит код-ревью — 181 фиксов (label week ревью 1)' (#1543) from fix/week-review-audit into main ... Reviewed-on: #1543	2026-06-16 05:46:11 +00:00
lekss361	91c2f6a6a8	Merge pull request 'fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108)' (#1546) from fix/avito-houses-dedup-merge into main ... Reviewed-on: #1546	2026-06-16 05:38:23 +00:00
bot-backend	bc504d9519	fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace)	2026-06-16 08:35:38 +03:00
bot-backend	be70cacf18	test: выровнять weather_cache/nspd_geo тесты под #1377/#1336 (CI red→green) ... backend-tests CI упал на 4 стейл-тестах — код автофиксов обновили, тесты нет: - test_weather_cache: фикстура _make_air_response отдавала hourly-форму, а код (#1377) теперь читает current-bucket → current.time пуст → None. Фикстура → current. - test_nspd_geo: 2 теста стерегли ошибочный ST_Transform(3857→4326), который #1336 убрал (геометрия уже WGS84, иначе Null Island). Ассерты через _strip_sql_comments (как сиблинг parcel-тест) → ST_SetSRID(...,4326) без ST_Transform. Переименованы. Локально: 38 passed (оба файла).	2026-06-16 08:35:14 +03:00