03e43f12b5
feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task ( #1553 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
- yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool;
UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/
description/repair_state/publish_date/views_total_yandex/publish_date_relative/
agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/
newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW()
- app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill;
snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper
async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard;
run lifecycle update_heartbeat/mark_done/mark_failed
- scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill'
в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC)
- migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text,
newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true
- tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed
Closes #1553
2026-06-16 13:58:20 +03:00
1fc375c2cc
Merge pull request 'feat(scrapers): ночной avito detail-enrichment backfill для legacy listings ( Closes #1551 )' ( #1552 ) from feat/avito-detail-backfill into main
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 34s
Deploy Trade-In / build-backend (push) Successful in 49s
Deploy Trade-In / deploy (push) Successful in 42s
2026-06-16 10:32:57 +00:00
227e82dc01
feat(tradein): avito detail-enrichment nightly backfill ( #1551 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
* Single snapshot SELECT at start (guarantees termination)
* Same proxy/AsyncSession path as scrape_pipeline.py step 5
* Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
* rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368 )
* start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
full CI suite 1809 passed
2026-06-16 13:25:29 +03:00
b968b811cb
Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' ( #1549 ) from fix/geocode-missing-listings-scheduler into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 53s
Deploy Trade-In / deploy (push) Successful in 43s
Reviewed-on: #1549
2026-06-16 08:32:22 +00:00
6747ce941b
Merge pull request 'fix(scrapers): avito detail parser — захват Жилая площадь / Высота потолков / Отделка' ( #1550 ) from fix/avito-detail-apartment-fields into main
...
Deploy Trade-In / changes (push) Has been cancelled
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Reviewed-on: #1550
2026-06-16 08:32:14 +00:00
e76a8dcaef
fix(tradein): review minors — ceiling_height_m numeric(5,2) + strengthen finishing test
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
- Migration 111: ceiling_height_m numeric → numeric(5,2) (schema consistency with living_area_m2/kitchen_area_m2)
- test_finishing_stored_lowercase_stripped: fixture now uses padded uppercase ' Чистовая ' to actually exercise .strip().lower(), asserts finishing == 'чистовая'
2026-06-16 10:24:20 +03:00
5285a03c98
fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing
...
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200.
Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков"
("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена.
- RUS_FIELD_MAP: добавлены три записи.
- DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing
(в dataclass attrs и в locals-блоке parse_detail_html).
- parse_detail_html: ветки elif для трёх полей; helper _parse_height_m().
- save_detail_enrichment: COALESCE-строки + params-dict для новых полей.
- migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing
(living_area_m2 уже существовала, migration не нужна).
- Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m.
Полный suite: 1795 passed, 2 deselected.
2026-06-16 10:19:09 +03:00
17fa6544a2
fix(geocode): set geocode_tried_at on successful geocode UPDATE
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Add geocode_tried_at = NOW() to the success UPDATE SET clause so
successfully-geocoded listings record their attempt time, matching
the per-ID admin endpoint contract and the column spec.
2026-06-16 10:07:51 +03:00
a56420bdfb
fix(tradein): wire nightly geocode backfill для listings в in-app scheduler
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address,
але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings.
Parts:
- geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d
(loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW().
Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m),
update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs.
- scheduler.py: trigger_geocode_missing_listings_run + dispatch elif.
- migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC).
- tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів.
1794 passed.
2026-06-16 10:01:49 +03:00
aab07845f6
Merge pull request 'fix(tradein): починить 3 теста-регрессии из #1543 — разблокировать deploy gate' ( #1548 ) from fix/week-review-test-regressions into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 44s
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 52s
Reviewed-on: #1548
2026-06-16 06:08:12 +00:00
55049046b1
fix(tradein): починить 3 теста-регрессии из #1543 (sber rollback mock, matching tier0, cian price history) — deploy gate green
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-16 09:03:27 +03:00
7d39aebbd4
Merge pull request 'fix(tradein): hotfix миграции 108 — deploy red (typo + collision-safe репойнт)' ( #1547 ) from fix/migration-108-collision-safe into main
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Failing after 35s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Reviewed-on: #1547
2026-06-16 05:48:38 +00:00
dca3de10e6
fix(tradein): миграция 108 — typo cl.cluster_key + collision-safe репойнт (dup-vs-dup)
...
CI / changes (push) Successful in 9s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Деплой упал на 108: (1) c.cluster_key → cl.cluster_key (несуществующий алиас);
(2) дедуп-перед-репойнтом сравнивал дубль только с каноном, но два разных дубля
одного кластера с одинаковым unique-ключом коллизили при репойнте на канон.
Переписаны 3c/houses_price_dynamics, 3d/house_imv_evaluations, 3e/house_suggestions,
3i/address_mismatch_audit на collision-safe паттерн (дедуп по целевому ключу,
канон-строка выживает). address_mismatch_audit имеет UNIQUE(house_id,audit_batch) —
заголовок ошибочно помечал «нет UNIQUE».
Валидировано dry-run против прод-данных (4478 реальных дублей):
houses 13456 → 8978, clusters_merged=3878, чисто end-to-end.
2026-06-16 08:46:44 +03:00
8d1a13ee5e
Merge pull request 'fix(week-review): аудит код-ревью — 181 фиксов (label week ревью 1)' ( #1543 ) from fix/week-review-audit into main
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Failing after 36s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Deploy / build-backend (push) Successful in 1m55s
Deploy / build-worker (push) Successful in 2m59s
Deploy / build-frontend (push) Successful in 3m10s
Deploy / deploy (push) Successful in 1m21s
Reviewed-on: #1543
2026-06-16 05:46:11 +00:00
91c2f6a6a8
Merge pull request 'fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108)' ( #1546 ) from fix/avito-houses-dedup-merge into main
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / test (push) Successful in 35s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 51s
Deploy Trade-In / deploy (push) Failing after 33s
Reviewed-on: #1546
2026-06-16 05:38:23 +00:00
bc504d9519
fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace)
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-16 08:35:38 +03:00
be70cacf18
test: выровнять weather_cache/nspd_geo тесты под #1377/#1336 (CI red→green)
...
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 9s
CI / frontend-tests (push) Successful in 1m4s
CI / frontend-tests (pull_request) Successful in 57s
CI / openapi-codegen-check (push) Successful in 2m7s
CI / openapi-codegen-check (pull_request) Successful in 1m41s
CI / backend-tests (push) Successful in 8m59s
CI / backend-tests (pull_request) Successful in 8m58s
backend-tests CI упал на 4 стейл-тестах — код автофиксов обновили, тесты нет:
- test_weather_cache: фикстура _make_air_response отдавала hourly-форму, а код
(#1377 ) теперь читает current-bucket → current.time пуст → None. Фикстура → current.
- test_nspd_geo: 2 теста стерегли ошибочный ST_Transform(3857→4326), который #1336
убрал (геометрия уже WGS84, иначе Null Island). Ассерты через _strip_sql_comments
(как сиблинг parcel-тест) → ST_SetSRID(...,4326) без ST_Transform. Переименованы.
Локально: 38 passed (оба файла).
2026-06-16 08:35:14 +03:00
b72af1cc49
fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108)
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Корень проблемы: avito_houses.py обходил канонический matching-путь через
прямой INSERT ON CONFLICT(source, ext_house_id), создавая дубли для физически
одного дома (~33% таблицы houses = ~4479 лишних строк).
Forward fix: upsert_house/upsert_house_with_url → _persist_house(db, h, url),
который вызывает match_or_create_house (3-tier: source_exact→fingerprint→geo→new,
с pg_advisory_xact_lock) и затем UPDATE с enrichment-полями. avito-авторитетные
поля (rating, developer, технические хар-ки) перезаписываются напрямую;
address/lat/lon/year_built — через COALESCE чтобы не затирать canonical-данные.
Merge-миграция 108: cluster_key по cadastral/address/geo → canonical (max
не-NULL полей, geom присутствует, min(id)) → репойнт всех 11 зависимых таблиц
с обработкой UNIQUE-коллизий → DELETE дублей → backfill house_sources/aliases.
2026-06-15 22:40:46 +03:00
edfdd04923
fix(frontend): regen api-types.ts — rate_ms в BulkGeoEnqueueRequest
...
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (push) Successful in 59s
CI / frontend-tests (pull_request) Successful in 55s
CI / openapi-codegen-check (push) Successful in 2m5s
CI / openapi-codegen-check (pull_request) Successful in 1m32s
CI / backend-tests (push) Failing after 8m53s
CI / backend-tests (pull_request) Failing after 8m50s
Бэкенд-схема BulkGeoEnqueueRequest получила поле rate_ms (int, default 600),
но api-types.ts не был перегенерён → CI openapi-codegen-check (drift-gate) RED.
Добавлено сгенерированное поле (проверено: openapi-typescript 7.13.0 + prettier
defaults даёт байт-идентичный результат).
2026-06-15 22:32:21 +03:00
c6bfce3d1a
fix(migrations): живые миграции для прод-фиксов #1361/#1419/#1364
...
CI / changes (push) Successful in 10s
CI / changes (pull_request) Successful in 10s
CI / frontend-tests (push) Successful in 1m22s
CI / openapi-codegen-check (push) Failing after 2m22s
CI / frontend-tests (pull_request) Successful in 1m5s
CI / openapi-codegen-check (pull_request) Failing after 1m45s
CI / backend-tests (push) Failing after 9m8s
CI / backend-tests (pull_request) Failing after 8m54s
Правки уже-применённых миграций (85/062/080) не долетают до прода через
_schema_migrations. Дописаны отдельные миграции на живых таблицах:
- 158: pzz_zones_ekb UNIQUE NULLS NOT DISTINCT (rosreestr_id) (#1361 )
- tradein 108: повторный backfill Avito-адресов улучшенным regexp (#1419 )
- tradein 109: пересчёт asking_to_sold_ratios secondary-only (#1364/#1186)
Все три dry-run (BEGIN..ROLLBACK) против прод-схемы: PSQL_RC=0.
2026-06-15 21:31:28 +03:00
291c000792
Merge pull request 'fix(tradein): yandex newbuilding rating drift — «4.3из 5» без пробела ( #974 )' ( #1545 ) from fix/yandex-nb-rating-drift into main
...
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 35s
Deploy Trade-In / build-backend (push) Successful in 1m12s
Deploy Trade-In / deploy (push) Successful in 46s
Reviewed-on: #1545
2026-06-15 18:31:12 +00:00
278dafba76
fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела ( #974 )
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count /
text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех
ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится
как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а
RE_RATING требовал \s+ перед «из» → match=None.
- RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален).
\s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий
разделитель перед «из».
- test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы.
Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+).
Refs #974
2026-06-15 21:29:43 +03:00
0f95a15c8e
Merge pull request 'fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction ( #972 )' ( #1544 ) from fix/cian-newbuilding-mfe-drift into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 35s
Deploy Trade-In / build-backend (push) Successful in 1m1s
Deploy Trade-In / deploy (push) Successful in 46s
Reviewed-on: #1544
2026-06-15 18:13:45 +00:00
891598a5d7
fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction ( #972 )
...
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Cian переименовал MFE ЖК-карточки и переместил reliability-контейнер, из-за чего
fetch_newbuilding получал initialState=None → failed_fetch для всех домов
(verified live 2026-06-15 на прод-браузере через мобильный egress):
- MFE: 'newbuilding-card-desktop-fichering-frontend' → 'newbuilding-card-desktop-frontend'
(старое имя на странице отсутствует → extract_state=None → backfill succeeded=0)
- reliability: initialState.reliability → initialState.reliabilityState.reliability
(inner dict сохранил форму {checkStatus, details[], banner, actions})
Проверено на ЖК «Светлый» (nb 17556): newbuilding.id, realtyValuation chart (27 точек),
reliabilityState.reliability.checkStatus + 2 details — всё извлекается.
test_fetch_newbuilding_parses_state: fixture обновлён под reliabilityState.reliability.
Полный suite зелёный (1779 passed; 2 несвязанных env-only локальных падения —
cian_valuation cache / search_api auth — в CI проходят).
Refs #972
2026-06-15 21:12:05 +03:00
479f01ae1a
Merge remote-tracking branch 'forgejo/main' into HEAD
...
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (push) Successful in 1m7s
CI / openapi-codegen-check (push) Failing after 2m9s
CI / frontend-tests (pull_request) Successful in 1m4s
CI / openapi-codegen-check (pull_request) Failing after 1m37s
CI / backend-tests (pull_request) Failing after 9m7s
CI / backend-tests (push) Failing after 9m11s
# Conflicts:
# backend/app/services/site_finder/gate_verdict.py
2026-06-15 21:07:05 +03:00
483fdb0caa
Merge pull request 'feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant ( #973 )' ( #1296 ) from feat/cian-city-sweep-schedule into main
...
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 42s
Reviewed-on: #1296
2026-06-15 17:48:33 +00:00
a77a175f66
Merge pull request 'fix(tradein): обновить test_scraper_proxy под BrowserFetcher-транспорт fetch_newbuilding' ( #1542 ) from fix/cian-newbuilding-proxy-tests into main
...
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Has been cancelled
Reviewed-on: #1542
2026-06-15 17:48:12 +00:00
7fd69e90cd
fix(site-finder/compare): thread velocityDataAvailable into compare columns ( #1422 )
CI / changes (push) Successful in 7s
CI / frontend-tests (push) Successful in 1m4s
CI / changes (pull_request) Successful in 15s
CI / openapi-codegen-check (push) Failing after 2m9s
CI / frontend-tests (pull_request) Successful in 1m12s
CI / openapi-codegen-check (pull_request) Failing after 1m46s
CI / backend-tests (push) Failing after 9m1s
CI / backend-tests (pull_request) Failing after 9m2s
2026-06-15 21:33:10 +05:00
1241d7fe4d
fix(tradein): переименовать seed-миграцию 101→103 (коллизия с gendesign_reader #976 )
CI / changes (pull_request) Successful in 7s
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-15 19:10:20 +03:00
b0a4df48e1
feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant ( #973 )
2026-06-15 19:09:26 +03:00
35bc792cf2
fix(tradein): обновить test_scraper_proxy под BrowserFetcher-транспорт fetch_newbuilding — зелёный CI-гейт ( #972 )
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-15 19:02:13 +03:00
b3e7a9f9fa
Merge pull request 'fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )' ( #1540 ) from fix/cian-newbuilding-browser-fetch into main
...
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 40s
Deploy Trade-In / deploy (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Reviewed-on: #1540
2026-06-15 15:44:42 +00:00
3859624d1d
Merge pull request 'feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant ( #974 )' ( #1541 ) from feat/yandex-newbuilding-sweep into main
...
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Has been cancelled
Reviewed-on: #1541
2026-06-15 15:44:27 +00:00
c9daf8e268
fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx ( #974 review M1/M3)
...
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии
jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue).
Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''.
M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()`
на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний
BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую.
Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.
2026-06-15 18:41:08 +03:00
8279b0ee1f
fix(week-review): cross-file добивка код-ревью ( #1338 , #1363 , #1419 , #1421 , #1424 , #1480 )
...
- #1338 concepts.py: geometry.generate вынесен в run_in_threadpool (не блокирует event loop)
- #1363 house_imv_backfill.py + scrape_pipeline.py: периодический heartbeat в долгих IMV-фазах (reap_zombies не помечает живой sweep zombie)
- #1419 site-finder-api.ts: placeholderData keepPreviousData для bbox-запроса (KPI не мигает «0/0» при пане/зуме)
- #1421 site-finder-api.ts: area_ha nullable + адаптер не подставляет 0; null-гейт потребителей EntryMap.tsx, ParcelDrawer.tsx
- #1424/#1480 EstimateResult.tsx: блок «Параметры объекта» скрыт при восстановлении по shared-ссылке (нет мусора «0 м²/Студия/Этаж 0 из 0»)
Верификация: tsc --noEmit 0 ошибок; py_compile OK.
2026-06-15 20:29:00 +05:00
feab585ad5
feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant ( #974 )
...
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 7s
- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
(SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)
2026-06-15 18:28:58 +03:00
86e9ea2937
fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1»)
...
Многоагентный аудит + имплементация: один воркер на файл, точечные правки.
Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты
не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv).
Полностью исправлено (169): #1336 , #1337 , #1339 , #1340 , #1341 , #1342 , #1343 , #1345 , #1346 , #1348 , #1349 , #1350 , #1351 , #1354 , #1356 , #1358 , #1359 , #1360 , #1362 , #1364 , #1365 , #1366 , #1367 , #1368 , #1369 , #1370 , #1371 , #1372 , #1373 , #1374 , #1375 , #1376 , #1377 , #1378 , #1379 , #1380 , #1381 , #1382 , #1384 , #1385 , #1386 , #1387 , #1388 , #1389 , #1390 , #1391 , #1392 , #1394 , #1395 , #1396 , #1397 , #1399 , #1400 , #1401 , #1402 , #1403 , #1404 , #1408 , #1409 , #1410 , #1411 , #1412 , #1413 , #1414 , #1415 , #1416 , #1417 , #1418 , #1420 , #1423 , #1425 , #1426 , #1427 , #1428 , #1429 , #1430 , #1431 , #1432 , #1433 , #1434 , #1435 , #1437 , #1438 , #1439 , #1440 , #1441 , #1442 , #1443 , #1444 , #1445 , #1446 , #1447 , #1448 , #1449 , #1450 , #1451 , #1452 , #1453 , #1454 , #1455 , #1456 , #1457 , #1458 , #1459 , #1460 , #1461 , #1462 , #1463 , #1464 , #1465 , #1466 , #1467 , #1468 , #1469 , #1471 , #1472 , #1473 , #1474 , #1476 , #1478 , #1479 , #1481 , #1482 , #1483 , #1484 , #1485 , #1487 , #1488 , #1489 , #1490 , #1491 , #1492 , #1493 , #1494 , #1495 , #1496 , #1497 , #1499 , #1500 , #1501 , #1502 , #1504 , #1505 , #1506 , #1507 , #1510 , #1514 , #1515 , #1516 , #1517 , #1518 , #1519 , #1521 , #1522 , #1523 , #1524 , #1525 , #1526 , #1527 , #1528 , #1529 , #1531 , #1532 , #1533 , #1534 , #1535 , #1536 , #1537 , #1538
Частично (9, in-file часть, остаток cross-file): #1361 , #1419 , #1422 , #1424 , #1470 , #1475 , #1477 , #1480 , #1498
Требуют cross-file (3, не тронуты): #1338 , #1363 , #1421
Пропущено (1): #1539
Не входило в партию: 22 needs-Leha issue (нужны решения владельца).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:21:11 +05:00
480fbba3ca
fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом ( #972 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
2026-06-15 18:19:32 +03:00
cab19020a9
perf(site-finder): pushdown obj_pricing CTE to nearby competitors (#70/#307) ( #1335 )
Deploy / changes (push) Successful in 5s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m40s
Deploy / build-worker (push) Successful in 2m38s
Deploy / deploy (push) Successful in 1m12s
2026-06-14 16:58:43 +00:00
863577beb6
feat(sql): corroborated trgm backfill objective_complex_mapping +17 ( #307 OBJ-2 r2) ( #1333 )
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 27s
Deploy / build-worker (push) Successful in 27s
Deploy / deploy (push) Successful in 1m4s
2026-06-14 16:45:55 +00:00
5bf30570a5
perf(sql): geography GIST + covering index для analyze-пути Site Finder (157) ( #1334 )
Deploy / deploy (push) Blocked by required conditions
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Has been cancelled
Deploy / build-worker (push) Has been cancelled
2026-06-14 16:45:22 +00:00
b405ae291d
feat(site-finder): additive objective_lots price fallback for competitors ( #307 OBJ-3) ( #1332 )
Deploy / changes (push) Successful in 7s
Deploy / build-backend (push) Successful in 1m57s
Deploy / build-worker (push) Successful in 2m51s
Deploy / build-frontend (push) Successful in 3m6s
Deploy / deploy (push) Successful in 1m12s
2026-06-14 16:10:02 +00:00
f9dd903f4d
feat(sql): conservative trgm backfill objective_complex_mapping +23 ( #307 OBJ-2) ( #1331 )
Deploy / build-backend (push) Successful in 29s
Deploy / build-worker (push) Successful in 29s
Deploy / deploy (push) Successful in 1m1s
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
2026-06-14 15:51:38 +00:00
f376d8ce05
fix(tradein-browser): non-fatal launch + background proxy retry (crash-loop 3141x) ( #1330 )
Deploy Trade-In / deploy (push) Successful in 1m18s
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / test (push) Has been skipped
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / build-browser (push) Successful in 2m2s
2026-06-14 15:18:58 +00:00
72e0b050f9
feat(site-finder): POI saturation per capita по районам ЕКБ ( #42 ) ( #1329 )
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m38s
Deploy / build-worker (push) Successful in 2m52s
Deploy / deploy (push) Successful in 1m12s
2026-06-14 15:05:22 +00:00
e5e0ccb283
feat(site-finder): сетевые обременения из ЗОУИТ ( #1070 , Refs #103 ) ( #1328 )
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m36s
Deploy / build-worker (push) Successful in 2m40s
Deploy / deploy (push) Successful in 1m12s
2026-06-14 14:49:32 +00:00
5507bbc1a4
feat(site-finder): domrf→cad geom-match + parking_ratio конкурентов ( #96 ) ( #1327 )
Deploy / build-frontend (push) Blocked by required conditions
Deploy / deploy (push) Blocked by required conditions
Deploy / build-backend (push) Blocked by required conditions
Deploy / build-worker (push) Blocked by required conditions
Deploy / changes (push) Has been cancelled
2026-06-14 14:49:28 +00:00
2c22c3f7ea
feat(site-finder): per-building помещения/машино-места + parking_ratio (on-demand MVP, #96 )
...
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 8s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (push) Successful in 1m59s
CI / openapi-codegen-check (pull_request) Successful in 1m58s
CI / backend-tests (push) Successful in 8m48s
CI / backend-tests (pull_request) Successful in 8m48s
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 4m58s
Deploy / build-worker (push) Successful in 5m46s
Deploy / deploy (push) Successful in 1m30s
Оживляет мёртвый foundation list_objects_in_building (#168 Q3-deferred). Fix реального
блокера: NSPDOptions не парсил objdocId (camelCase из NSPD search) → objdoc_id всегда None,
метод нельзя было вызвать. AliasChoices(objdoc_id, objdocId) + registers.
premises_lookup.get_building_premises(cad_num) — резолв objdoc → list_objects_in_building,
graceful (WAF/сеть/not-found → None). parking_ratio = машино-места/помещения (None при 0
помещений, 0.0 при реальном отсутствии паркинга). Verified live 66:41:0106036:183.
MVP on-demand (без bulk/схемы/prod-записей). parking_ratio готов, но НЕ wired в analyze:
конкуренты приходят из ДОМ.РФ без cad_num → нужен domrf↔cad_buildings geom-match (отдельная
задача). Сервис примет cad_num как только matching появится. 11 тестов, ruff clean.
Refs #96
2026-06-14 18:48:16 +05:00
11aa011d11
perf(db): covering-index objective_lots district-price ( #70 ) + 152-ФЗ/perf audit docs ( #70 #79 )
...
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (push) Successful in 2m3s
CI / openapi-codegen-check (pull_request) Successful in 2m7s
CI / backend-tests (push) Successful in 8m55s
CI / backend-tests (pull_request) Successful in 8m58s
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 29s
Deploy / build-worker (push) Successful in 29s
Deploy / deploy (push) Successful in 1m10s
#70 : EXPLAIN top-5 hot-path на prod — analyze district-price block по objective_lots
(1.12M) делал Bitmap Heap Scan ~41k блоков (cold 5.3s/warm 474ms), пробивая p95 ≤3с.
Partial covering-index (district, price_per_m2_rub) WHERE price_per_m2_rub IS NOT NULL →
Index Only Scan (verified scratch: 87ms, Heap Fetches 0). Требует VACUUM ANALYZE
objective_lots post-deploy (stale VM иначе глушит выигрыш). database-expert ✅ verified.
#79 : 152-ФЗ PII grep-аудит — plain-PII в логах НЕ найдено; Sentry scrub / PRINZIP-хеширование
/ LLM §19-redaction подтверждены. Остаточный риск: pilot_requests хранит сырые ПДн без
retention (рекомендация в docs). httpx DEBUG off (ops-инвариант).
Closes #70
Closes #79
2026-06-14 00:09:09 +05:00
d030b04cf2
docs(infra): secrets rotation policy + procedure ( #78 )
...
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Compliance-doc: реестр классов секретов (CI/CD + prod runtime), процедуры ротации
per-class с downtime-эффектом, периодичность, quarterly audit-чеклист. OIDC verdict:
future (Forgejo 10.0.3 без Actions OIDC token provider; deploy = прямой SSH, не cloud
federation). Флагает sshkey.txt (PR #391 ) достижимый в git-истории — deploy-key
скомпрометирован, нужна ротация. Значения — в vault meta/00_credentials.md (не в репо).
Closes #78
2026-06-13 23:48:14 +05:00