Commit graph

422 commits

Author SHA1 Message Date
262adbf762 Merge pull request 'fix(estimator): DKP corridor uses P10/P90 not min/max (#1520)' (#1695) from fix/dkp-corridor-percentile-1520 into main
Some checks are pending
Deploy Trade-In / changes (push) Waiting to run
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
2026-06-17 18:05:39 +00:00
1fffb7a0a7 Merge pull request 'fix(sql): restore partial WHERE is_active predicate on listings indexes (#1398)' (#1693) from fix/listings-partial-indexes-1398 into main
Some checks are pending
Deploy Trade-In / changes (push) Waiting to run
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
2026-06-17 18:05:30 +00:00
5c44233ed6 Merge pull request 'fix(scrapers): cian _walk_price_range stop silently truncating tail (#1393)' (#1689) from fix/cian-paginate-tail-1393 into main
Some checks are pending
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 7s
2026-06-17 17:55:36 +00:00
403020ad56 fix(sql): restore partial predicate on 5 listings indexes (#1398)
All checks were successful
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
097_index_hygiene.sql de-partialized 5 listings indexes on the false premise
that is_active=100% true. 104_index_hygiene_geom_dedup.sql acknowledged the
premise is stale (49% active on prod, 2026-06-13) but only cleaned up the
redundant geom duplicate, leaving 3 composite indexes full.

Restore WHERE is_active=true on the composite hot-path indexes:
- listings_active_filter_idx (rooms, price_rub, area_m2, scraped_at DESC)
- listings_house_price_idx   (house_id_fk, price_rub)
- listings_rooms_area_idx    (rooms, area_m2)

Excluded: listings_geom_active_idx (dropped by 104, listings_geom_idx covers it),
listings_scraped_desc_idx (not in issue #1398 suggested fix scope).

CONCURRENTLY not used — migration runner wraps in BEGIN/COMMIT (see 117 note).
2026-06-17 20:50:51 +03:00
6eda4a6d0a fix(estimator): DKP corridor uses P10/P90 not min/max (#1520)
All checks were successful
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Replace absolute min/max with robust percentiles so a single outlier ДКП
deal cannot shift the soft-bound corridor boundaries. Small samples fall
back gracefully via linear interpolation (n=3: P10≈index 0.2, P90≈index 2.8).
2026-06-17 20:50:15 +03:00
17c494a063 fix(scrapers): cian _walk_price_range stop silently truncating price-range tail (#1393)
All checks were successful
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 20:39:16 +03:00
f8ea119d22 fix(tradein): trade-in PDF cover uses estimate.expires_at, not hardcoded +30d (#1530)
All checks were successful
CI / changes (push) Successful in 10s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-17 20:27:57 +03:00
f8779c0812 feat(yandex): switch SERP scraping to JSON gate-API (bypasses bandwidth tarpit) (#1665)
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 51s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 42s
Deploy Trade-In / changes (push) Successful in 6s
2026-06-17 08:20:46 +00:00
b19c4a0b8c fix(tradein): yandex fetch_around rotate-on-tarpit + retry (combos sweep resilience)
All checks were successful
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
Add _YANDEX_TARPIT_MAX_RETRIES=2 constant and rewrite fetch_around with a
loop: on status_code==0 (curl exit 28, bandwidth tarpit) or captcha detected,
call _rotate_ip() + sleep(2s) and retry. Non-zero non-200 codes (404 etc.)
return [] immediately without rotation. Mirrors the existing exhaustive-path
pattern in _walk_price_range. 4 new unit tests cover: tarpit→rotate→success,
captcha→rotate→success, exhausted retries→empty, 404→no-rotate.
2026-06-17 10:12:24 +03:00
7a88964ef5 fix(avito): scale per-anchor timeout for detail-enrich + durable SERP counters (#1661)
All checks were successful
Deploy Trade-In / test (push) Successful in 36s
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 54s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 45s
2026-06-17 06:07:50 +00:00
4811920d28 fix(tradein): yandex combos watchdog timeout (P0) + deploy scraper filter gap (#1659)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-browser (push) Successful in 28s
Deploy Trade-In / test (push) Successful in 34s
Deploy Trade-In / build-frontend (push) Successful in 1m46s
Deploy Trade-In / deploy (push) Successful in 46s
Deploy Trade-In / build-backend (push) Successful in 53s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-16 20:22:27 +00:00
e7901bc1e8 feat(tradein): yandex city sweep — EKB-center rooms×price combos (replaces 5-anchor shallow) (#1658)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / deploy (push) Successful in 43s
2026-06-16 19:57:37 +00:00
66433dfe1b feat(cian): collect price history via BrowserFetcher (fixes silent no-op backfill) (#1657)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 52s
Deploy Trade-In / deploy (push) Successful in 44s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-06-16 19:47:16 +00:00
b5829fa226 feat(scrapers): enable cian_city_sweep (proxy alive, sweep is cookieless) (#1568)
All checks were successful
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 35s
Deploy Trade-In / build-backend (push) Successful in 26s
Deploy Trade-In / deploy (push) Successful in 42s
2026-06-16 17:08:47 +00:00
e46867fa7f feat(tradein/db): derived price-history trigger (avito/yandex) + last_seen_at deactivate index (#1559)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 27s
Deploy Trade-In / deploy (push) Successful in 41s
2026-06-16 16:48:58 +00:00
91f9a95f93 feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 45s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 56s
2026-06-16 16:09:19 +00:00
6b7391ba93 fix(db): deactivate pre-#753 ghost duplicate listings (167 rows)
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Before #753 dedup_hash included price, so each price change created a new
active row. The canonical rows are now upserted correctly; ghost rows sit
frozen with stale hashes. Deactivates only active non-canonical rows in
groups that already have a canonical twin, excluding all-ghost groups.

Dry-run on prod (2026-06-16): UPDATE 167 rows as expected (yandex ~164,
cian ~1, n1 ~2, avito 0).

Refs #753
2026-06-16 15:40:30 +03:00
c07b5d7c5d chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian
All checks were successful
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 7s
N1.ru перестал быть самостоятельным источником (переехал в Cian, подтверждено
владельцем 2026-06). Активный скрап прекращён; 389 исторических listings в БД
сохраняются без изменений.

Убрано:
- app/services/scrapers/n1.py (удалён)
- run_n1_city_sweep + N1CitySweepCounters + N1_SWEEP_MAX_CONSECUTIVE_FAILURES
  из scrape_pipeline.py
- trigger_n1_city_sweep_run + dispatch elif в scheduler.py
- POST /scrape/n1 + GET /scrape/n1/runs endpoints из admin.py
- N1Scraper из scrape_around dispatch dict и ScrapeRequest.multi_room_n1
- "n1": 5.0 из scraper_settings._DEFAULT_DELAY_BY_SOURCE
- tests/test_n1_city_sweep.py, tests/test_n1_floor_parse.py (удалены)

Добавлено:
- data/sql/114_disable_n1_sweep.sql: UPDATE scrape_schedules SET enabled=false
  WHERE source='n1_city_sweep' (idempotent, строка сохраняется для audit-истории)

Сохранено (историческое):
- schemas/search.py Literal[..."n1"] — API-фильтр исторических listings
- config.py scrape_allowed_hosts n1.ru / ekaterinburg.n1.ru
- exporters/trade_in_pdf.py — цвет/лейбл N1 для отображения старых записей
- geocoder.py и admin.py docstring N1-комментарии где не активный код

1802 passed, 2 deselected (test_search_cache_hit, test_cian_valuation.cache_hit)
2026-06-16 14:50:51 +03:00
b390c8334c fix(tradein): cian detail-парсер не извлекал kitchen_area_m2 из offer.kitchenArea
All checks were successful
CI / changes (push) Successful in 8s
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Добавлено поле kitchen_area_m2 в DetailEnrichment dataclass,
извлечение через _parse_float(offer.get("kitchenArea")) в fetch_detail,
и COALESCE(CAST(:ka AS double precision), kitchen_area_m2) в UPDATE listings
внутри save_detail_enrichment.

Имя поля kitchenArea — стандартное cian offer API (live-verify не делался,
прокси недоступен в dev-окружении).

Closes #1553
2026-06-16 14:37:49 +03:00
fd4a0d643d fix(tradein): review minors — scheduler comment + save-error log label
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- scheduler.py: исправлен stale-комментарий yandex_detail_backfill
  (был: «YandexDetailScraper httpx, no proxy layer»;
   теперь: fetch via curl_cffi chrome120 + scraper_proxy_url,
           parse via YandexDetailScraper.parse)
- yandex_detail_backfill.py: outer except — сообщение лога изменено на
  «save/iteration error for listing_id=%d» чтобы отличать save-ошибку
  от fetch/parse-None путей (run_id убран — он менее важен в этом scope)
2026-06-16 14:14:07 +03:00
e2e0291a0a fix(tradein): yandex detail backfill — заменить fetch_detail на curl_cffi+proxy
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
YandexDetailScraper.fetch_detail использует plain httpx (BaseScraper._http_get)
без прокси и без TLS-fingerprinting → на датацентровом IP Yandex отдаёт капчу /
shell-HTML → parse всегда None → backfill 0% эффективен.

Замена: curl_cffi AsyncSession(chrome120 + scraper_proxy_url), идентично
yandex_address_backfill.py (доказано работающий путь на проде). HTTP 200 →
YandexDetailScraper.parse(html) (чистая функция, без сети). parse→None
считается fail (captcha wall); consecutive None → abort после max_consecutive_blocks.

Тесты обновлены: мокается AsyncSession.get + YandexDetailScraper.parse
(не fetch_detail). Добавлены кейсы: non-200 → abort, proxy=None → proxies=None.
2026-06-16 14:05:22 +03:00
03e43f12b5 feat(tradein): yandex detail backfill — save_detail_enrichment + nightly task (#1553)
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
- yandex_detail.py: добавить save_detail_enrichment(db, listing_id, enrichment) → bool;
  UPDATE listings COALESCE по 17 полям (rooms/area_m2/floor/total_floors/address/
  description/repair_state/publish_date/views_total_yandex/publish_date_relative/
  agency_name/agency_founded_year/agency_objects_count/metro_stations/photo_urls/
  newbuilding_url/newbuilding_id); metro/photos → jsonb; detail_enriched_at = NOW()
- app/tasks/yandex_detail_backfill.py: новый task run_yandex_detail_backfill;
  snapshot SELECT source='yandex' AND detail_enriched_at IS NULL; YandexDetailScraper
  async context manager; consecutive None abort (max_consecutive_blocks=5); budget guard;
  run lifecycle update_heartbeat/mark_done/mark_failed
- scheduler.py: trigger_yandex_detail_backfill_run + elif source=='yandex_detail_backfill'
  в scheduler_loop; window 12-15 UTC (после avito_detail 09-12 UTC)
- migration 113: ADD COLUMN IF NOT EXISTS photo_urls jsonb, newbuilding_url text,
  newbuilding_id text; seed scrape_schedules yandex_detail_backfill enabled=true
- tests: 10 тестов (8 backfill + 2 save_detail_enrichment); 1819 passed 0 failed

Closes #1553
2026-06-16 13:58:20 +03:00
227e82dc01 feat(tradein): avito detail-enrichment nightly backfill (#1551)
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
- New task app/tasks/avito_detail_backfill.py with run_avito_detail_backfill()
  * Single snapshot SELECT at start (guarantees termination)
  * Same proxy/AsyncSession path as scrape_pipeline.py step 5
  * Budget guard (budget_sec), consecutive block abort (mark_done not mark_failed)
  * rotate_ip() on every AvitoBlockedError; rollback on generic Exception (#1368)
  * start = time.monotonic() initialized before try so except can reference it
- Scheduler wiring: trigger_avito_detail_backfill_run() + elif in scheduler_loop()
- Migration 112: scrape_schedules INSERT window 09-12 UTC, batch_size=800,
  budget_sec=3600, request_delay_sec=6, max_consecutive_blocks=5
- 7 unit tests (no pytest-mock, unittest.mock only): all 7 passing,
  full CI suite 1809 passed
2026-06-16 13:25:29 +03:00
b968b811cb Merge pull request 'fix(scrapers): geocode-missing listings nightly scheduler — listings geom coverage backfill (all sources)' (#1549) from fix/geocode-missing-listings-scheduler into main
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 53s
Deploy Trade-In / deploy (push) Successful in 43s
Reviewed-on: #1549
2026-06-16 08:32:22 +00:00
e76a8dcaef fix(tradein): review minors — ceiling_height_m numeric(5,2) + strengthen finishing test
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
- Migration 111: ceiling_height_m numeric → numeric(5,2) (schema consistency with living_area_m2/kitchen_area_m2)
- test_finishing_stored_lowercase_stripped: fixture now uses padded uppercase '  Чистовая  ' to actually exercise .strip().lower(), asserts finishing == 'чистовая'
2026-06-16 10:24:20 +03:00
5285a03c98 fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing
All checks were successful
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200.
Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков"
("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена.

- RUS_FIELD_MAP: добавлены три записи.
- DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing
  (в dataclass attrs и в locals-блоке parse_detail_html).
- parse_detail_html: ветки elif для трёх полей; helper _parse_height_m().
- save_detail_enrichment: COALESCE-строки + params-dict для новых полей.
- migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing
  (living_area_m2 уже существовала, migration не нужна).
- Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m.
  Полный suite: 1795 passed, 2 deselected.
2026-06-16 10:19:09 +03:00
17fa6544a2 fix(geocode): set geocode_tried_at on successful geocode UPDATE
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Add geocode_tried_at = NOW() to the success UPDATE SET clause so
successfully-geocoded listings record their attempt time, matching
the per-ID admin endpoint contract and the column spec.
2026-06-16 10:07:51 +03:00
a56420bdfb fix(tradein): wire nightly geocode backfill для listings в in-app scheduler
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Root cause: 7674 avito/yandex/n1 listings з null-geom мають ненульовий address,
але geocode_tried_at IS NULL — OS cron геокодує тільки deals, ніколи listings.

Parts:
- geocode_missing.py: WHERE додає geocode_tried_at IS NULL OR tried_at < 7d
  (loop-safe); при failure (geo=None / exception) позначаємо tried_at=NOW().
  Новий run_geocode_missing_listings: batch loop з wall-clock budget (30m),
  update_heartbeat / mark_done / mark_failed lifecycle через scrape_runs.
- scheduler.py: trigger_geocode_missing_listings_run + dispatch elif.
- migration 110: seed scrape_schedules row (enabled=true, window 06-09 UTC).
- tests: оновлено 2 тести (commit на tried_at UPDATE); +7 нових тестів.
  1794 passed.
2026-06-16 10:01:49 +03:00
55049046b1 fix(tradein): починить 3 теста-регрессии из #1543 (sber rollback mock, matching tier0, cian price history) — deploy gate green
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-16 09:03:27 +03:00
dca3de10e6 fix(tradein): миграция 108 — typo cl.cluster_key + collision-safe репойнт (dup-vs-dup)
All checks were successful
CI / changes (push) Successful in 9s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Деплой упал на 108: (1) c.cluster_key → cl.cluster_key (несуществующий алиас);
(2) дедуп-перед-репойнтом сравнивал дубль только с каноном, но два разных дубля
одного кластера с одинаковым unique-ключом коллизили при репойнте на канон.
Переписаны 3c/houses_price_dynamics, 3d/house_imv_evaluations, 3e/house_suggestions,
3i/address_mismatch_audit на collision-safe паттерн (дедуп по целевому ключу,
канон-строка выживает). address_mismatch_audit имеет UNIQUE(house_id,audit_batch) —
заголовок ошибочно помечал «нет UNIQUE».

Валидировано dry-run против прод-данных (4478 реальных дублей):
houses 13456 → 8978, clusters_merged=3878, чисто end-to-end.
2026-06-16 08:46:44 +03:00
8d1a13ee5e Merge pull request 'fix(week-review): аудит код-ревью — 181 фиксов (label week ревью 1)' (#1543) from fix/week-review-audit into main
Some checks failed
Deploy Trade-In / changes (push) Successful in 7s
Deploy / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Failing after 36s
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / deploy (push) Has been skipped
Deploy / build-backend (push) Successful in 1m55s
Deploy / build-worker (push) Successful in 2m59s
Deploy / build-frontend (push) Successful in 3m10s
Deploy / deploy (push) Successful in 1m21s
Reviewed-on: #1543
2026-06-16 05:46:11 +00:00
bc504d9519 fix(tradein): M1 review — short_address для match_or_create_house (alias keyspace)
All checks were successful
CI / changes (push) Successful in 8s
CI / changes (pull_request) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-16 08:35:38 +03:00
b72af1cc49 fix(tradein): avito houses dedup через match_or_create_house + merge-миграция дублей (108)
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
Корень проблемы: avito_houses.py обходил канонический matching-путь через
прямой INSERT ON CONFLICT(source, ext_house_id), создавая дубли для физически
одного дома (~33% таблицы houses = ~4479 лишних строк).

Forward fix: upsert_house/upsert_house_with_url → _persist_house(db, h, url),
который вызывает match_or_create_house (3-tier: source_exact→fingerprint→geo→new,
с pg_advisory_xact_lock) и затем UPDATE с enrichment-полями. avito-авторитетные
поля (rating, developer, технические хар-ки) перезаписываются напрямую;
address/lat/lon/year_built — через COALESCE чтобы не затирать canonical-данные.

Merge-миграция 108: cluster_key по cadastral/address/geo → canonical (max
не-NULL полей, geom присутствует, min(id)) → репойнт всех 11 зависимых таблиц
с обработкой UNIQUE-коллизий → DELETE дублей → backfill house_sources/aliases.
2026-06-15 22:40:46 +03:00
c6bfce3d1a fix(migrations): живые миграции для прод-фиксов #1361/#1419/#1364
Some checks failed
CI / changes (push) Successful in 10s
CI / changes (pull_request) Successful in 10s
CI / frontend-tests (push) Successful in 1m22s
CI / openapi-codegen-check (push) Failing after 2m22s
CI / frontend-tests (pull_request) Successful in 1m5s
CI / openapi-codegen-check (pull_request) Failing after 1m45s
CI / backend-tests (push) Failing after 9m8s
CI / backend-tests (pull_request) Failing after 8m54s
Правки уже-применённых миграций (85/062/080) не долетают до прода через
_schema_migrations. Дописаны отдельные миграции на живых таблицах:
- 158: pzz_zones_ekb UNIQUE NULLS NOT DISTINCT (rosreestr_id) (#1361)
- tradein 108: повторный backfill Avito-адресов улучшенным regexp (#1419)
- tradein 109: пересчёт asking_to_sold_ratios secondary-only (#1364/#1186)

Все три dry-run (BEGIN..ROLLBACK) против прод-схемы: PSQL_RC=0.
2026-06-15 21:31:28 +03:00
278dafba76 fix(tradein): yandex newbuilding rating drift — "4.3из 5" без пробела (#974)
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Валидационный yandex_newbuilding_sweep (run 80) заполнял ratings_count /
text_reviews_count / developer_name / lat-lon, но rating оставался NULL для всех
ЖК. Live-диагностика (прод-браузер, ЖК Татлин/Рио 2026-06-15): рейтинг рендерится
как «4.3из 5» / «4,5из 5» — БЕЗ пробела перед «из» (nbsp внутри «из 5»), а
RE_RATING требовал \s+ перед «из» → match=None.

- RE_RATING: (\d[.,]\d)\s+из\s+5 → (\d[.,]\d)\s*из\s+5 (пробел перед «из» опционален).
  \s уже матчит nbsp, поэтому «из\xa05» обрабатывался; ломал именно отсутствующий
  разделитель перед «из».
- test_parse_rating_no_space_before_iz: регрессия на обе дрейфовые формы.

Существующие rating-тесты («4,3 из 5» с пробелом) остаются зелёными (\s* ⊇ \s+).

Refs #974
2026-06-15 21:29:43 +03:00
891598a5d7 fix(tradein): cian newbuilding MFE/reliability drift — initialState extraction (#972)
All checks were successful
CI / changes (push) Successful in 8s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Cian переименовал MFE ЖК-карточки и переместил reliability-контейнер, из-за чего
fetch_newbuilding получал initialState=None → failed_fetch для всех домов
(verified live 2026-06-15 на прод-браузере через мобильный egress):

- MFE: 'newbuilding-card-desktop-fichering-frontend' → 'newbuilding-card-desktop-frontend'
  (старое имя на странице отсутствует → extract_state=None → backfill succeeded=0)
- reliability: initialState.reliability → initialState.reliabilityState.reliability
  (inner dict сохранил форму {checkStatus, details[], banner, actions})

Проверено на ЖК «Светлый» (nb 17556): newbuilding.id, realtyValuation chart (27 точек),
reliabilityState.reliability.checkStatus + 2 details — всё извлекается.

test_fetch_newbuilding_parses_state: fixture обновлён под reliabilityState.reliability.
Полный suite зелёный (1779 passed; 2 несвязанных env-only локальных падения —
cian_valuation cache / search_api auth — в CI проходят).

Refs #972
2026-06-15 21:12:05 +03:00
479f01ae1a Merge remote-tracking branch 'forgejo/main' into HEAD
Some checks failed
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (push) Successful in 1m7s
CI / openapi-codegen-check (push) Failing after 2m9s
CI / frontend-tests (pull_request) Successful in 1m4s
CI / openapi-codegen-check (pull_request) Failing after 1m37s
CI / backend-tests (pull_request) Failing after 9m7s
CI / backend-tests (push) Failing after 9m11s
# Conflicts:
#	backend/app/services/site_finder/gate_verdict.py
2026-06-15 21:07:05 +03:00
483fdb0caa Merge pull request 'feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant (#973)' (#1296) from feat/cian-city-sweep-schedule into main
All checks were successful
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 32s
Deploy Trade-In / build-backend (push) Successful in 56s
Deploy Trade-In / deploy (push) Successful in 42s
Reviewed-on: #1296
2026-06-15 17:48:33 +00:00
1241d7fe4d fix(tradein): переименовать seed-миграцию 101→103 (коллизия с gendesign_reader #976)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-15 19:10:20 +03:00
b0a4df48e1 feat(tradein): scheduler-источник cian_city_sweep (newbuilding Phase 4), seed dormant (#973) 2026-06-15 19:09:26 +03:00
35bc792cf2 fix(tradein): обновить test_scraper_proxy под BrowserFetcher-транспорт fetch_newbuilding — зелёный CI-гейт (#972)
All checks were successful
CI / changes (push) Successful in 6s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-15 19:02:13 +03:00
b3e7a9f9fa Merge pull request 'fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом (#972)' (#1540) from fix/cian-newbuilding-browser-fetch into main
Some checks failed
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Failing after 40s
Deploy Trade-In / deploy (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Reviewed-on: #1540
2026-06-15 15:44:42 +00:00
c9daf8e268 fix(tradein): yandex sweep — guard ext_id=NULL перед fetch_jk + cleanup httpx (#974 review M1/M3)
All checks were successful
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 7s
CI / changes (pull_request) Successful in 8s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
M1: добавлен guard перед вызовом fetch_jk — если ext_id пустой/None при наличии
jk_slug, логируем warning, incrementим failed_fetch и пропускаем (continue).
Пустой jk_id формирует битый URL /{slug}-/ и коллизию на UNIQUE ext_id=''.

M3: в admin.py /scrape/yandex-newbuilding заменён `async with YandexNewbuildingScraper()`
на прямое `scraper = YandexNewbuildingScraper()` — fetch_jk использует внутренний
BrowserFetcher, httpx-клиент BaseScraper открывался вхолостую.

Тесты: добавлен test_sweep_slug_present_but_ext_id_null_skips_fetch.
2026-06-15 18:41:08 +03:00
8279b0ee1f fix(week-review): cross-file добивка код-ревью (#1338, #1363, #1419, #1421, #1424, #1480)
- #1338 concepts.py: geometry.generate вынесен в run_in_threadpool (не блокирует event loop)
- #1363 house_imv_backfill.py + scrape_pipeline.py: периодический heartbeat в долгих IMV-фазах (reap_zombies не помечает живой sweep zombie)
- #1419 site-finder-api.ts: placeholderData keepPreviousData для bbox-запроса (KPI не мигает «0/0» при пане/зуме)
- #1421 site-finder-api.ts: area_ha nullable + адаптер не подставляет 0; null-гейт потребителей EntryMap.tsx, ParcelDrawer.tsx
- #1424/#1480 EstimateResult.tsx: блок «Параметры объекта» скрыт при восстановлении по shared-ссылке (нет мусора «0 м²/Студия/Этаж 0 из 0»)

Верификация: tsc --noEmit 0 ошибок; py_compile OK.
2026-06-15 20:29:00 +05:00
feab585ad5 feat(tradein): yandex_newbuilding sweep-entrypoint + market-схема enrichment, seed dormant (#974)
All checks were successful
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / changes (push) Successful in 7s
- SQL 105: CREATE SCHEMA market + market.yandex_jk_enrichment (ext_id UNIQUE, upsert-ready),
  houses.yandex_jk_slug, GRANT gendesign_reader на market-слой
- SQL 106: seed scrape_schedules yandex_newbuilding_sweep (enabled=false dormant, окно 02-05 UTC)
- yandex_newbuilding.py: fetch_jk → BrowserFetcher (camoufox), add resolve_yandex_jk_slug
  (SERP siteId query через BrowserFetcher, slug из _JK_SLUG_RE href)
- tasks/yandex_newbuilding_sweep.py: enrich_yandex_newbuilding_sweep (resolve slug →
  fetch_jk → UPSERT market.yandex_jk_enrichment, SAVEPOINT per house, dry_run, resumable)
- admin.py: POST /scrape/yandex-newbuilding-sweep + GET .../runs
- scheduler.py: trigger_yandex_newbuilding_sweep_run + dispatch elif branch
- tests: 8 task tests + 3 scheduler dispatch tests (20 pass)
2026-06-15 18:28:58 +03:00
86e9ea2937 fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1»)
Многоагентный аудит + имплементация: один воркер на файл, точечные правки.
Верификация: py_compile (47/47 .py) + tsc --noEmit (0 ошибок). Unit-тесты
не прогонялись (окружение не поднято: rollup native dep / нет pytest-venv).

Полностью исправлено (169): #1336, #1337, #1339, #1340, #1341, #1342, #1343, #1345, #1346, #1348, #1349, #1350, #1351, #1354, #1356, #1358, #1359, #1360, #1362, #1364, #1365, #1366, #1367, #1368, #1369, #1370, #1371, #1372, #1373, #1374, #1375, #1376, #1377, #1378, #1379, #1380, #1381, #1382, #1384, #1385, #1386, #1387, #1388, #1389, #1390, #1391, #1392, #1394, #1395, #1396, #1397, #1399, #1400, #1401, #1402, #1403, #1404, #1408, #1409, #1410, #1411, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1420, #1423, #1425, #1426, #1427, #1428, #1429, #1430, #1431, #1432, #1433, #1434, #1435, #1437, #1438, #1439, #1440, #1441, #1442, #1443, #1444, #1445, #1446, #1447, #1448, #1449, #1450, #1451, #1452, #1453, #1454, #1455, #1456, #1457, #1458, #1459, #1460, #1461, #1462, #1463, #1464, #1465, #1466, #1467, #1468, #1469, #1471, #1472, #1473, #1474, #1476, #1478, #1479, #1481, #1482, #1483, #1484, #1485, #1487, #1488, #1489, #1490, #1491, #1492, #1493, #1494, #1495, #1496, #1497, #1499, #1500, #1501, #1502, #1504, #1505, #1506, #1507, #1510, #1514, #1515, #1516, #1517, #1518, #1519, #1521, #1522, #1523, #1524, #1525, #1526, #1527, #1528, #1529, #1531, #1532, #1533, #1534, #1535, #1536, #1537, #1538

Частично (9, in-file часть, остаток cross-file): #1361, #1419, #1422, #1424, #1470, #1475, #1477, #1480, #1498
Требуют cross-file (3, не тронуты): #1338, #1363, #1421
Пропущено (1): #1539

Не входило в партию: 22 needs-Leha issue (нужны решения владельца).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 20:21:11 +05:00
480fbba3ca fix(tradein): cian fetch_newbuilding через BrowserFetcher (camoufox) — initialState под анти-ботом (#972)
All checks were successful
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
curl_cffi получает zhk-*.cian.ru без initialState; tradein-browser (camoufox) тянет
страницу с 1.17MB initialState. Заменяем транспорт только для page-fetch;
session= param сохранён для backward-compat вызывающих. resolve_cian_zhk_url_via_search
остаётся на curl_cffi SERP. Тесты: 21 passed.
2026-06-15 18:19:32 +03:00
f376d8ce05 fix(tradein-browser): non-fatal launch + background proxy retry (crash-loop 3141x) (#1330)
All checks were successful
Deploy Trade-In / deploy (push) Successful in 1m18s
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / test (push) Has been skipped
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been skipped
Deploy Trade-In / build-browser (push) Successful in 2m2s
2026-06-14 15:18:58 +00:00
4454244ac2 chore(tradein-db): drop duplicate listings_geom_active_idx (#730)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 33s
Deploy Trade-In / build-backend (push) Successful in 24s
Deploy Trade-In / deploy (push) Successful in 39s
После того как 097 снял предикат WHERE is_active с listings_geom_active_idx,
тот стал точным дублем оригинального full listings_geom_idx (002) — оба
GIST(geom) без предиката. listings_geom_idx покрывает нагрузку (active-запросы
добавляют WHERE is_active поверх того же GIST-скана). Освобождает ~3.2 МБ.

Остальное из аудита #730 (5 dup/redundant дропов + 5 is_active де-партиалов)
уже применено миграцией 097 на prod (verify read-only подтвердил: все ABSENT
/ без предиката). Это follow-up на единственный дубль — побочный эффект 097.

Closes #730
2026-06-13 11:53:13 +00:00
6444dc0ebd fix(tradein-frontend): остановить retry-storm на 401 — refetchOnMount:false + backoff (#800)
Some checks failed
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / changes (push) Successful in 7s
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / build-browser (push) Blocked by required conditions
Deploy Trade-In / test (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Has been cancelled
На стойком 401/протухшей сессии фронт повторял /me + /trade-in/quota в
бесконечном цикле (184+ запросов): errored query + дефолтный
refetchOnMount=true → каждый новый observer (useMe дёргают RouteGuard,
Topbar, UserMenu, useBrand) ре-фетчил при mount = re-subscribe storm.
Глобально выключаем refetchOnMount/onReconnect в QueryClient + экспоненциальный
retryDelay (cap 30s); те же гарды на useQuota. retry-предикат 4xx→false уже был.
Graceful session-screen (RouteGuard/NoAccessScreen) уже на месте.

Closes #800
2026-06-13 16:47:58 +05:00