Commit graph

16 commits

Author SHA1 Message Date
0202720096 feat(tradein): in-app scheduler — UI-managed schedule (replaces SSH crontab) (#482)
Some checks failed
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 36s
Deploy Trade-In / deploy (push) Has been cancelled
2026-05-23 14:54:25 +00:00
2914ff2ef2 feat(tradein): city sweep — auto ЕКБ pipeline (#477)
All checks were successful
Deploy Trade-In / build-frontend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-backend (push) Successful in 38s
Stage 4d: full city sweep ЕКБ (5 anchors × N pages + enrichment + cooperative cancel).

- avito.py fetch_around(pages=N) — pagination backward compat default=1
- scrape_pipeline.run_avito_city_sweep — anchors loop + heartbeat + cancel check
- scrape_runs.py utility (create/heartbeat/done/failed/cancelled/list_recent)
- admin.py 3 endpoints: start (BackgroundTasks) / runs / cancel
- migration 051 ADD COLUMN IF NOT EXISTS + DROP/ADD CHECK constraint (text+CHECK, не enum — safe в transaction)
- deploy/avito-city-sweep.sh с random delay 0-3h (anti-pattern)
- 16 offline tests

Deep review approved: migration idempotent, cancel priority correct (mark_done guarded by status='running'), no SQL injection, psycopg v3 compliant.
2026-05-23 14:38:07 +00:00
e46b7d778e feat(tradein): search matview + indexes (Phase 3.1) (#469)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 22s
Deploy Trade-In / build-backend (push) Successful in 43s
Multi-Source Integration Phase 3.1. Foundation for /api/v1/search p95 < 100ms on 100K+ listings.

SQL 050_search_optimization.sql:
- CREATE EXTENSION IF NOT EXISTS pg_trgm
- ALTER listings ADD COLUMN tsv tsvector GENERATED ALWAYS AS (...) STORED
- 10 indexes (geom GIST, composite, gin_trgm, GIN tsv, partial kadastr, ...)
- CREATE MATERIALIZED VIEW listings_search_mv (per-listing aggregated across sources)
- 5 matview indexes (UNIQUE on listing_id for REFRESH CONCURRENTLY)

Celery task tradein.refresh_search_matview:
- REFRESH MATERIALIZED VIEW CONCURRENTLY listings_search_mv (psycopg v3)
- Logged duration via logger.info
- Graceful try/except ImportError fallback (no celery_app.py in tradein-mvp yet)
- Beat schedule snippet in docstring for future wiring

Fixups (deep-code-reviewer 2026-05-23):
- h.ratings_count → h.reviews_count (alias house_ratings_count preserved)
- Strip +psycopg dialect prefix from DSN for libpq compat

Refs Master Plan sec 5.1 + 6.5 + 10.1.
2026-05-23 14:01:18 +00:00
5e90797a9f feat(tradein): SQL 040-046 — matching schema delta (Phase 1.1) (#464)
Some checks failed
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Has been cancelled
2026-05-23 13:30:22 +00:00
0db71033b4 fix(tradein): 030 — UNIQUE constraint on avito_imv_evaluations.cache_key (unblocks PR #452) (#454)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 20s
Deploy Trade-In / build-backend (push) Successful in 21s
Hotfix unblocking PR #452 (Stage 3 IMV estimator integration).

Adds 030_avito_imv_cache_key_unique.sql: creates UNIQUE INDEX imv_cache_key_uniq_idx
on avito_imv_evaluations(cache_key), allowing save_imv_evaluation's ON CONFLICT (cache_key)
DO UPDATE to resolve. Migration 018 only created a non-unique composite index, which would
have caused runtime error on first IMV cache write.

DELETE-USING dedup guards against historical duplicates (no-op on fresh table).
BEGIN/COMMIT atomic; IF NOT EXISTS idempotent; CONCURRENTLY unnecessary (table empty).

Lexicographic apply order with PR #453's 030_listings_alter_yandex.sql is safe — different
tables, no functional interaction.

Reviewed by deep-code-reviewer: APPROVE.
2026-05-23 13:04:21 +00:00
f452b132c9 feat(tradein): SQL migrations 030-032 — Yandex schema (listings/houses/history) (#453)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 19s
2026-05-23 13:00:54 +00:00
7fad898c7f feat(tradein): SQL migration 029 — extend matching/valuation/dynamics tables (PR #445 follow-up) (#448)
All checks were successful
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 19s
Deploy Trade-In / build-backend (push) Successful in 38s
2026-05-23 12:35:03 +00:00
lekss361
1ad2565af1 feat(tradein): SQL migrations 019-028 — Cian schema + matching + cookies
- 019: ALTER listings (windows_view_type, wcs counts, ceiling_height,
       repair_type, views, living_area, bedrooms, balconies, loggias,
       description_minhash, cadastral_number, phones, bargain_allowed,
       sale_type, house_source/ext_id, metro_stations)
- 020: ALTER houses (BTI: series_name, entrances, flat_count, is_emergency,
       heat/gas/overlap types; Cian: cian_internal_house_id,
       management_company_id, transport_accessibility_rate,
       advantages/banks/builders/houses_by_turn jsonb + GIN indexes)
- 021: CREATE management_companies + FK to houses.management_company_id
- 022: CREATE agents table + agent_id_fk on listings
- 023: CREATE offer_price_history (per-listing price change timeline)
- 024: CREATE houses_price_dynamics (monthly price/sqm series)
- 025: CREATE house_reliability_checks (наш.дом.рф checks)
- 026: CREATE external_valuations (unified Avito IMV + Cian Valuation cache)
- 027: CREATE EXTENSION pgcrypto + cian_session_cookies (encrypted)
- 028: CREATE house_sources, listing_sources, house_address_aliases
       + address_fingerprint/canonical/merged_into alters

Refs: decisions/CianScraper_v1_Implementation_Plan.md Stage 1
      decisions/Schema_Cian_SERP_Inventory.md sec 16, 20.3, 26
      decisions/Cross_Source_Matching_Strategy.md sec 2
2026-05-23 15:14:49 +03:00
a2147872ed feat(tradein): SQL migrations 009-018 — houses + listings extension + IMV cache (#440)
All checks were successful
Deploy Trade-In / build-backend (push) Successful in 23s
Deploy Trade-In / deploy (push) Successful in 27s
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
2026-05-23 11:41:44 +00:00
TradeIn Deploy
48ccbdafc8 feat(tradein): CRM-поля в форме оценки (#395)
Доп. поля для трейд-ин менеджера: тип собственности, ипотека/
обременение, имя и телефон клиента. На расчёт оценки не влияют —
сохраняются в trade_in_estimates вместе с записью.

- data/sql/008_crm_fields.sql — 4 колонки в trade_in_estimates.
- TradeInEstimateInput + estimator INSERT — приём и сохранение.
- EstimateForm — секция «CRM — для менеджера» (4 поля).

Closes #395
2026-05-22 11:15:15 +05:00
TradeIn Deploy
5413cce953 feat(tradein): backend загрузки фото квартиры (#394)
Фото квартиры прикрепляются к оценке, хранятся в Postgres (bytea) —
без отдельного файлового тома, уезжают вместе с бэкапом БД.

- data/sql/007_estimate_photos.sql — таблица estimate_photos
  (FK на trade_in_estimates, ON DELETE CASCADE).
- POST /estimate/{id}/photos — multipart-загрузка (image/*, ≤10 МБ,
  ≤12 фото на оценку).
- GET  /estimate/{id}/photos — список метаданных.
- GET  /estimate/{id}/photos/{photo_id} — отдать содержимое.
- python-multipart добавлен в зависимости (FastAPI UploadFile).

Фронтенд (file-drop в форме) — отдельным PR.
2026-05-22 11:05:35 +05:00
TradeIn Deploy
9cc8ab96f7 feat(tradein): кадастр зданий как источник house_metadata (#393)
В gendesign-БД есть cad_buildings — ~36к зданий ЕКБ с годом постройки
и этажностью (Росреестр). Подключаем как ПРИОРИТЕТНЫЙ источник
обогащения house_metadata, OSM Overpass остаётся fallback-ом.

- data/sql/006_cad_buildings.sql — таблица cad_buildings в tradein-БД.
- deploy/import-cadastre.sh — ETL: ЕКБ-срез cad_buildings из
  gendesign-postgres → tradein (по образцу import-rosreestr.sh).
- house_metadata._cad_buildings_get — lookup по гео-близости (60 м),
  проверяется ПЕРВЫМ в get_house_metadata, до OSM.

cad_buildings — кадастр ЗДАНИЙ, не квартир: поле ручного ввода кадастра
(исходная идея #393) не делаем — гео-обогащение полезнее и работает
автоматически для каждой оценки.

Closes #393
2026-05-22 10:54:42 +05:00
TradeIn Deploy
4ea35d5740 feat(tradein): geocode-missing для deals (#4 завершение)
geocode-missing получил target=listings|deals. deals.geocode_tried_at
добавлен в миграцию 005. cron геокодит и сделки Росреестра по улицам.
2026-05-21 20:04:19 +03:00
TradeIn Deploy
609744df91 fix(tradein): чанковый геокодинг — cron больше не таймаутится (#1)
geocode_tried_at: после каждой попытки NOW(), повтор только через 7 дней.
Эндпоинт чанкуется (limit=100), отдаёт remaining. cron вызывает в цикле.
2026-05-21 19:43:12 +03:00
TradeIn Deploy
fb49c56b81 fix(tradein): убрать TRUNCATE из seed — деплой больше не wipe-ит данные
КРИТИЧНЫЙ БАГ: 003_seed_listings.sql содержал TRUNCATE listings,deals,
а deploy-tradein.yml гоняет весь data/sql/*.sql на каждом деплое →
каждый деплой стирал реальные данные парсеров и подсовывал 120 синтетики.

Удалён 003_seed_listings.sql. Вместо него 003_seed_deals.sql —
идемпотентный (INSERT ... WHERE NOT EXISTS), сидит ТОЛЬКО сделки и
ТОЛЬКО когда таблица deals пуста. listings больше не трогается seed-ом.
2026-05-21 13:19:24 +03:00
lekss361
02267d0306 feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in)
- FastAPI backend: PostGIS estimator + 3 scrapers (Avito/Cian/Yandex)
- Next.js 15 frontend: tradein.html mockup design, basePath=/trade-in
- WeasyPrint PDF (Брусника-style 4-page report)
- Address autocomplete с typo-tolerance + 6 EKB presets
- Изолированный docker stack gendesign-tradein (отдельная postgres БД)
- Caddy inline routes: gendsgn.ru/trade-in/* и /trade-in/api/v1/*
- Forgejo Actions: .forgejo/workflows/deploy-tradein.yml (shell-based GHCR login)
- Триггер только по paths: tradein-mvp/** (не пересекается с deploy.yml)
- Образы: ghcr.io/lekss361/gendesign-tradein-{backend,frontend}:latest

Первый запуск на сервере (вручную, один раз):
  - создать /opt/gendesign/tradein-mvp/.env.runtime (postgres pwd, contact email)
  - docker network create gendesign_shared (если нет)
  - docker compose -p gendesign-tradein up -d
  - docker compose -p gendesign exec caddy caddy reload
2026-05-21 00:25:39 +03:00