Objective API (api.objctv.ru) интегрирован как новый source of truth для
per-flat данных по новостройкам УрФО. Заменяет промежуточный Anton-SQLite
(legacy bootstrap-ETL остался как fallback в свёрнутом блоке UI).
Schema (data/sql/68_v2 — applied на проде, 6 таблиц):
- objective_lots (UPSERT по lot_id; 303 677 rows)
- objective_corpus_room_month (long-формат месяц×корпус×room_bucket; 19 738 rows)
- objective_lots_history (append-only weekly snapshots для elasticity)
- objective_complex_mapping (Objective ComplexName ↔ domrf_kn_objects.obj_id)
- objective_raw_reports (jsonb страховка на смену схемы API)
- objective_scrape_runs (журнал прогонов)
+ data/sql/72_objective_sync_config (single-row динамический конфиг)
Backend:
- services/scrapers/objective.py: ObjectiveClient — Bearer-токен (Redis +
in-memory fallback), retry на 401/429/5xx, Retry-After header support
- services/objective_etl.py: ETL SQLite Антона → PG (legacy)
- services/objective_sync_config.py: read/update single-row config
- workers/tasks/scrape_objective.py:
* sync_objective_group: 2 рабочих отчёта (corp_sum, lots_pf), inline-парсинг
* sync_all_groups: wrapper, перебирает группы из БД-конфига с
inter-group паузой; PATCH-merge explicit args > DB config
- workers/tasks/objective_etl.py: Celery task для legacy bootstrap
- workers/celery_app.py: beat читает cron из БД при старте (fallback на env)
- api/v1/admin_scrape.py: 5 новых endpoints для /objective/*
Frontend (frontend/src/app/admin/scrape/objective/page.tsx):
- PRIMARY blue блок «🌐 Наш sync» с input для override групп
- Collapsible «⚙️ Настройки» с формой (cron + 8 параметров) → PUT в БД
- Coverage-панель с PG counts + строка про SQLite Антона как legacy
- Collapsible «🛠 Bootstrap ETL» — legacy-инструмент
Beat schedule: вторник 06:00 МСК, ~10-15 мин на 4 группы (Свердл.обл +
Челябинск + Тюмень + Пермь = ~700K квартир УрФО). Расписание и параметры
меняются через админку без редеплоя (cron требует restart beat).
Эмпирические находки об API (probe 2026-05-10):
- 13 из 21 проверенных group_name доступны на тарифе (включая «Свердловская
область», «Челябинск», «Тюмень», «Пермь», но НЕ «Свердловская обл» —
формат имени строгий)
- ComplexName требует БЕЗ префикса «ЖК» и БЕЗ кавычек
- Поле «Банк» для всех 303k = NULL (тариф не отдаёт), но «Тип обременения»
работает (36% строк = ипотека) → ipoteka_share возможен, банковская
атрибуция — нет
Docker:
- bind-mount /opt/gendesign/site-finder:/data/anton-sqlite:ro в worker
GitHub backlog: добавлены #22-25 (recommend_mix v3 — 4 сабтаска по
улучшению алгоритма на основе фидбэка про POI / границы районов /
конкурентов / окно данных / success-driven mix).
Knowledge graph (memory/memory-gendesign.jsonl): обновлены entities
Objective_Integration_May07_2026, Schema_Objective_v2_May07,
Objective_API_Findings_May07, Module_Objective_Client + новый
Session_End_May07_2026.
TODO для прода: прописать OBJECTIVE_API_KEY=<key> в backend/.env +
docker compose restart worker beat.
132 lines
4.2 KiB
YAML
132 lines
4.2 KiB
YAML
# Production compose. Pulls pre-built images from GHCR
|
||
# (built and pushed by .github/workflows/deploy.yml).
|
||
#
|
||
# Place at /opt/gendesign/docker-compose.prod.yml on the VM.
|
||
# Required env in /opt/gendesign/.env:
|
||
# POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD
|
||
# IMAGE_TAG (defaults to "latest")
|
||
#
|
||
# In production the browser talks to Caddy on :80/:443.
|
||
# Caddy proxies /api/* and /health straight to backend, the rest to frontend.
|
||
# So the frontend uses same-origin relative URLs — no NEXT_PUBLIC_API_BASE_URL needed.
|
||
#
|
||
# Postgres + Redis run alongside the app on the same VM (Discovery mode).
|
||
# Volumes are shared with docker-compose.yml so switching between files preserves data.
|
||
|
||
services:
|
||
postgres:
|
||
image: postgis/postgis:16-3.4
|
||
restart: unless-stopped
|
||
environment:
|
||
POSTGRES_DB: ${POSTGRES_DB}
|
||
POSTGRES_USER: ${POSTGRES_USER}
|
||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||
# Bind to 127.0.0.1 only — accessible from host (for SSH tunnel) but not from public internet.
|
||
# UFW additionally blocks 5432 from outside.
|
||
ports:
|
||
- "127.0.0.1:5432:5432"
|
||
volumes:
|
||
- postgres_data:/var/lib/postgresql/data
|
||
- ./backend/db/init:/docker-entrypoint-initdb.d:ro
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER}"]
|
||
interval: 5s
|
||
timeout: 5s
|
||
retries: 10
|
||
|
||
redis:
|
||
image: redis:7-alpine
|
||
restart: unless-stopped
|
||
volumes:
|
||
- redis_data:/data
|
||
healthcheck:
|
||
test: ["CMD", "redis-cli", "ping"]
|
||
interval: 10s
|
||
timeout: 3s
|
||
retries: 5
|
||
|
||
backend:
|
||
image: ghcr.io/lekss361/gendesign-backend:${IMAGE_TAG:-latest}
|
||
restart: unless-stopped
|
||
# .env.runtime пишется deploy.yml через SSH (SENTRY_RELEASE=$IMAGE_TAG).
|
||
# required: false — compose не падает если файла нет (первый деплой).
|
||
env_file:
|
||
- path: ./backend/.env
|
||
- path: ./backend/.env.runtime
|
||
required: false
|
||
depends_on:
|
||
postgres:
|
||
condition: service_healthy
|
||
redis:
|
||
condition: service_started
|
||
ports:
|
||
- "127.0.0.1:8000:8000"
|
||
healthcheck:
|
||
test: ["CMD", "curl", "-fsS", "http://localhost:8000/health"]
|
||
interval: 30s
|
||
timeout: 5s
|
||
retries: 5
|
||
start_period: 20s
|
||
|
||
frontend:
|
||
image: ghcr.io/lekss361/gendesign-frontend:${IMAGE_TAG:-latest}
|
||
restart: unless-stopped
|
||
ports:
|
||
- "127.0.0.1:3000:3000"
|
||
depends_on:
|
||
- backend
|
||
|
||
worker:
|
||
# Отдельный chromium-образ (+200 МБ Playwright). См. backend/Dockerfile target=runner-with-chromium.
|
||
image: ghcr.io/lekss361/gendesign-worker:${IMAGE_TAG:-latest}
|
||
restart: unless-stopped
|
||
env_file:
|
||
- path: ./backend/.env
|
||
- path: ./backend/.env.runtime
|
||
required: false
|
||
depends_on:
|
||
postgres:
|
||
condition: service_healthy
|
||
redis:
|
||
condition: service_started
|
||
volumes:
|
||
- ./data:/app/data # playwright_state.json + photo binaries
|
||
# Read-only bind-mount Антоновского /sf/ SQLite — для objective_etl task.
|
||
# На VPS файл лежит в /opt/gendesign/site-finder/analysis.db; путь внутри
|
||
# контейнера задан settings.objective_anton_sqlite_path (default
|
||
# /data/anton-sqlite/analysis.db).
|
||
- /opt/gendesign/site-finder:/data/anton-sqlite:ro
|
||
command: ["celery", "-A", "app.workers.celery_app", "worker", "--loglevel=info", "--concurrency=2"]
|
||
|
||
beat:
|
||
# Lean backend-образ (без Chromium) — beat только триггерит таски в Redis.
|
||
image: ghcr.io/lekss361/gendesign-backend:${IMAGE_TAG:-latest}
|
||
restart: unless-stopped
|
||
env_file:
|
||
- path: ./backend/.env
|
||
- path: ./backend/.env.runtime
|
||
required: false
|
||
depends_on:
|
||
redis:
|
||
condition: service_started
|
||
command: ["celery", "-A", "app.workers.celery_app", "beat", "--loglevel=info"]
|
||
|
||
caddy:
|
||
image: caddy:2
|
||
restart: unless-stopped
|
||
ports:
|
||
- "80:80"
|
||
- "443:443"
|
||
volumes:
|
||
- ./Caddyfile:/etc/caddy/Caddyfile:ro
|
||
- caddy_data:/data
|
||
- caddy_config:/config
|
||
depends_on:
|
||
- backend
|
||
- frontend
|
||
|
||
volumes:
|
||
postgres_data:
|
||
redis_data:
|
||
caddy_data:
|
||
caddy_config:
|