bot-backend
1a4778d006
- docker-compose.prod.yml: backend + worker добавлены в сеть gendesign_shared
(нужен прямой TCP к tradein-postgres для psycopg ETL)
- tradein-mvp/data/sql/101_gendesign_reader_role.sql: роль gendesign_reader
(LOGIN, SELECT на houses/house_sources/houses_price_dynamics/
house_reliability_checks/house_reviews; пароль через TRADEIN_READER_PASSWORD
bootstrap в deploy-tradein.yml)
- .forgejo/workflows/deploy-tradein.yml: bootstrap ALTER ROLE gendesign_reader
PASSWORD из env после миграций; warn (не exit 1) если переменная не задана
- backend/app/core/config.py: settings.tradein_database_url (env TRADEIN_DATABASE_URL)
- backend/app/services/etl/newbuilding_crossload.py: run_crossload() — psycopg
server-side cursor, batch 500, UPSERT ON CONFLICT (source, ext_house_id),
SAVEPOINT per-row; disabled-режим если url пуст
- backend/app/workers/tasks/etl_newbuilding_crossload.py: Celery task
- backend/app/workers/beat_schedule.py: nightly 03:30 МСК (00:30 UTC)
- backend/app/api/v1/admin_scrape.py: POST /admin/scrape/newbuilding-crossload
- backend/tests/services/test_newbuilding_crossload.py: 10 тестов без реальной БД
needs-human: TRADEIN_READER_PASSWORD → tradein .env.runtime;
TRADEIN_DATABASE_URL → gendesign backend/.env.runtime
27 lines
1.1 KiB
Python
27 lines
1.1 KiB
Python
"""Celery task: ночной cross-load ETL tradein.houses → gendesign.newbuilding_listings.
|
||
|
||
Расписание: 03:30 МСК (hardcoded в beat_schedule, как cbr/poi/supply-layers).
|
||
Идемпотентен: UPSERT ON CONFLICT (source, ext_house_id).
|
||
Если TRADEIN_DATABASE_URL не задан → warn-log, возвращает {"disabled": True}.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
|
||
from app.services.etl.newbuilding_crossload import run_crossload
|
||
from app.workers.celery_app import celery_app
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
|
||
@celery_app.task(name="tasks.etl_newbuilding_crossload.etl_newbuilding_crossload")
|
||
def etl_newbuilding_crossload() -> dict:
|
||
"""Cross-load tradein.houses → gendesign.newbuilding_listings (#976).
|
||
|
||
Вызывается beat'ом в 03:30 МСК и вручную через POST /api/v1/admin/scrape/newbuilding-crossload.
|
||
"""
|
||
logger.info("task etl_newbuilding_crossload: start")
|
||
result = run_crossload()
|
||
logger.info("task etl_newbuilding_crossload: done — %s", result)
|
||
return result
|