51655b6aaf
fix(workers): per-doc error-isolation в izyatie OCR + breadcrumb в newbuilding crossload (#2445-D5/D6)
...
CI Trade-In / changes (pull_request) Successful in 21s
CI / changes (pull_request) Successful in 13s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 2m9s
CI / backend-tests (pull_request) Successful in 15m30s
D5: расширен per-doc try/except в izyatie_ocr_ingest на весь body
(fetch→OCR→extract→upsert→commit), не только fetch, + db.rollback() на сбое —
один битый документ теперь skip'ается, а не убивает весь недельный батч.
D6: etl_newbuilding_crossload обёрнут в тот же start/success/failure breadcrumb
паттерн, что у sibling-тасков (objective_scrape_runs, group='newbuilding-crossload')
— падения видны в operational-логе, не только Celery FAILURE + GlitchTip.
На failure re-raise (Celery/GlitchTip срабатывают), breadcrumb-flush failure не
маскирует оригинальное исключение. Без миграции (таблица есть).
Refs #2445
2026-07-06 00:44:52 +05:00
1a4778d006
feat(sf): cross-load ETL tradein→gendesign newbuilding_listings — вариант A ( #976 )
...
- docker-compose.prod.yml: backend + worker добавлены в сеть gendesign_shared
(нужен прямой TCP к tradein-postgres для psycopg ETL)
- tradein-mvp/data/sql/101_gendesign_reader_role.sql: роль gendesign_reader
(LOGIN, SELECT на houses/house_sources/houses_price_dynamics/
house_reliability_checks/house_reviews; пароль через TRADEIN_READER_PASSWORD
bootstrap в deploy-tradein.yml)
- .forgejo/workflows/deploy-tradein.yml: bootstrap ALTER ROLE gendesign_reader
PASSWORD из env после миграций; warn (не exit 1) если переменная не задана
- backend/app/core/config.py: settings.tradein_database_url (env TRADEIN_DATABASE_URL)
- backend/app/services/etl/newbuilding_crossload.py: run_crossload() — psycopg
server-side cursor, batch 500, UPSERT ON CONFLICT (source, ext_house_id),
SAVEPOINT per-row; disabled-режим если url пуст
- backend/app/workers/tasks/etl_newbuilding_crossload.py: Celery task
- backend/app/workers/beat_schedule.py: nightly 03:30 МСК (00:30 UTC)
- backend/app/api/v1/admin_scrape.py: POST /admin/scrape/newbuilding-crossload
- backend/tests/services/test_newbuilding_crossload.py: 10 тестов без реальной БД
needs-human: TRADEIN_READER_PASSWORD → tradein .env.runtime;
TRADEIN_DATABASE_URL → gendesign backend/.env.runtime
2026-06-12 11:18:47 +03:00
