feat(sf): cross-load ETL tradein→gendesign newbuilding_listings — вариант A (#976) #1185

Merged

lekss361 merged 2 commits from feat/sf-newbuilding-crossload into main

2026-06-12 08:29:55 +00:00

Author	SHA1	Message	Date
bot-backend	04da30b57a	fix(sf): psql stdin-bootstrap пароля gendesign_reader + COALESCE-protect внешних id (review fixup, #976 ) All checks were successful CI / changes (pull_request) Successful in 6s Details CI / changes (push) Successful in 7s Details CI / frontend-tests (push) Has been skipped Details CI / frontend-tests (pull_request) Has been skipped Details CI / backend-tests (push) Successful in 6m23s Details CI / backend-tests (pull_request) Successful in 6m21s Details - Создан tradein-mvp/ops/db-bootstrap/set_gendesign_reader_password.sql: set_config GUC + format('%L') + \o /dev/null — зеркальный паттерн ops/db-bootstrap/set_tradein_fdw_password.sql. Пароль не попадает в stdout/logs (leak-protected). - deploy-tradein.yml: шаг bootstrap переписан на stdin-redirect (< ops/db-bootstrap/set_gendesign_reader_password.sql) вместо сломанного -c "ALTER ROLE ... PASSWORD :'TRADEIN_READER_PASSWORD'", которое давало syntax error at ':' (psql не интерполирует :'var' внутри -c строки). - newbuilding_crossload.py ON CONFLICT DO UPDATE: yandex_jk_id и cian_internal_house_id → COALESCE(EXCLUDED.col, table.col), чтобы cian-запись не затирала yandex_jk_id NULL'ом и наоборот. - Тест test_upsert_sql_coalesce_external_ids проверяет наличие COALESCE в DO UPDATE секции для обоих id.	2026-06-12 11:26:40 +03:00
bot-backend	1a4778d006	feat(sf): cross-load ETL tradein→gendesign newbuilding_listings — вариант A (#976 ) - docker-compose.prod.yml: backend + worker добавлены в сеть gendesign_shared (нужен прямой TCP к tradein-postgres для psycopg ETL) - tradein-mvp/data/sql/101_gendesign_reader_role.sql: роль gendesign_reader (LOGIN, SELECT на houses/house_sources/houses_price_dynamics/ house_reliability_checks/house_reviews; пароль через TRADEIN_READER_PASSWORD bootstrap в deploy-tradein.yml) - .forgejo/workflows/deploy-tradein.yml: bootstrap ALTER ROLE gendesign_reader PASSWORD из env после миграций; warn (не exit 1) если переменная не задана - backend/app/core/config.py: settings.tradein_database_url (env TRADEIN_DATABASE_URL) - backend/app/services/etl/newbuilding_crossload.py: run_crossload() — psycopg server-side cursor, batch 500, UPSERT ON CONFLICT (source, ext_house_id), SAVEPOINT per-row; disabled-режим если url пуст - backend/app/workers/tasks/etl_newbuilding_crossload.py: Celery task - backend/app/workers/beat_schedule.py: nightly 03:30 МСК (00:30 UTC) - backend/app/api/v1/admin_scrape.py: POST /admin/scrape/newbuilding-crossload - backend/tests/services/test_newbuilding_crossload.py: 10 тестов без реальной БД needs-human: TRADEIN_READER_PASSWORD → tradein .env.runtime; TRADEIN_DATABASE_URL → gendesign backend/.env.runtime	2026-06-12 11:18:47 +03:00

Author

SHA1

Message

Date

bot-backend

04da30b57a

fix(sf): psql stdin-bootstrap пароля gendesign_reader + COALESCE-protect внешних id (review fixup, #976 )

CI / changes (pull_request) Successful in 6s

Details

CI / changes (push) Successful in 7s

Details

CI / frontend-tests (push) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / backend-tests (push) Successful in 6m23s

Details

CI / backend-tests (pull_request) Successful in 6m21s

Details

- Создан tradein-mvp/ops/db-bootstrap/set_gendesign_reader_password.sql:
  set_config GUC + format('%L') + \o /dev/null — зеркальный паттерн
  ops/db-bootstrap/set_tradein_fdw_password.sql. Пароль не попадает
  в stdout/logs (leak-protected).
- deploy-tradein.yml: шаг bootstrap переписан на stdin-redirect
  (< ops/db-bootstrap/set_gendesign_reader_password.sql) вместо сломанного
  -c "ALTER ROLE ... PASSWORD :'TRADEIN_READER_PASSWORD'", которое давало
  syntax error at ':' (psql не интерполирует :'var' внутри -c строки).
- newbuilding_crossload.py ON CONFLICT DO UPDATE: yandex_jk_id и
  cian_internal_house_id → COALESCE(EXCLUDED.col, table.col), чтобы
  cian-запись не затирала yandex_jk_id NULL'ом и наоборот.
- Тест test_upsert_sql_coalesce_external_ids проверяет наличие COALESCE
  в DO UPDATE секции для обоих id.

2026-06-12 11:26:40 +03:00

bot-backend

1a4778d006

feat(sf): cross-load ETL tradein→gendesign newbuilding_listings — вариант A (#976 )

- docker-compose.prod.yml: backend + worker добавлены в сеть gendesign_shared
  (нужен прямой TCP к tradein-postgres для psycopg ETL)
- tradein-mvp/data/sql/101_gendesign_reader_role.sql: роль gendesign_reader
  (LOGIN, SELECT на houses/house_sources/houses_price_dynamics/
   house_reliability_checks/house_reviews; пароль через TRADEIN_READER_PASSWORD
   bootstrap в deploy-tradein.yml)
- .forgejo/workflows/deploy-tradein.yml: bootstrap ALTER ROLE gendesign_reader
  PASSWORD из env после миграций; warn (не exit 1) если переменная не задана
- backend/app/core/config.py: settings.tradein_database_url (env TRADEIN_DATABASE_URL)
- backend/app/services/etl/newbuilding_crossload.py: run_crossload() — psycopg
  server-side cursor, batch 500, UPSERT ON CONFLICT (source, ext_house_id),
  SAVEPOINT per-row; disabled-режим если url пуст
- backend/app/workers/tasks/etl_newbuilding_crossload.py: Celery task
- backend/app/workers/beat_schedule.py: nightly 03:30 МСК (00:30 UTC)
- backend/app/api/v1/admin_scrape.py: POST /admin/scrape/newbuilding-crossload
- backend/tests/services/test_newbuilding_crossload.py: 10 тестов без реальной БД

needs-human: TRADEIN_READER_PASSWORD → tradein .env.runtime;
             TRADEIN_DATABASE_URL → gendesign backend/.env.runtime

2026-06-12 11:18:47 +03:00