feat(etl): read tradein data via market.v_* contract views (#2138 step 2/3)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 2m28s
CI / backend-tests (pull_request) Successful in 14m31s

This commit is contained in:
bot-backend 2026-07-02 17:59:21 +03:00
parent 03b3cb9fe4
commit 8451791c64

View file

@ -1,9 +1,12 @@
"""Cross-load ETL: tradein.houses → gendesign.newbuilding_listings (#976 950-E5).
Читает houses + house_sources из tradein-БД (gendesign_reader role),
Читает market.v_houses + market.v_house_sources из tradein-БД (gendesign_reader role),
UPSERT-ит в gendesign newbuilding_listings по (source, ext_house_id).
Соединение к tradein через psycopg v3 (прямой коннект, НЕ SQLAlchemy).
Источник contract-view market.v_* (#2138), а не сырые public.*-таблицы: явный
column-контракт, устойчивый к scraper-driven ALTER TABLE на стороне tradein.
Запускается:
- ночью Celery beat (03:30 МСК, tasks/etl_newbuilding_crossload.py)
- вручную через POST /api/v1/admin/scrape/newbuilding-crossload
@ -54,8 +57,10 @@ SELECT
h.last_scraped_at,
-- yandex_jk_id из house_sources (LEFT JOIN может отсутствовать)
hs_y.ext_id AS yandex_jk_id
FROM houses h
LEFT JOIN house_sources hs_y
-- Читаем через contract-view market.v_* (#2138 step 2/3), НЕ сырые public.* —
-- стабильный явный column-контракт, не ломается на scraper-driven ALTER TABLE.
FROM market.v_houses h
LEFT JOIN market.v_house_sources hs_y
ON hs_y.house_id = h.id
AND hs_y.ext_source = 'yandex'
WHERE h.lat IS NOT NULL