Commit graph

5 commits

Author SHA1 Message Date
14f3ef2019 fix(week-review): backend-аудит v2 — 82 фиксов (#1660)
All checks were successful
Deploy / build-worker (push) Successful in 2m47s
Deploy / deploy (push) Successful in 1m20s
Deploy / changes (push) Successful in 9s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 1m52s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-06-17 17:13:38 +00:00
04da30b57a fix(sf): psql stdin-bootstrap пароля gendesign_reader + COALESCE-protect внешних id (review fixup, #976)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 7s
CI / frontend-tests (push) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / backend-tests (push) Successful in 6m23s
CI / backend-tests (pull_request) Successful in 6m21s
- Создан tradein-mvp/ops/db-bootstrap/set_gendesign_reader_password.sql:
  set_config GUC + format('%L') + \o /dev/null — зеркальный паттерн
  ops/db-bootstrap/set_tradein_fdw_password.sql. Пароль не попадает
  в stdout/logs (leak-protected).
- deploy-tradein.yml: шаг bootstrap переписан на stdin-redirect
  (< ops/db-bootstrap/set_gendesign_reader_password.sql) вместо сломанного
  -c "ALTER ROLE ... PASSWORD :'TRADEIN_READER_PASSWORD'", которое давало
  syntax error at ':' (psql не интерполирует :'var' внутри -c строки).
- newbuilding_crossload.py ON CONFLICT DO UPDATE: yandex_jk_id и
  cian_internal_house_id → COALESCE(EXCLUDED.col, table.col), чтобы
  cian-запись не затирала yandex_jk_id NULL'ом и наоборот.
- Тест test_upsert_sql_coalesce_external_ids проверяет наличие COALESCE
  в DO UPDATE секции для обоих id.
2026-06-12 11:26:40 +03:00
1a4778d006 feat(sf): cross-load ETL tradein→gendesign newbuilding_listings — вариант A (#976)
- docker-compose.prod.yml: backend + worker добавлены в сеть gendesign_shared
  (нужен прямой TCP к tradein-postgres для psycopg ETL)
- tradein-mvp/data/sql/101_gendesign_reader_role.sql: роль gendesign_reader
  (LOGIN, SELECT на houses/house_sources/houses_price_dynamics/
   house_reliability_checks/house_reviews; пароль через TRADEIN_READER_PASSWORD
   bootstrap в deploy-tradein.yml)
- .forgejo/workflows/deploy-tradein.yml: bootstrap ALTER ROLE gendesign_reader
  PASSWORD из env после миграций; warn (не exit 1) если переменная не задана
- backend/app/core/config.py: settings.tradein_database_url (env TRADEIN_DATABASE_URL)
- backend/app/services/etl/newbuilding_crossload.py: run_crossload() — psycopg
  server-side cursor, batch 500, UPSERT ON CONFLICT (source, ext_house_id),
  SAVEPOINT per-row; disabled-режим если url пуст
- backend/app/workers/tasks/etl_newbuilding_crossload.py: Celery task
- backend/app/workers/beat_schedule.py: nightly 03:30 МСК (00:30 UTC)
- backend/app/api/v1/admin_scrape.py: POST /admin/scrape/newbuilding-crossload
- backend/tests/services/test_newbuilding_crossload.py: 10 тестов без реальной БД

needs-human: TRADEIN_READER_PASSWORD → tradein .env.runtime;
             TRADEIN_DATABASE_URL → gendesign backend/.env.runtime
2026-06-12 11:18:47 +03:00
lekss361
2694e3180b feat(etl): fuzzy matcher v2 — pruned threshold 0.85→0.80 for objective mapping coverage
- Add AUTO_ACCEPT_THRESHOLD_V2 = 0.80 constant to objective_backfill.py
- Add min_threshold param to find_match_candidates() (default REVIEW_THRESHOLD=0.6)
- Add match_method param to auto_apply_matches() (default 'fuzzy_trgm')
- Add ?v2=true query param to POST /api/v1/admin/etl/objective-backfill
  - v2 mode: threshold=0.80, method='fuzzy_v2', search from 0.80 (not 0.60)
- Return type widened to dict[str,object] to include threshold_used + match_method_used

DB audit: 1068 unmapped EKB objs; v1 adds ~133 rows, v2 adds ~47 more.
Expected coverage: 8.5% → ~20% after sequential v1+v2 runs on prod.
Run: POST /objective-backfill (v1) then POST /objective-backfill?v2=true (v2).
Part of task #44 Part A, epic #271.
2026-05-17 16:28:20 +03:00
dbf31cc911 feat(etl): objective_complex_mapping backfill via pg_trgm (#203) (#214)
All checks were successful
Deploy / changes (push) Successful in 4s
Deploy / build-backend (push) Successful in 1m38s
Deploy / build-frontend (push) Has been skipped
Deploy / build-worker (push) Successful in 2m36s
Deploy / deploy (push) Successful in 38s
2026-05-16 12:47:09 +00:00