feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563) #630

Merged

Light1YT merged 2 commits from feat/560-563-scrapers-scheduler into main

2026-05-28 15:31:24 +00:00

Light1YT commented

2026-05-28 15:29:43 +00:00

Owner

Summary

scheduler.py: cian_history_backfill + rosreestr_dkp_import branches; avito-sweep refactored to shared _claim_run() (deep-review verified byte-equivalent).
migration tradein 072: FDW foreign table gendesign_rosreestr_deals + scrape_schedules seed (night-window next_run_at, NOT immediate-fire).
migration main 122: GRANT SELECT ON rosreestr_deals TO tradein_fdw_reader (prereq — FDW reader was REVOKE ALL).

Prereqs handled

FDW grant (migration 122, deploy.yml)
seed next_run_at = next night window (no immediate fire)

Known caveats (follow-up)

deploy-tradein.yml does not track tradein migrations (re-runs each deploy, swallows errors) — 072 is idempotent so safe.
rosreestr import is full re-scan per run (dedup via dedup_hash), not cursor-resume.
cian backfill needs valid session cookies; skips with GlitchTip alert if expired.
geocode backfill of imported deals stays manual (follow-up).

Closes #560
Closes #563

## Summary - scheduler.py: cian_history_backfill + rosreestr_dkp_import branches; avito-sweep refactored to shared _claim_run() (deep-review verified byte-equivalent). - migration tradein 072: FDW foreign table gendesign_rosreestr_deals + scrape_schedules seed (night-window next_run_at, NOT immediate-fire). - migration main 122: GRANT SELECT ON rosreestr_deals TO tradein_fdw_reader (prereq — FDW reader was REVOKE ALL). ## Prereqs handled - [x] FDW grant (migration 122, deploy.yml) - [x] seed next_run_at = next night window (no immediate fire) ## Known caveats (follow-up) - deploy-tradein.yml does not track tradein migrations (re-runs each deploy, swallows errors) — 072 is idempotent so safe. - rosreestr import is full re-scan per run (dedup via dedup_hash), not cursor-resume. - cian backfill needs valid session cookies; skips with GlitchTip alert if expired. - geocode backfill of imported deals stays manual (follow-up). Closes #560 Closes #563

Light1YT added 2 commits 2026-05-28 15:29:43 +00:00

feat(tradein): автоматизация cian-backfill + rosreestr-import через scheduler (#560 #563 ) 21888bd866

Оба источника данных до этого требовали ручного запуска (HTTP-запрос в admin UI
и bash-скрипт на хосте соответственно). Теперь оба интегрированы в in-app scheduler
(тот же механизм что avito_city_sweep) — запускаются автоматически ночью.

#560 (cian_history_backfill):
- Window 02:00–05:00 UTC; pre-check verify_session() перед claim'ом слота —
  если куки истекли, run не создаётся, отправляется GlitchTip capture_message.
- Вызывает существующий backfill_cian_history() как есть; checkpoint через
  update_heartbeat() до и после вызова (для zombie-detection).

#563 (rosreestr_dkp_import):
- Window 04:00–06:00 UTC; sync функция в run_in_executor (DB-only, нет HTTP).
- Cursor-based пагинация (WHERE id > last_id ORDER BY id LIMIT batch_size),
  heartbeat = checkpoint каждый батч (~2000 строк по умолчанию).
- SAVEPOINT per row через begin_nested() — один сбойный row не откатывает батч.
- Идемпотентен: ON CONFLICT (dedup_hash) DO NOTHING; dedup_hash = md5('ros:dkp:'+id)
  соответствует Fix #549.
- Cleanup при старте: удаляет legacy rows address='Екатеринбург, реальная сделка'.
- Координаты NULL — геокодинг остаётся ручным follow-up.

Migration 072: seed rows scrape_schedules + foreign table gendesign_rosreestr_deals
(через gendesign_remote FDW server, настроен в 060).

feat(tradein): FDW grant + night-window seed for scrapers scheduler (#560 #563 ) edd50a04fd

Prod prerequisites identified in deep review of the scheduler automation:

- data/sql/122: GRANT SELECT ON rosreestr_deals TO tradein_fdw_reader.
  scheduler import reads gendesign.rosreestr_deals via FDW, but the reader
  role was REVOKE ALL (100_tradein_fdw_role.sql) with no grant → import
  would fail "permission denied for table rosreestr_deals". Single parent
  grant suffices for declarative partitioning.

- tradein 072: seed next_run_at = next night-window (tomorrow + window_start
  hour, pinned UTC) instead of NULL. NULL made get_due_schedules() fire both
  rows within ~1 tick of deploy — outside the night window and (for cian)
  before the FDW grant lands. Stays enabled; just shifts first fire to night.