feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) #894

Merged

bot-reviewer merged 1 commit from feat/888-rosreestr-poll into main

2026-05-31 13:13:56 +00:00

bot-backend commented

2026-05-31 12:58:43 +00:00

Collaborator

Summary

Ops automation (low priority). Today the Rosreestr ДКП ingest is manual (JOBS hardcoded to 2026Q1); Q2'26 lands ~Aug 2026. This adds a monthly scheduler job that detects a newly-published Rosreestr quarter and logs an actionable ingest alert — it does NOT download the multi-GB zip or shell out from the scheduler tick (that stays a deliberate human/ops step).

services/rosreestr_poll.py: latest_loaded_quarter (MAX(deal_date) from deals) + next-quarter calc + HEAD-probe of the dataset URL for availability. Fully defensive — network errors log a warning and return False, never raise into the scheduler.
tasks/rosreestr_quarter_poll.py: thin task wrapper (mirrors sber_index_pull).
scheduler.py: trigger_rosreestr_quarter_poll_run + dispatch branch for source=='rosreestr_quarter_poll' — mirrors the #887 sber wiring exactly.
Migration 096: scrape_schedules monthly seed, ON CONFLICT (source) DO NOTHING, BEGIN/COMMIT.

When a new quarter is detected it logs: Run ingest: data/sql/02_load_all_quarters.sh + tradein-mvp/deploy/import-rosreestr.sh.

Test plan

new tests/test_rosreestr_poll.py (13) + scheduler (9) → 22 passed; ruff clean.
Live: the schedule row appears in scrape_schedules; next monthly tick logs available=False for the current state (Q2'26 not yet published) — correct.

Safe/additive (new module + new schedule row + dormant until a quarter actually publishes). Migration is additive seed only.

Refs #888, #727, #724

## Summary Ops automation (low priority). Today the Rosreestr ДКП ingest is manual (JOBS hardcoded to 2026Q1); Q2'26 lands ~Aug 2026. This adds a monthly scheduler job that **detects** a newly-published Rosreestr quarter and **logs an actionable ingest alert** — it does NOT download the multi-GB zip or shell out from the scheduler tick (that stays a deliberate human/ops step). - `services/rosreestr_poll.py`: `latest_loaded_quarter` (`MAX(deal_date)` from `deals`) + next-quarter calc + HEAD-probe of the dataset URL for availability. Fully defensive — network errors log a warning and return `False`, never raise into the scheduler. - `tasks/rosreestr_quarter_poll.py`: thin task wrapper (mirrors `sber_index_pull`). - `scheduler.py`: `trigger_rosreestr_quarter_poll_run` + dispatch branch for `source=='rosreestr_quarter_poll'` — mirrors the #887 sber wiring exactly. - Migration **096**: `scrape_schedules` monthly seed, `ON CONFLICT (source) DO NOTHING`, `BEGIN/COMMIT`. When a new quarter is detected it logs: `Run ingest: data/sql/02_load_all_quarters.sh + tradein-mvp/deploy/import-rosreestr.sh`. ## Test plan - [x] new `tests/test_rosreestr_poll.py` (13) + scheduler (9) → **22 passed**; ruff clean. - [ ] Live: the schedule row appears in `scrape_schedules`; next monthly tick logs `available=False` for the current state (Q2'26 not yet published) — correct. Safe/additive (new module + new schedule row + dormant until a quarter actually publishes). Migration is additive seed only. Refs #888, #727, #724

bot-backend added 1 commit 2026-05-31 12:58:43 +00:00

feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888 ) aaf2bacfa3

Ops automation — today the ДКП ingest is manual (JOBS hardcoded to 2026Q1);
Q2'26 lands ~Aug 2026. Adds a monthly scheduler job that detects a new
Rosreestr quarter and logs an actionable ingest alert. Detect+alert only —
does NOT download the multi-GB zip or shell out from the tick.

- rosreestr_poll.py: latest_loaded_quarter (MAX deal_date) + next-quarter calc
  + HEAD-probe dataset availability; defensive (network err → False, never raises)
- tasks/rosreestr_quarter_poll.py: task wrapper (mirrors sber_index_pull)
- scheduler.py: trigger_rosreestr_quarter_poll_run + dispatch branch
- migration 096: scrape_schedules monthly seed, ON CONFLICT DO NOTHING

Refs #888, #727, #724