fix(tradein): repoint rosreestr_poll to /data-sets/ + verify Content-Type (СДЕЛКИ dataset moved off /opendata) #2477

Merged

bot-backend merged 1 commit from fix/rosreestr-poll-datasets-repoint into main

2026-07-12 13:21:59 +00:00

bot-backend commented

2026-07-12 13:17:37 +00:00

Collaborator

Summary

The Rosreestr СДЕЛКИ open-data portal moved off /opendata/ (Bitrix redesign). The old flat dataset URL https://rosreestr.gov.ru/opendata/dataset_СДЕЛКИ_...csv.zip now returns HTTP 200 with Content-Type: text/html (soft-404) instead of 404, so check_new_quarter_available() was falsely reporting every quarter as available. /opendata/f.json also no longer serves JSON.
Rewrote the check to parse the new open Apache autoindex at https://rosreestr.gov.ru/data-sets/:
1. GET the autoindex, decode each folder href as cp1251 percent-encoding (portal encodes Cyrillic as cp1251, not utf-8) and match "{quarter} квартал {year}г.".
2. GET that folder, find the dataset_СДЕЛКИ_r-r_01-92_y_{year}_q_{quarter}.csv.zip href the same way.
3. HEAD that file — available only if Content-Type: application/zip and Content-Length > 100 KB. A bare HTTP 200 is no longer sufficient (that was the soft-404 bug).
Sends a browser User-Agent and uses verify=False (RU TLS root CA not in the default trust store) — same pattern already used by sber_index.py (#922) for the analogous sberindex.ru issue.
rosreestr_dataset_url() now builds a best-effort cp1251-encoded link into /data-sets/ for the alert message (verified byte-for-byte against the live portal); detection itself never depends on this guessed URL, only on parsing the real autoindex.
Public function signatures and the poll_rosreestr_new_quarter() return dict shape are unchanged, so app/tasks/rosreestr_quarter_poll.py needed no changes.
Note: the manual loaders (data/sql/02_load_all_quarters.sh, tradein-mvp/deploy/import-rosreestr.sh) live outside this repo and still point at the old /opendata/ base — the alert log message now calls this out explicitly so ops know to update them too.

Test plan

uv run pytest -q tests/test_rosreestr_poll.py — 21 tests green, including a regression test asserting available=False for an HTTP 200 + Content-Type: text/html response (the exact soft-404 bug), plus tests for real-zip true, tiny-Content-Length false, missing-folder false, missing-file false, and network/timeout error handling.
uv run pytest -q tests/test_backtest_regression_gate.py — untouched, stays green (2 tests).
ruff check / ruff format --check / pre-commit — all green.
Encoding verified against a live-downloaded real file: https://rosreestr.gov.ru/data-sets/1%20%EA%E2%E0%F0%F2%E0%EB%202026%E3./dataset_%D1%C4%C5%CB%CA%C8_r-r_01-92_y_2026_q_1.csv.zip → HTTP 200, Content-Type: application/zip, ~9.5MB.

## Summary - The Rosreestr СДЕЛКИ open-data portal moved off `/opendata/` (Bitrix redesign). The old flat dataset URL `https://rosreestr.gov.ru/opendata/dataset_СДЕЛКИ_...csv.zip` now returns **HTTP 200 with Content-Type: text/html** (soft-404) instead of 404, so `check_new_quarter_available()` was falsely reporting *every* quarter as available. `/opendata/f.json` also no longer serves JSON. - Rewrote the check to parse the new open Apache autoindex at `https://rosreestr.gov.ru/data-sets/`: 1. GET the autoindex, decode each folder href as **cp1251** percent-encoding (portal encodes Cyrillic as cp1251, not utf-8) and match `"{quarter} квартал {year}г."`. 2. GET that folder, find the `dataset_СДЕЛКИ_r-r_01-92_y_{year}_q_{quarter}.csv.zip` href the same way. 3. HEAD that file — available **only if** `Content-Type: application/zip` **and** `Content-Length > 100 KB`. A bare HTTP 200 is no longer sufficient (that was the soft-404 bug). - Sends a browser `User-Agent` and uses `verify=False` (RU TLS root CA not in the default trust store) — same pattern already used by `sber_index.py` (#922) for the analogous `sberindex.ru` issue. - `rosreestr_dataset_url()` now builds a best-effort cp1251-encoded link into `/data-sets/` for the alert message (verified byte-for-byte against the live portal); detection itself never depends on this guessed URL, only on parsing the real autoindex. - Public function signatures and the `poll_rosreestr_new_quarter()` return dict shape are unchanged, so `app/tasks/rosreestr_quarter_poll.py` needed no changes. - Note: the manual loaders (`data/sql/02_load_all_quarters.sh`, `tradein-mvp/deploy/import-rosreestr.sh`) live outside this repo and still point at the old `/opendata/` base — the alert log message now calls this out explicitly so ops know to update them too. ## Test plan - [x] `uv run pytest -q tests/test_rosreestr_poll.py` — 21 tests green, including a regression test asserting `available=False` for an HTTP 200 + `Content-Type: text/html` response (the exact soft-404 bug), plus tests for real-zip true, tiny-Content-Length false, missing-folder false, missing-file false, and network/timeout error handling. - [x] `uv run pytest -q tests/test_backtest_regression_gate.py` — untouched, stays green (2 tests). - [x] `ruff check` / `ruff format --check` / pre-commit — all green. - [x] Encoding verified against a live-downloaded real file: `https://rosreestr.gov.ru/data-sets/1%20%EA%E2%E0%F0%F2%E0%EB%202026%E3./dataset_%D1%C4%C5%CB%CA%C8_r-r_01-92_y_2026_q_1.csv.zip` → HTTP 200, `Content-Type: application/zip`, ~9.5MB.

Rows
Columns