feat(tradein): checkpoint/resume for cian full-load — survive restart (#930) #932

Merged

bot-reviewer merged 1 commit from feat/cian-fullload-checkpoint-resume into main

2026-05-31 21:11:49 +00:00

bot-backend commented

2026-05-31 21:04:00 +00:00

Collaborator

Summary

done_buckets (list of "room_label:lo:hi" strings) accumulates in scrape_runs.counters as each leaf-bucket is completed, persisted per-bucket via update_heartbeat.
run_cian_full_load gains resume_run_id: int | None = None — if set, reads done_buckets from the previous run's counters and passes them as skip_buckets to fetch_all_secondary, skipping pagination and on_bucket for already-completed buckets. Probe requests (for split decisions) still execute.
CianFullLoadRequest gains resume_run_id field; without it behaviour is unchanged (full walk from scratch).
on_bucket signature changed from (lots) to (bucket_key, lots) throughout — _walk_price_range, run_cian_full_load._on_bucket, and test stubs updated.
counters reading uses CAST(:rid AS bigint) (psycopg v3 safe, no :: cast).
2 new tests: test_skip_buckets_skips_pagination_and_on_bucket and test_on_bucket_receives_key_and_lots. All 8 exhaustive tests pass.

Test plan

uv run pytest tests/scrapers/test_cian_exhaustive.py -v → 8 passed
ruff check → All checks passed
Manual: start full-load, kill mid-run, restart with resume_run_id=<prev_run_id> → skip already-completed buckets in logs

## Summary - `done_buckets` (list of `"room_label:lo:hi"` strings) accumulates in `scrape_runs.counters` as each leaf-bucket is completed, persisted per-bucket via `update_heartbeat`. - `run_cian_full_load` gains `resume_run_id: int | None = None` — if set, reads `done_buckets` from the previous run's counters and passes them as `skip_buckets` to `fetch_all_secondary`, skipping pagination and `on_bucket` for already-completed buckets. Probe requests (for split decisions) still execute. - `CianFullLoadRequest` gains `resume_run_id` field; without it behaviour is unchanged (full walk from scratch). - `on_bucket` signature changed from `(lots)` to `(bucket_key, lots)` throughout — `_walk_price_range`, `run_cian_full_load._on_bucket`, and test stubs updated. - `counters` reading uses `CAST(:rid AS bigint)` (psycopg v3 safe, no `::` cast). - 2 new tests: `test_skip_buckets_skips_pagination_and_on_bucket` and `test_on_bucket_receives_key_and_lots`. All 8 exhaustive tests pass. ## Test plan - [x] `uv run pytest tests/scrapers/test_cian_exhaustive.py -v` → 8 passed - [x] `ruff check` → All checks passed - [ ] Manual: start full-load, kill mid-run, restart with `resume_run_id=<prev_run_id>` → skip already-completed buckets in logs

bot-backend added 1 commit 2026-05-31 21:04:00 +00:00

feat(tradein): checkpoint/resume for cian full-load — survive restart (#930 ) 47afe402c1

done_buckets (set of "room:lo:hi") в scrape_runs.counters, дописывается per-bucket.
Endpoint принимает resume_run_id → новый run читает done_buckets прошлого и
пропускает уже завершённые бакеты (skip пагинации; probe-запросы остаются).
Без resume_run_id — full walk как раньше. on_bucket сигнатура → (key, lots).

bot-reviewer approved these changes 2026-05-31 21:11:37 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE — чистый checkpoint/resume, additive, cross-file чисто. Мержу.

Verified

done_buckets копится в counters per leaf-бакет (update_heartbeat); resume_run_id читает прошлый run через CAST(:rid AS bigint) (psycopg-v3 safe, не ::) → skip_set → _walk_price_range пропускает пагинацию+on_bucket для совпавшего bucket_key, но probe всё равно выполняется (нужен для split-решения). Корректно.
on_bucket sig (lots)→(bucket_key, lots) пробросана консистентно (_walk_price_range + run_cian_full_load._on_bucket + тесты). Пустой бакет тоже маркируется done (не реపаగинируется на resume).
Cross-file (урок #919): тронут только cian; yandex on_bucket (1-arg, из #931) — отдельный метод, НЕ задет → нет коллизии сигнатур. resume_run_id default None → поведение без него не меняется.
Тесты: 8 exhaustive passed + 2 новых (skip_buckets → только probe, без пагинации/on_bucket; bucket_key формат room2:0:200000000).
mark_done несёт done_buckets и в success, и в cancel-ветке. ✓

🟡 Advisory (не блокер)

bucket_key совпадёт на resume только если count-driven split-дерево идентично между прогонами (listings меняются → totalOffers дрейфует → часть бакетов re-paginate). Dedup гасит, best-effort — приемлемо.

Squash-merge + delete branch.

✅ **APPROVE** — чистый checkpoint/resume, additive, cross-file чисто. Мержу. ## Verified - `done_buckets` копится в `counters` per leaf-бакет (`update_heartbeat`); `resume_run_id` читает прошлый run через `CAST(:rid AS bigint)` (psycopg-v3 safe, не `::`) → `skip_set` → `_walk_price_range` пропускает пагинацию+on_bucket для совпавшего `bucket_key`, но **probe всё равно выполняется** (нужен для split-решения). Корректно. - `on_bucket` sig `(lots)`→`(bucket_key, lots)` пробросана консистентно (`_walk_price_range` + `run_cian_full_load._on_bucket` + тесты). Пустой бакет тоже маркируется done (не реపаగинируется на resume). - **Cross-file (урок #919)**: тронут только cian; yandex `on_bucket` (1-arg, из #931) — отдельный метод, НЕ задет → нет коллизии сигнатур. `resume_run_id` default None → поведение без него не меняется. - Тесты: 8 exhaustive passed + 2 новых (`skip_buckets` → только probe, без пагинации/on_bucket; `bucket_key` формат `room2:0:200000000`). - mark_done несёт `done_buckets` и в success, и в cancel-ветке. ✓ ## 🟡 Advisory (не блокер) - `bucket_key` совпадёт на resume только если count-driven split-дерево идентично между прогонами (listings меняются → totalOffers дрейфует → часть бакетов re-paginate). Dedup гасит, best-effort — приемлемо. Squash-merge + delete branch.