feat(avito): incremental daily + exhaustive weekly schedule split (+ scheduler interval_days) #1770

Merged

lekss361 merged 2 commits from feat/avito-incremental-schedule-split into main

2026-06-18 20:25:39 +00:00

lekss361 commented

2026-06-18 20:11:23 +00:00

Owner

What

Activates the (merged, dormant) incremental SERP engine and splits the avito full-load schedule:

avito_full_load → flipped to incremental daily (incremental_days: 2 → since = today-2d, shallow date early-stop). Avoids the deep pagination (page 36) where the 429 bans happen.
avito_full_load_exhaustive (new source) → weekly full walk to refresh last_seen_at (feeds the 10-day delisting TTL) and catch silent price edits on old listings.

Pieces

run_avito_full_load(incremental_days) — when set, passes since=today-incremental_days to fetch_all_secondary; None → exhaustive (unchanged).
Scheduler — avito_full_load trigger reads incremental_days from default_params; new avito_full_load_exhaustive dispatch forces exhaustive.
Scheduler cadence (interval_days) — scrape_schedules had no interval column, so every source was hardwired daily. Added interval_days (from default_params, default 1 = backward-compatible) to compute_next_run_at, honored at both recurrence call sites (_claim_run, defer path). Without this, "weekly exhaustive" wasn't expressible.
Migration 129 — avito_full_load gets incremental_days: 2; new avito_full_load_exhaustive seeded with interval_days: 7, first run staggered to CURRENT_DATE + 3d. Idempotent (guarded UPDATE backfills the key on re-run).

Why this is the core ban fix

Run 216 proved 403s recover (merged fix) but it still banned on a 429 at page 36 — deep pagination. Incremental never reaches deep pages → that vector disappears for the daily run. The weekly exhaustive (hardened separately in #1769) backstops delisting + silent price edits.

Tests

run_avito_full_load: incremental_days=2 → since=today-2d; None → since=None.
Scheduler dispatch: avito_full_load_exhaustive routes to exhaustive; incremental_days propagation.
compute_next_run_at: default → +1d, interval_days=7 → +7d (in-window, incl. cross-midnight window), explicit =1 → +1d.
-k "scheduler or full_load": 36 passed (focused), 318 passed across the broader avito/scheduler/full_load selection earlier. Ruff clean.

Depends on the merged incremental engine (#1766). Pairs with #1769 (429-hardening for the weekly exhaustive run). Recommend reviewing/merging #1769 alongside or before flipping, so the weekly exhaustive is ban-resilient on day one.

## What Activates the (merged, dormant) incremental SERP engine and splits the avito full-load schedule: - **`avito_full_load`** → flipped to **incremental daily** (`incremental_days: 2` → `since = today-2d`, shallow date early-stop). Avoids the deep pagination (page 36) where the 429 bans happen. - **`avito_full_load_exhaustive`** (new source) → **weekly** full walk to refresh `last_seen_at` (feeds the 10-day delisting TTL) and catch silent price edits on old listings. ## Pieces 1. **`run_avito_full_load(incremental_days)`** — when set, passes `since=today-incremental_days` to `fetch_all_secondary`; `None` → exhaustive (unchanged). 2. **Scheduler** — `avito_full_load` trigger reads `incremental_days` from `default_params`; new `avito_full_load_exhaustive` dispatch forces exhaustive. 3. **Scheduler cadence (`interval_days`)** — `scrape_schedules` had no interval column, so every source was hardwired daily. Added `interval_days` (from `default_params`, **default 1 = backward-compatible**) to `compute_next_run_at`, honored at both recurrence call sites (`_claim_run`, defer path). Without this, "weekly exhaustive" wasn't expressible. 4. **Migration `129`** — `avito_full_load` gets `incremental_days: 2`; new `avito_full_load_exhaustive` seeded with `interval_days: 7`, first run staggered to `CURRENT_DATE + 3d`. Idempotent (guarded UPDATE backfills the key on re-run). ## Why this is the core ban fix Run 216 proved 403s recover (merged fix) but it still banned on a **429 at page 36** — deep pagination. Incremental never reaches deep pages → that vector disappears for the daily run. The weekly exhaustive (hardened separately in #1769) backstops delisting + silent price edits. ## Tests - `run_avito_full_load`: `incremental_days=2` → `since=today-2d`; `None` → `since=None`. - Scheduler dispatch: `avito_full_load_exhaustive` routes to exhaustive; `incremental_days` propagation. - `compute_next_run_at`: default → +1d, `interval_days=7` → +7d (in-window, incl. cross-midnight window), explicit `=1` → +1d. - `-k "scheduler or full_load"`: **36 passed** (focused), 318 passed across the broader avito/scheduler/full_load selection earlier. Ruff clean. Depends on the merged incremental engine (#1766). Pairs with #1769 (429-hardening for the weekly exhaustive run). Recommend reviewing/merging #1769 alongside or before flipping, so the weekly exhaustive is ban-resilient on day one.

lekss361 added 2 commits 2026-06-18 20:11:24 +00:00

feat(avito): incremental daily + exhaustive weekly schedule split bf63683833

run_avito_full_load gains incremental_days param -> passes since= to the
(merged) incremental SERP engine. avito_full_load schedule flipped to
incremental_days=2 (shallow, date early-stop -> avoids deep-pagination 429
bans). New avito_full_load_exhaustive source runs the full walk weekly to
refresh last_seen (10-day delisting TTL) and catch silent price edits.

feat(scheduler): per-source interval_days cadence (weekly exhaustive avito)

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

1f7018d415

scrape_schedules had no interval column -> every source ran daily. Add
interval_days from default_params (default 1, backward-compatible) so
compute_next_run_at can schedule N days ahead. Set avito_full_load_exhaustive
to interval_days=7 (weekly full pass for last_seen/delisting + silent price
edits); avito_full_load stays daily incremental.