feat(avito): incremental SERP pagination with date early-stop (param-gated, default off) #1766

Merged

lekss361 merged 1 commit from feat/avito-incremental-pagination into main

2026-06-18 19:03:38 +00:00

lekss361 commented

2026-06-18 19:02:38 +00:00

Owner

What

Adds an opt-in incremental mode to the Avito exhaustive SERP walker. fetch_all_secondary(..., since: date | None = None):

since=None (default) → exactly today's exhaustive bisection walk. Zero behavior change.
since=<date> → per (room × seed-bracket), paginate sequentially newest-first and stop early once a page has no card newer than since. Skips deep pagination (page 20–31) entirely — which is precisely the source of the 403 ban clusters that killed runs 211/212.

No scheduler/schedule change in this PR — the engine ships dormant. Wiring run_avito_full_load + a split schedule (incremental daily + exhaustive weekly) is a separate follow-up, to flip only after this engine is proven on a manual incremental run.

Why

Prod data: of 4359 active vtorichka, only ~431 are new/day (~10%). A full pass re-paginates the whole catalog deep every time (99.6% re-saves), and the deep pages are where 403s cluster. Incremental reads only the fresh top pages → ~4× fewer requests and, more importantly, shallow (page 1–2) so the deep-pagination ban vector disappears.

Delisting (is_active=false after 10-day TTL) still needs a periodic full pass to refresh last_seen_at — hence the planned daily-incremental + weekly-exhaustive split (next PR), which fits the 10-day TTL with margin.

Design notes

Newest-first sort (s=104) + per-card listing_date (from sortTimeStamp) make date early-stop sound.
Incremental path does not price-bisect (it never approaches the SERP cap — only reads top recent pages).
None listing_date doesn't stop on its own; a 2-consecutive-undated-page streak triggers a grace stop (runaway guard).
AvitoBlockedError/AvitoRateLimitedError propagate as in the leaf bucket. _walk_price_range/_paginate_leaf_bucket untouched.

Tests

New tests/scrapers/test_avito_incremental_pagination.py (8 tests): early-stop at the first all-older page (page 4 not fetched), no-stop-when-all-fresh, all-None grace bound, single-undated-page doesn't stop, secondary_only filter, block propagation, and since=None→exhaustive path spy (asserts _walk_price_range used, incremental method not). tests/scrapers/ -k avito: 65 passed. Ruff clean.

## What Adds an opt-in **incremental** mode to the Avito exhaustive SERP walker. `fetch_all_secondary(..., since: date | None = None)`: - `since=None` (default) → **exactly today's exhaustive bisection walk. Zero behavior change.** - `since=<date>` → per (room × seed-bracket), paginate **sequentially newest-first** and **stop early** once a page has no card newer than `since`. Skips deep pagination (page 20–31) entirely — which is precisely the source of the 403 ban clusters that killed runs 211/212. No scheduler/schedule change in this PR — the engine ships dormant. Wiring `run_avito_full_load` + a split schedule (incremental daily + exhaustive weekly) is a separate follow-up, to flip only after this engine is proven on a manual incremental run. ## Why Prod data: of 4359 active vtorichka, only ~431 are new/day (~10%). A full pass re-paginates the whole catalog deep every time (99.6% re-saves), and the deep pages are where 403s cluster. Incremental reads only the fresh top pages → ~4× fewer requests and, more importantly, shallow (page 1–2) so the deep-pagination ban vector disappears. Delisting (`is_active=false` after 10-day TTL) still needs a periodic full pass to refresh `last_seen_at` — hence the planned daily-incremental + weekly-exhaustive split (next PR), which fits the 10-day TTL with margin. ## Design notes - Newest-first sort (`s=104`) + per-card `listing_date` (from `sortTimeStamp`) make date early-stop sound. - Incremental path does **not** price-bisect (it never approaches the SERP cap — only reads top recent pages). - `None` listing_date doesn't stop on its own; a 2-consecutive-undated-page streak triggers a grace stop (runaway guard). - `AvitoBlockedError`/`AvitoRateLimitedError` propagate as in the leaf bucket. `_walk_price_range`/`_paginate_leaf_bucket` untouched. ## Tests New `tests/scrapers/test_avito_incremental_pagination.py` (8 tests): early-stop at the first all-older page (page 4 not fetched), no-stop-when-all-fresh, all-None grace bound, single-undated-page doesn't stop, `secondary_only` filter, block propagation, and `since=None`→exhaustive path spy (asserts `_walk_price_range` used, incremental method not). `tests/scrapers/ -k avito`: **65 passed**. Ruff clean.

lekss361 added 1 commit 2026-06-18 19:02:38 +00:00

feat(avito): incremental SERP pagination with date early-stop (param-gated, default off)

CI / changes (pull_request) Successful in 7s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

bdedac3012

Adds since=date mode to fetch_all_secondary: per room x seed-bracket, paginate
sequentially newest-first and stop once cards fall below the watermark, skipping
deep pagination (and the 403 clusters it triggers). since=None keeps the exact
exhaustive bisection walk. No scheduler/schedule change here — wiring follows
in a separate PR.