lekss361/gendesign

Fork 0

feat(scrapers): segment-aware generic deactivate_stale for yandex/cian #1558

Merged

lekss361 merged 2 commits from feat/deactivate-stale-generic-segment-aware into main

2026-06-16 16:09:20 +00:00

lekss361 commented

2026-06-16 15:59:05 +00:00

Owner

Summary

Generalizes the avito-only stale-listing deactivation (#759) into a per-source, segment-aware task and enables it for yandex/cian vtorichka only.

Why segment-aware (not a blanket TTL like avito)

Verified on prod (2026-06-16):

avito_city_sweep runs daily and works → deactivate_stale_avito TTL=10 is safe (251 active, all fresh).
yandex_city_sweep is narrow (5-anchor nightly sweep, pages_per_anchor=2) — only 48 of 3969 active rows refreshed in 14d.
cian_city_sweep is disabled (enabled=false); only cian_history_backfill drips ~200/day — only 28 of 15067 active cian rows refreshed in 14d.

So for yandex/cian, "not seen in N days" means "our scraper stopped covering it," not "it sold." A blanket TTL would deactivate ~19k rows including 9659 live first-party novostroyki. Therefore yandex/cian deactivate only listing_segment='vtorichka' at TTL=30; novostroyki and NULL-segment are protected. avito behaviour unchanged (all segments, TTL=10).

TTL=30 immediate blast radius is 0 rows (oldest active vtorichka last_seen is ~26d) — it only catches genuinely-aged resale going forward.

Changes

deactivate_stale_listings(db, run_id, *, listing_source, ttl_days, segments) generic fn (two SQL variants: all-segments / = ANY(CAST(:segments AS text[]))). deactivate_stale_avito_listings kept as a back-compat wrapper.
scheduler dispatch: source.startswith("deactivate_stale_") → trigger_deactivate_stale_run, reading listing_source/ttl_days/segments from default_params jsonb.
114_scrape_schedules_seed_deactivate_stale_yandex_cian.sql: seeds deactivate_stale_yandex + deactivate_stale_cian (vtorichka, TTL=30, window 07-08 UTC), idempotent ON CONFLICT DO NOTHING.
segments is None ⇒ all segments; segments=[] ⇒ none (ANY(ARRAY[]) matches nothing) — guards against an accidental empty-list wiping a whole source.

Test plan

uv run pytest full gate: 1834 passed, 2 deselected
uv run ruff check clean
Pre-push code-reviewer: ✅ APPROVE (3 minors, 2 of them addressed in this PR; dead old trigger_deactivate_stale_avito_run left as a post-merge cleanup)
Post-deploy: confirm migration 114 seeds the two schedules; first fire deactivates ~0 vtorichka (expected)

Refs #759

## Summary Generalizes the avito-only stale-listing deactivation (#759) into a **per-source, segment-aware** task and enables it for yandex/cian **vtorichka only**. ### Why segment-aware (not a blanket TTL like avito) Verified on prod (2026-06-16): - `avito_city_sweep` runs daily and works → `deactivate_stale_avito` TTL=10 is safe (251 active, all fresh). - `yandex_city_sweep` is **narrow** (5-anchor nightly sweep, `pages_per_anchor=2`) — only **48** of 3969 active rows refreshed in 14d. - `cian_city_sweep` is **disabled** (`enabled=false`); only `cian_history_backfill` drips ~200/day — only **28** of 15067 active cian rows refreshed in 14d. So for yandex/cian, "not seen in N days" means **"our scraper stopped covering it,"** not "it sold." A blanket TTL would deactivate ~19k rows including **9659 live first-party novostroyki**. Therefore yandex/cian deactivate **only `listing_segment='vtorichka'`** at **TTL=30**; novostroyki and NULL-segment are protected. avito behaviour unchanged (all segments, TTL=10). TTL=30 immediate blast radius is **0 rows** (oldest active vtorichka last_seen is ~26d) — it only catches genuinely-aged resale going forward. ## Changes - `deactivate_stale_listings(db, run_id, *, listing_source, ttl_days, segments)` generic fn (two SQL variants: all-segments / `= ANY(CAST(:segments AS text[]))`). `deactivate_stale_avito_listings` kept as a back-compat wrapper. - scheduler dispatch: `source.startswith("deactivate_stale_")` → `trigger_deactivate_stale_run`, reading `listing_source`/`ttl_days`/`segments` from `default_params` jsonb. - `114_scrape_schedules_seed_deactivate_stale_yandex_cian.sql`: seeds `deactivate_stale_yandex` + `deactivate_stale_cian` (vtorichka, TTL=30, window 07-08 UTC), idempotent `ON CONFLICT DO NOTHING`. - `segments is None` ⇒ all segments; `segments=[]` ⇒ none (`ANY(ARRAY[])` matches nothing) — guards against an accidental empty-list wiping a whole source. ## Test plan - [x] `uv run pytest` full gate: **1834 passed, 2 deselected** - [x] `uv run ruff check` clean - [x] Pre-push code-reviewer: ✅ APPROVE (3 minors, 2 of them addressed in this PR; dead old `trigger_deactivate_stale_avito_run` left as a post-merge cleanup) - [ ] Post-deploy: confirm migration 114 seeds the two schedules; first fire deactivates ~0 vtorichka (expected) Refs #759

lekss361 added 1 commit 2026-06-16 15:59:06 +00:00

feat(scrapers): segment-aware generic deactivate_stale for yandex/cian

CI / changes (push) Successful in 8s

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

CI / changes (pull_request) Successful in 6s

Details

CI / backend-tests (pull_request) Has been skipped

Details

ddd31dce82

Generalize the avito-only stale-listing deactivation into a per-source,
segment-aware task and enable it for yandex/cian vtorichka.

Why segment-aware (not blanket TTL like avito): yandex/cian city sweeps do
not maintain full inventory coverage (cian_city_sweep disabled, yandex narrow
5-anchor nightly sweep refreshes ~48/day of 3969 active). A blanket TTL would
deactivate live inventory — including 9659 active first-party novostroyki —
because 'not seen in N days' means 'not re-covered', not 'sold'. So yandex/cian
deactivate ONLY listing_segment='vtorichka' at TTL=30; novostroyki and
NULL-segment are protected. avito unchanged (all segments, TTL=10).

- deactivate_stale_listings(db, run_id, *, listing_source, ttl_days, segments)
  generic fn; deactivate_stale_avito_listings kept as back-compat wrapper.
- scheduler dispatch: source.startswith('deactivate_stale_') -> generic trigger
  reading listing_source/ttl_days/segments from default_params jsonb.
- 114 seed: deactivate_stale_yandex + deactivate_stale_cian (vtorichka, TTL=30).
- segments is None => all segments; [] => none (ANY(ARRAY[]) matches nothing).

Refs #759

lekss361 added the

scope/backend

scrapers

labels 2026-06-16 15:59:16 +00:00

bot-backend added 1 commit 2026-06-16 16:06:24 +00:00

fix(scrapers): renumber migration 114->115 (114 taken by disable_n1_sweep)

CI / changes (pull_request) Successful in 6s

Details

CI / changes (push) Successful in 8s

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI / backend-tests (push) Has been skipped

Details

CI / frontend-tests (push) Has been skipped

Details

CI / openapi-codegen-check (push) Has been skipped

Details

0267d51731

114_disable_n1_sweep.sql landed on main from #1556 — avoid duplicate prefix.
No logic change; rename + test reference update only.

lekss361 merged commit 91f9a95f93 into main

2026-06-16 16:09:20 +00:00

lekss361 deleted branch feat/deactivate-stale-generic-segment-aware

2026-06-16 16:09:20 +00:00

lekss361 referenced this pull request from a commit

2026-06-16 16:09:20 +00:00

feat(scrapers): segment-aware generic deactivate_stale for yandex/cian (#1558)

lekss361 commented

2026-06-16 16:09:33 +00:00

Author

Owner

Deep review — APPROVE + merged (squash).

Verified at head 0267d517:

SQL param safety: both UPDATE variants use bound params, CAST(:ttl_days || ' days' AS interval) + = ANY(CAST(:segments AS text[])), no ❌:type shorthand in app code.
Segment semantics: if segments is not None guard correct — empty list [] matches nothing (not all-segments), None = all-segments. avito compat wrapper passes segments=None.
Dispatch prefix: source.startswith("deactivate_stale_") is unambiguous — only avito (090) + yandex/cian (115) seeds use that prefix; no other scrape_schedules source collides.
Migration 115 unique on main (verified via merge-tree 3-way: 113_deactivate_ghost survives, not deleted — the 'D' in the 2-dot diff is a base-drift display artifact only). Idempotent ON CONFLICT, next_run bootstrapped to tomorrow 07:00 UTC.
Back-compat: deactivate_stale_avito_listings wrapper + dead trigger_deactivate_stale_avito_run intact (test-pinned). avito row default_params={} → avito defaults via new trigger.
Gate: 1834 passed, 2 deselected, 0 fail. ruff clean on all 4 PR files (the 2 B011 errors are pre-existing in test_cadastral_reverse.py, untouched here).

Optional post-merge cleanup (non-blocking): trigger_deactivate_stale_avito_run is now dead code, kept only for a test pin; and consider git merge main into future branches to avoid the stale-base diff artifact.

Deep review — APPROVE + merged (squash). Verified at head 0267d517: - SQL param safety: both UPDATE variants use bound params, CAST(:ttl_days || ' days' AS interval) + = ANY(CAST(:segments AS text[])), no :x::type shorthand in app code. - Segment semantics: `if segments is not None` guard correct — empty list [] matches nothing (not all-segments), None = all-segments. avito compat wrapper passes segments=None. - Dispatch prefix: source.startswith("deactivate_stale_") is unambiguous — only avito (090) + yandex/cian (115) seeds use that prefix; no other scrape_schedules source collides. - Migration 115 unique on main (verified via merge-tree 3-way: 113_deactivate_ghost survives, not deleted — the 'D' in the 2-dot diff is a base-drift display artifact only). Idempotent ON CONFLICT, next_run bootstrapped to tomorrow 07:00 UTC. - Back-compat: deactivate_stale_avito_listings wrapper + dead trigger_deactivate_stale_avito_run intact (test-pinned). avito row default_params={} → avito defaults via new trigger. - Gate: 1834 passed, 2 deselected, 0 fail. ruff clean on all 4 PR files (the 2 B011 errors are pre-existing in test_cadastral_reverse.py, untouched here). Optional post-merge cleanup (non-blocking): trigger_deactivate_stale_avito_run is now dead code, kept only for a test pin; and consider `git merge main` into future branches to avoid the stale-base diff artifact.