feat(scrapers): segment-aware generic deactivate_stale for yandex/cian #1558

Merged
lekss361 merged 2 commits from feat/deactivate-stale-generic-segment-aware into main 2026-06-16 16:09:20 +00:00

2 commits

Author SHA1 Message Date
0267d51731 fix(scrapers): renumber migration 114->115 (114 taken by disable_n1_sweep)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / changes (push) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
114_disable_n1_sweep.sql landed on main from #1556 — avoid duplicate prefix.
No logic change; rename + test reference update only.
2026-06-16 19:06:13 +03:00
ddd31dce82 feat(scrapers): segment-aware generic deactivate_stale for yandex/cian
All checks were successful
CI / changes (push) Successful in 8s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / backend-tests (push) Has been skipped
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
Generalize the avito-only stale-listing deactivation into a per-source,
segment-aware task and enable it for yandex/cian vtorichka.

Why segment-aware (not blanket TTL like avito): yandex/cian city sweeps do
not maintain full inventory coverage (cian_city_sweep disabled, yandex narrow
5-anchor nightly sweep refreshes ~48/day of 3969 active). A blanket TTL would
deactivate live inventory — including 9659 active first-party novostroyki —
because 'not seen in N days' means 'not re-covered', not 'sold'. So yandex/cian
deactivate ONLY listing_segment='vtorichka' at TTL=30; novostroyki and
NULL-segment are protected. avito unchanged (all segments, TTL=10).

- deactivate_stale_listings(db, run_id, *, listing_source, ttl_days, segments)
  generic fn; deactivate_stale_avito_listings kept as back-compat wrapper.
- scheduler dispatch: source.startswith('deactivate_stale_') -> generic trigger
  reading listing_source/ttl_days/segments from default_params jsonb.
- 114 seed: deactivate_stale_yandex + deactivate_stale_cian (vtorichka, TTL=30).
- segments is None => all segments; [] => none (ANY(ARRAY[]) matches nothing).

Refs #759
2026-06-16 18:58:38 +03:00