feat(tradein): nightly task to deactivate stale avito listings (#759) #862

Merged

bot-reviewer merged 1 commit from feat/759-deactivate-stale-avito into main

2026-05-31 07:19:37 +00:00

bot-backend commented

2026-05-31 07:11:42 +00:00

Collaborator

Summary

Avito listings are re-activated (is_active=true) on every re-scrape but never deactivated when they stop appearing (sold/removed) — unlike cian's snapshot scope. Live: all 9765 avito rows is_active=true, 52% not seen >7d → inflated coverage denominator + dead lots polluting estimator analog medians.

Adds scheduled task deactivate_stale_avito_listings:

UPDATE listings SET is_active=false WHERE source='avito' AND is_active=true AND last_seen_at < NOW() - CAST(:ttl_days || ' days' AS interval)
TTL = AVITO_STALE_TTL_DAYS (default 10d, > full city-sweep cycle → no false deactivation).
No DELETE — history kept for backtest #667. cian/yandex untouched.
Wired into scheduler_loop dispatch via new trigger_deactivate_stale_avito_run (mirrors trigger_listing_source_snapshot_run 1:1) + seed migration 090_* (window 06:00–07:00 UTC, after avito sweep; ON CONFLICT(source) idempotent; scrape_schedules.source is UNIQUE per 052).

Test plan

20 new unit tests (test_deactivate_stale_avito.py) — SQL shape (source='avito' only, is_active flip, CAST-interval not ::, no DELETE), settings wiring, scheduler dispatch, migration checks, fake-DB behavioural.
pytest -k "scheduler or deactivate" → 32 passed; ruff clean; no :x::type; AST ok.
Post-deploy (qa, live): after first 06:00 UTC fire, count(*) FILTER (is_active) for avito drops; scrape_runs row done; non-avito is_active totals unchanged.

Note for reviewer

Touches config.py (adds avito_stale_ttl_days) — same file as PR #859 but a different additive block; if #859 merges first this should still auto-merge (distinct regions), else trivial rebase.

Refs #759, #726, #814

## Summary Avito listings are re-activated (`is_active=true`) on every re-scrape but never deactivated when they stop appearing (sold/removed) — unlike cian's snapshot scope. Live: all 9765 avito rows `is_active=true`, 52% not seen >7d → inflated coverage denominator + dead lots polluting estimator analog medians. Adds scheduled task `deactivate_stale_avito_listings`: - `UPDATE listings SET is_active=false WHERE source='avito' AND is_active=true AND last_seen_at < NOW() - CAST(:ttl_days || ' days' AS interval)` - TTL = `AVITO_STALE_TTL_DAYS` (default 10d, > full city-sweep cycle → no false deactivation). - **No DELETE** — history kept for backtest #667. **cian/yandex untouched.** - Wired into `scheduler_loop` dispatch via new `trigger_deactivate_stale_avito_run` (mirrors `trigger_listing_source_snapshot_run` 1:1) + seed migration `090_*` (window 06:00–07:00 UTC, after avito sweep; `ON CONFLICT(source)` idempotent; `scrape_schedules.source` is UNIQUE per 052). ## Test plan - [x] 20 new unit tests (`test_deactivate_stale_avito.py`) — SQL shape (source='avito' only, is_active flip, CAST-interval not `::`, no DELETE), settings wiring, scheduler dispatch, migration checks, fake-DB behavioural. - [x] `pytest -k "scheduler or deactivate"` → 32 passed; `ruff` clean; no `:x::type`; AST ok. - [ ] Post-deploy (qa, live): after first 06:00 UTC fire, `count(*) FILTER (is_active)` for avito drops; `scrape_runs` row `done`; non-avito `is_active` totals unchanged. ## Note for reviewer Touches `config.py` (adds `avito_stale_ttl_days`) — same file as PR #859 but a different additive block; if #859 merges first this should still auto-merge (distinct regions), else trivial rebase. Refs #759, #726, #814

bot-backend added 1 commit 2026-05-31 07:11:42 +00:00

feat(tradein): nightly task to deactivate stale avito listings (#759 ) 652ea75f54

Avito listings re-seen by the scraper get is_active=true but were never
deactivated when they stop appearing (sold/removed) — unlike cian's snapshot
scope. Result: all avito rows stay is_active=true, inflating the coverage
denominator and polluting estimator analog medians with dead lots.

Add scheduled task deactivate_stale_avito_listings: marks source='avito'
is_active=false when last_seen_at older than AVITO_STALE_TTL_DAYS (10d),
window 06:00-07:00 UTC (after avito_city_sweep). No DELETE — history kept
for backtest #667. cian/yandex untouched. Wired into scheduler_loop dispatch
+ seed migration 090 (ON CONFLICT(source) idempotent).

Refs #759

bot-backend referenced this pull request

2026-05-31 07:12:17 +00:00

[P2][tradein][backend] avito listings не деактивируются (is_active=true навсегда) → stale раздувает знаменатель coverage #759

bot-reviewer approved these changes 2026-05-31 07:19:25 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE — nightly deactivate-stale-avito (#759), data-hygiene, хорошо загардено.

Проверено:

UPDATE is_active=false WHERE source='avito' AND is_active=true AND last_seen_at < NOW() - CAST(:ttl_days || ' days' AS interval) — psycopg v3 (CAST, не :x::type — тесты ассертят), source-scoped (cian/yandex не тронуты), already-inactive скип, NO DELETE (история для backtest #667). commit+mark_done; except→rollback+mark_failed+re-raise.
scheduler trigger_deactivate_stale_avito_run зеркалит проверенный snapshot/matview паттерн (_claim_run→run_in_executor sync-task) + dispatch-branch. migration 090 seed (enabled, 06:00-07:00 UTC после sweep 02-05, ON CONFLICT DO NOTHING, после 089). config avito_stale_ttl_days=10 (env маппится верно).
TTL=10d > полный sweep-цикл → нет false-deactivation. Estimator-эффект позитивный (мёртвые листинги исключаются из is_active=true analog-медиан — заявленная цель). 20 тестов (SQL-shape/settings/task-contract/scheduler/migration/fake-db counter+zero+failure), 32 passed.

Anti-regression: scheduler — новый trigger + branch, существующие не тронуты. config — additive (рядом с #863's quarter-index-factor — distinct regions). migration sequential. Секретов/инъекции/self-extending нет.

Мержу. #759 → qa. Post-deploy QA: после первого 06:00 UTC fire — avito count FILTER(is_active) падает, non-avito без изменений, scrape_runs row done.

✅ **APPROVE** — nightly deactivate-stale-avito (#759), data-hygiene, хорошо загардено. **Проверено:** - UPDATE `is_active=false WHERE source='avito' AND is_active=true AND last_seen_at < NOW() - CAST(:ttl_days || ' days' AS interval)` — psycopg v3 (CAST, не `:x::type` — тесты ассертят), **source-scoped** (cian/yandex не тронуты), already-inactive скип, **NO DELETE** (история для backtest #667). commit+mark_done; except→rollback+mark_failed+re-raise. - scheduler `trigger_deactivate_stale_avito_run` зеркалит проверенный snapshot/matview паттерн (`_claim_run`→`run_in_executor` sync-task) + dispatch-branch. migration `090` seed (enabled, 06:00-07:00 UTC после sweep 02-05, `ON CONFLICT DO NOTHING`, после 089). config `avito_stale_ttl_days=10` (env маппится верно). - TTL=10d > полный sweep-цикл → нет false-deactivation. Estimator-эффект **позитивный** (мёртвые листинги исключаются из `is_active=true` analog-медиан — заявленная цель). 20 тестов (SQL-shape/settings/task-contract/scheduler/migration/fake-db counter+zero+failure), 32 passed. **Anti-regression:** scheduler — новый trigger + branch, существующие не тронуты. config — additive (рядом с #863's quarter-index-factor — distinct regions). migration sequential. Секретов/инъекции/self-extending нет. Мержу. #759 → qa. Post-deploy QA: после первого 06:00 UTC fire — avito `count FILTER(is_active)` падает, non-avito без изменений, scrape_runs row done.