bot-backend
652ea75f54
Avito listings re-seen by the scraper get is_active=true but were never deactivated when they stop appearing (sold/removed) — unlike cian's snapshot scope. Result: all avito rows stay is_active=true, inflating the coverage denominator and polluting estimator analog medians with dead lots. Add scheduled task deactivate_stale_avito_listings: marks source='avito' is_active=false when last_seen_at older than AVITO_STALE_TTL_DAYS (10d), window 06:00-07:00 UTC (after avito_city_sweep). No DELETE — history kept for backtest #667. cian/yandex untouched. Wired into scheduler_loop dispatch + seed migration 090 (ON CONFLICT(source) idempotent). Refs #759
70 lines
2.9 KiB
Python
70 lines
2.9 KiB
Python
"""Deactivate stale avito listings that have not been seen in >TTL days (#759).
|
||
|
||
Нечасто обновляемые объявления на Avito (last_seen_at < NOW() - TTL) помечаются
|
||
is_active=false, чтобы coverage/estimator denominators не раздувались мёртвыми
|
||
объявлениями. Строки НЕ удаляются — история нужна для бэктеста (#667).
|
||
|
||
Только source='avito'. Cian/Yandex не затрагиваются.
|
||
|
||
Задача синхронная (DB-only, никаких внешних HTTP-вызовов) — запускается in-app
|
||
scheduler'ом через trigger_deactivate_stale_avito_run() (scheduler.py),
|
||
по образцу snapshot_listing_sources / recompute_asking_to_sold_ratios
|
||
(sync task в run_in_executor).
|
||
|
||
TTL берётся из settings.avito_stale_ttl_days (env AVITO_STALE_TTL_DAYS, default 10).
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import logging
|
||
|
||
from sqlalchemy import text
|
||
from sqlalchemy.orm import Session
|
||
|
||
from app.core.config import settings
|
||
from app.services import scrape_runs as runs_mod
|
||
|
||
logger = logging.getLogger(__name__)
|
||
|
||
# UPDATE: помечаем is_active=false для avito-объявлений, не виденных >TTL дней.
|
||
# CAST(:ttl_days || ' days' AS interval) — psycopg v3 safe (никаких :param::type).
|
||
# Только source='avito' и is_active=true — чтобы не трогать уже неактивные строки
|
||
# и не затрагивать cian/yandex.
|
||
_DEACTIVATE_SQL = text(
|
||
"""
|
||
UPDATE listings
|
||
SET is_active = false
|
||
WHERE source = 'avito'
|
||
AND is_active = true
|
||
AND last_seen_at < NOW() - CAST(:ttl_days || ' days' AS interval)
|
||
"""
|
||
)
|
||
|
||
|
||
def deactivate_stale_avito_listings(db: Session, run_id: int) -> dict[str, int]:
|
||
"""Пометить is_active=false все avito-объявления с last_seen_at > TTL дней назад.
|
||
|
||
Sync (вызывается scheduler-триггером в executor, как snapshot_listing_sources).
|
||
Один UPDATE в транзакции. Финализирует scrape_runs (mark_done / mark_failed).
|
||
|
||
Returns {"deactivated": N} — количество обновлённых строк.
|
||
"""
|
||
counters: dict[str, int] = {"deactivated": 0}
|
||
try:
|
||
result = db.execute(_DEACTIVATE_SQL, {"ttl_days": settings.avito_stale_ttl_days})
|
||
counters["deactivated"] = result.rowcount or 0
|
||
|
||
db.commit()
|
||
runs_mod.mark_done(db, run_id, counters)
|
||
logger.info(
|
||
"deactivate_stale_avito run_id=%d done: deactivated=%d (ttl_days=%d)",
|
||
run_id,
|
||
counters["deactivated"],
|
||
settings.avito_stale_ttl_days,
|
||
)
|
||
return counters
|
||
except Exception as exc:
|
||
logger.exception("deactivate_stale_avito run_id=%d failed", run_id)
|
||
db.rollback()
|
||
runs_mod.mark_failed(db, run_id, str(exc)[:1000], counters)
|
||
raise
|