0e0911dd28
All checks were successful
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 10s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m2s
Avito emits 1970-epoch and near-future listing_date; a single guard in save_listings() nulls anything outside [2010-01-01, today+2d], covering all providers. + migration 181 nulls existing bad rows. LOW audit R2 (#6).
60 lines
2.1 KiB
Python
60 lines
2.1 KiB
Python
"""LOW audit R2 (#6) — `_clamp_plausible_date` в `scraper_kit.base`.
|
||
|
||
Choke point для `listing_date`/`publish_date` в `save_listings()` (единый для
|
||
avito/cian/yandex/domclick). Проверено в БД: avito пишет 1970-01-01
|
||
(epoch-sentinel, ~698 строк) и даты до +11 дней в будущем (~79 строк); у
|
||
yandex — единичные аномально старые годы. Правдоподобный диапазон —
|
||
[2010-01-01, today+2d]; всё остальное → None (колонка nullable).
|
||
|
||
Офлайн: без сети/curl_cffi/БД, только импорт + чистая функция.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
import os
|
||
from datetime import date, timedelta
|
||
|
||
os.environ.setdefault("DATABASE_URL", "postgresql+psycopg://test:test@localhost/test_db")
|
||
|
||
from scraper_kit.base import _clamp_plausible_date
|
||
|
||
|
||
def test_avito_epoch_sentinel_rejected() -> None:
|
||
"""1970-01-01 — классический Avito 0-epoch sentinel."""
|
||
assert _clamp_plausible_date(date(1970, 1, 1)) is None
|
||
|
||
|
||
def test_just_before_min_plausible_rejected() -> None:
|
||
assert _clamp_plausible_date(date(2009, 12, 31)) is None
|
||
|
||
|
||
def test_far_future_rejected() -> None:
|
||
assert _clamp_plausible_date(date(2099, 1, 1)) is None
|
||
|
||
|
||
def test_none_passes_through_as_none() -> None:
|
||
assert _clamp_plausible_date(None) is None
|
||
|
||
|
||
def test_normal_recent_date_passes_through() -> None:
|
||
d = date(2024, 6, 1)
|
||
assert _clamp_plausible_date(d) == d
|
||
|
||
|
||
def test_today_passes() -> None:
|
||
today = date.today()
|
||
assert _clamp_plausible_date(today) == today
|
||
|
||
|
||
def test_today_plus_one_day_passes() -> None:
|
||
d = date.today() + timedelta(days=1)
|
||
assert _clamp_plausible_date(d) == d
|
||
|
||
|
||
def test_today_plus_three_days_rejected() -> None:
|
||
assert _clamp_plausible_date(date.today() + timedelta(days=3)) is None
|
||
|
||
|
||
def test_min_plausible_boundary_passes() -> None:
|
||
"""2010-01-01 сам по себе — граница включительно (в окне)."""
|
||
assert _clamp_plausible_date(date(2010, 1, 1)) == date(2010, 1, 1)
|