feat(tradein-scrapers): parse listing_date from Avito + Yandex #508

Merged

lekss361 merged 1 commit from feat/tradein-listing-date-avito-yandex into main

2026-05-24 11:17:14 +00:00

lekss361 commented

2026-05-24 11:00:43 +00:00

Owner

Problem

6/10 analogs в a0a0b820-… пришли с listing_date=null, days_on_market=null. → est_days_on_market=null на UI.

Fix

avito.py: _parse_relative_date() regex для relative dates (N дней/недель/месяцев назад). Извлекаем из data-marker=item-date + fallback на полный текст карточки.
yandex_helpers.py: новая parse_listing_date() — absolute RU date (9 мая 2026) + relative fallback (N дней назад).
yandex_realty.py: заменяем parse_ru_date() → parse_listing_date() (backward compat, расширяет coverage).
cian.py не тронут — уже парсит listing_date из addedTimestamp.
Unit test test_avito_relative_date.py: days/weeks/hours/None cases.

Notes

Regex исправлен (изначальный дн[яей] не матчит «дней» — заменён на дн(?:я|ей?)).
Fallback на полный текст карточки Avito покрывает случаи когда data-marker=item-date отсутствует или изменил структуру.

## Problem 6/10 analogs в `a0a0b820-…` пришли с `listing_date=null`, `days_on_market=null`. → `est_days_on_market=null` на UI. ## Fix - `avito.py`: `_parse_relative_date()` regex для relative dates (`N дней/недель/месяцев назад`). Извлекаем из `data-marker=item-date` + fallback на полный текст карточки. - `yandex_helpers.py`: новая `parse_listing_date()` — absolute RU date (`9 мая 2026`) + relative fallback (`N дней назад`). - `yandex_realty.py`: заменяем `parse_ru_date()` → `parse_listing_date()` (backward compat, расширяет coverage). - `cian.py` не тронут — уже парсит `listing_date` из `addedTimestamp`. - Unit test `test_avito_relative_date.py`: days/weeks/hours/None cases. ## Notes - Regex исправлен (изначальный `дн[яей]` не матчит «дней» — заменён на `дн(?:я|ей?)`). - Fallback на полный текст карточки Avito покрывает случаи когда `data-marker=item-date` отсутствует или изменил структуру.

lekss361 added 1 commit 2026-05-24 11:00:43 +00:00

feat(tradein-scrapers): parse listing_date from Avito + Yandex cards 628d8ad08c

60% analogs пришли с listing_date=null → est_days_on_market не считался.
- avito.py: _parse_relative_date() regex для 'N дней/недель/месяцев назад',
  извлекаем из data-marker=item-date + fallback на card text
- yandex_helpers.py: parse_listing_date() — absolute RU date + relative fallback
- yandex_realty.py: заменяем parse_ru_date() → parse_listing_date()
- тест test_avito_relative_date.py: days/weeks/hours/None cases
- cian.py не тронут (уже парсит listing_date из addedTimestamp)

Source: estimate a0a0b820-e8a8-4eee-aa73-0ab3b98ac233 (6/10 null listing_date).

lekss361 commented

2026-05-24 11:17:06 +00:00

Author

Owner

Deep Code Review — PR #508

Status: APPROVE
Files: 4 (P0: 0, P1: 2 scrapers, P2: 1 helper, P3: 1 test) · Lines: +126 / -3 · SHA 628d8ad

Summary

Narrow, well-scoped fix for documented data-quality gap (Decision_TradeIn_DataQuality_8PR_Roadmap → est_days_on_market=null root cause). Adds relative-date parser (N дней/недель/месяцев/часов назад) for Avito + Yandex SERP cards, plus absolute-RU-date fallback chain for Yandex. Zero regression risk on existing fields.

Correctness 🎯

Regex validated against 18 real-world phrasings (День/дня/дней, неделю/недели/недель, месяц/месяца/месяцев, час/часа/часов, минут, секунд, год/года/годов) — all behave correctly.
5 лет назад does NOT match (Avito caps at «год»; acceptable gap).
Fallback chain in avito.py (item-date → item-date-info → full-card-text search) is sane; primary data-marker="item-date" runs first so fallback only triggers on selector miss.
parse_listing_date() preserves backward compat: tries parse_ru_date() first (absolute), then relative — no Yandex absolute-date regression.
Month=30d / year=365d are coarse approximations, but days_on_market is downstream-computed as NOW() - listing_date, so the convention is consistent with how the metric is consumed.

Pre-existing concern (NOT introduced by this PR)

base.py UPSERT ON CONFLICT DO UPDATE does not refresh listing_date — pre-fix rows with listing_date=null will stay null even after re-scrape with new logic. Backfill task needed (separate PR; not a blocker for this one).

Architecture 🏗

Minor DRY violation: _REL_DATE_RE + unit-prefix table duplicated verbatim in avito.py and yandex_helpers.py. Could be consolidated into a shared helper. Non-blocking — small, isolated, and the two copies have already been kept in sync.

Tests 🧪

test_avito_relative_date.py covers 4 cases (days/weeks/hours/None+empty+random). Adequate primary coverage.
Gap (medium): Yandex-side parse_listing_date() and _parse_relative_date_yandex() have NO dedicated tests (logic is structurally identical to Avito's, but two copies = two test suites ideally). Recommend adding test_yandex_helpers.py::test_parse_listing_date_* in a follow-up.
Tests use date.today() — micro flake risk at UTC midnight, acceptable for unit scope.

Security 🔒 / Performance ⚡

No findings. Pure parsing, per-card single regex match.

Conventions 📋

from __future__ import annotations ✅, type hints ✅, dict[str, int] 3.12 syntax ✅, ruff-clean line lengths ✅.
yandex_realty.py import switch parse_ru_date → parse_listing_date is the only call-site change — clean.

Vault cross-ref 📚

Aligned with decisions/Decision_TradeIn_DataQuality_8PR_Roadmap (item 3: est_days_on_market=null UX fallback). This PR fixes upstream root cause, restoring real values instead of fallback text.

Recommended follow-ups (non-blocking)

Consolidate duplicated _REL_DATE_RE/_UNIT_DAYS into a single shared helper.
Add test_yandex_helpers.py::test_parse_listing_date_* cases.
Backfill task for pre-existing rows with listing_date=null (separate PR).
Consider extending UPSERT ON CONFLICT DO UPDATE to refresh listing_date when it transitions from null → value.

Verdict: APPROVE — merge any-scope.

## Deep Code Review — PR #508 **Status:** APPROVE **Files:** 4 (P0: 0, P1: 2 scrapers, P2: 1 helper, P3: 1 test) · **Lines:** +126 / -3 · SHA `628d8ad` ### Summary Narrow, well-scoped fix for documented data-quality gap (Decision_TradeIn_DataQuality_8PR_Roadmap → `est_days_on_market=null` root cause). Adds relative-date parser (`N дней/недель/месяцев/часов назад`) for Avito + Yandex SERP cards, plus absolute-RU-date fallback chain for Yandex. Zero regression risk on existing fields. ### Correctness 🎯 - Regex validated against 18 real-world phrasings (День/дня/дней, неделю/недели/недель, месяц/месяца/месяцев, час/часа/часов, минут, секунд, год/года/годов) — all behave correctly. - `5 лет назад` does NOT match (Avito caps at «год»; acceptable gap). - Fallback chain in `avito.py` (`item-date` → `item-date-info` → full-card-text search) is sane; primary `data-marker="item-date"` runs first so fallback only triggers on selector miss. - `parse_listing_date()` preserves backward compat: tries `parse_ru_date()` first (absolute), then relative — no Yandex absolute-date regression. - Month=30d / year=365d are coarse approximations, but `days_on_market` is downstream-computed as `NOW() - listing_date`, so the convention is consistent with how the metric is consumed. ### Pre-existing concern (NOT introduced by this PR) - `base.py` UPSERT `ON CONFLICT DO UPDATE` does **not** refresh `listing_date` — pre-fix rows with `listing_date=null` will stay null even after re-scrape with new logic. **Backfill task needed** (separate PR; not a blocker for this one). ### Architecture 🏗 - **Minor DRY violation**: `_REL_DATE_RE` + unit-prefix table duplicated verbatim in `avito.py` and `yandex_helpers.py`. Could be consolidated into a shared helper. Non-blocking — small, isolated, and the two copies have already been kept in sync. ### Tests 🧪 - `test_avito_relative_date.py` covers 4 cases (days/weeks/hours/None+empty+random). Adequate primary coverage. - **Gap (medium)**: Yandex-side `parse_listing_date()` and `_parse_relative_date_yandex()` have NO dedicated tests (logic is structurally identical to Avito's, but two copies = two test suites ideally). Recommend adding `test_yandex_helpers.py::test_parse_listing_date_*` in a follow-up. - Tests use `date.today()` — micro flake risk at UTC midnight, acceptable for unit scope. ### Security 🔒 / Performance ⚡ - No findings. Pure parsing, per-card single regex match. ### Conventions 📋 - `from __future__ import annotations` ✅, type hints ✅, `dict[str, int]` 3.12 syntax ✅, ruff-clean line lengths ✅. - `yandex_realty.py` import switch `parse_ru_date → parse_listing_date` is the only call-site change — clean. ### Vault cross-ref 📚 - Aligned with `decisions/Decision_TradeIn_DataQuality_8PR_Roadmap` (item 3: `est_days_on_market=null` UX fallback). This PR fixes upstream root cause, restoring real values instead of fallback text. ### Recommended follow-ups (non-blocking) 1. Consolidate duplicated `_REL_DATE_RE`/`_UNIT_DAYS` into a single shared helper. 2. Add `test_yandex_helpers.py::test_parse_listing_date_*` cases. 3. Backfill task for pre-existing rows with `listing_date=null` (separate PR). 4. Consider extending UPSERT `ON CONFLICT DO UPDATE` to refresh `listing_date` when it transitions from null → value. **Verdict: APPROVE — merge any-scope.**