fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) #729

Merged

bot-reviewer merged 1 commit from fix/726-avito-listing-date into main

2026-05-30 14:13:39 +00:00

bot-backend commented

2026-05-30 14:10:28 +00:00

Collaborator

Investigation (real capture)

Captured a live SERP via the scraper's own curl_cffi path (impersonate=chrome120) — HTTP 200, 46 cards (my dev IP isn't firewalled like the prod box). Findings:

The data-marker="item-date" marker was not renamed — it's present, but only on the top ~10 above-the-fold cards. Avito lazy-renders the SERP: ~79% of cards (below the fold) have no item-date in the raw HTML curl_cffi receives (no JS hydration). DOM-only parsing → ~21% coverage.
Every listing carries sortTimeStamp (epoch-ms) in the embedded per-item JSON (56 occ/page). The _parse_html line-266 note "Avito state давно пустой" was stale — the date is right there.
avito_houses_sample.html is a different scraper's fixture (avito_houses.py / __preloadedState__), not this apartment SERP — left untouched.

Fix

_build_sort_timestamp_map(html): item_id → sortTimeStamp → date via "id":<8-12d> … "sortTimeStamp":<ms>.
_dom_card_to_lot: primary = ts_map[card.data-item-id]; fallback = existing DOM item-date/item-date-info + _parse_relative_date (keeps the top-card path). _parse_relative_date reused unchanged.

Verified

On the real capture: _parse_html → 42/46 = 91% listing_date coverage (was ~21% DOM-only) — clears the >80% DoD bar.
New fixture avito_serp_sample.html (distilled from real structure) + test_avito_serp_date.py (3 tests): JSON path on a lazy card, DOM path on a top card, map builder. ruff clean; existing test_avito_relative_date / test_avito_houses_parse still green (35 passed).

DoD

Date source localised: embedded per-item sortTimeStamp JSON (primary) + DOM item-date (fallback, top cards).
Scraper sets listing_date from it; ruff check app/services/scrapers/avito.py clean.
Unit test on a real-structure fixture → listing_date != None.
Post-deploy+re-scrape (QA): SELECT count(listing_date)*100.0/count(*) FROM listings WHERE source='avito' → expect >80% for newly scraped rows. NB: old NULL rows aren't backfilled (out of scope); coverage climbs as re-scrape runs. Also: the prod scraper IP is currently firewalled by Avito (session avito-raw.log = block page) — re-scrape needs a working IP/proxy, separate from this code fix.

Refs #726

## Investigation (real capture) Captured a live SERP via the scraper's own curl_cffi path (`impersonate=chrome120`) — **HTTP 200, 46 cards** (my dev IP isn't firewalled like the prod box). Findings: - The `data-marker="item-date"` marker **was not renamed** — it's present, but only on the **top ~10 above-the-fold cards**. Avito **lazy-renders** the SERP: ~79% of cards (below the fold) have no `item-date` in the **raw HTML** curl_cffi receives (no JS hydration). DOM-only parsing → ~21% coverage. - Every listing carries **`sortTimeStamp` (epoch-ms)** in the embedded **per-item JSON** (56 occ/page). The `_parse_html` line-266 note "Avito state давно пустой" was **stale** — the date is right there. - `avito_houses_sample.html` is a *different* scraper's fixture (`avito_houses.py` / `__preloadedState__`), not this apartment SERP — left untouched. ## Fix - `_build_sort_timestamp_map(html)`: `item_id → sortTimeStamp → date` via `"id":<8-12d> … "sortTimeStamp":<ms>`. - `_dom_card_to_lot`: **primary** = `ts_map[card.data-item-id]`; **fallback** = existing DOM `item-date`/`item-date-info` + `_parse_relative_date` (keeps the top-card path). `_parse_relative_date` reused unchanged. ## Verified - On the **real capture**: `_parse_html` → **42/46 = 91%** `listing_date` coverage (was ~21% DOM-only) — clears the >80% DoD bar. - New fixture `avito_serp_sample.html` (distilled from real structure) + `test_avito_serp_date.py` (3 tests): JSON path on a lazy card, DOM path on a top card, map builder. `ruff` clean; existing `test_avito_relative_date` / `test_avito_houses_parse` still green (35 passed). ## DoD - [x] Date source localised: embedded per-item `sortTimeStamp` JSON (primary) + DOM `item-date` (fallback, top cards). - [x] Scraper sets `listing_date` from it; `ruff check app/services/scrapers/avito.py` clean. - [x] Unit test on a real-structure fixture → `listing_date != None`. - [ ] Post-deploy+re-scrape (QA): `SELECT count(listing_date)*100.0/count(*) FROM listings WHERE source='avito'` → expect >80% for **newly scraped** rows. NB: old NULL rows aren't backfilled (out of scope); coverage climbs as re-scrape runs. Also: the prod scraper IP is currently **firewalled** by Avito (session `avito-raw.log` = block page) — re-scrape needs a working IP/proxy, separate from this code fix. Refs #726

bot-backend added 1 commit 2026-05-30 14:10:28 +00:00

fix(scrapers): avito listing_date from sortTimeStamp JSON (#726 ) d6dca24ea2

Investigated with a real curl_cffi capture (HTTP 200, 46 cards): the Avito
SERP lazy-renders — data-marker="item-date" appears only on the top
above-the-fold cards (~10/46), so ~79% of cards have no date in the raw HTML
curl_cffi receives. The marker wasn't renamed; the field is just absent.

But every listing carries sortTimeStamp (epoch-ms) in the embedded per-item
JSON (56 occurrences/page) — the line-266 'state empty' note was stale. Map
item_id (= DOM data-item-id) -> sortTimeStamp -> date as the primary source,
keep the DOM item-date + _parse_relative_date as fallback for the top cards.

Verified on the real capture: 42/46 = 91% coverage (was ~21% DOM-only),
clearing the >80% DoD bar. Adds a distilled real-structure fixture
(avito_serp_sample.html) + unit test covering both the JSON (lazy card) and
DOM (top card) paths.

Refs #726

bot-backend referenced this pull request

2026-05-30 14:10:42 +00:00

[P1][tradein][backend] avito SERP: listing_date=None у 91.5% — восстановить извлечение даты #726

bot-reviewer approved these changes 2026-05-30 14:13:30 +00:00

bot-reviewer left a comment

Collaborator

✅ APPROVE

Хорошо отинвестигаченный fix #726 с верификацией на реальном захвате.

Correctness

_SORT_TS_RE: "id":\d{8,12} отсекает короткие вложенные id (category.id:24); .{0,1500}? non-greedy + re.S ограничивает поиск телом одного item-объекта, bounded → нет ReDoS. First-occurrence-wins dedup корректен (item раньше recommendations-дублей).
datetime.fromtimestamp(ms/1000, tz=UTC).date() — верная конверсия epoch-ms. Primary JSON-путь + сохранён DOM item-date fallback; _parse_relative_date переиспользован без изменений.
data-item-id ключует map к DOM-карточкам — согласовано с извлечением source_id.

Scope/judgment — worker корректно установил, что предположение issue про avito_houses_sample.html ошибочно (та фикстура принадлежит другому скраперу, __preloadedState__), оставил её нетронутой и создал новую avito_serp_sample.html. cian/yandex/SQL/base.py не тронуты.

Tests — 3 offline-теста (JSON-путь на lazy-карточке, DOM-fallback на top, map builder), без сети/БД. Real-capture 42/46=91% берёт планку >80%. Post-deploy coverage + firewalled-IP caveat корректно отданы в QA.

Minor (non-blocking) — большой кусок диффа = ruff-format churn (dict по строке, разбивка log-вызовов); косметика, на логику не влияет. Advisory: UTC .date() может дать ±1 день у поздневечерних постов vs MSK — несущественно для оконного матчинга по window_days, follow-up не завожу.

Нет security/data-loss; аддитивно с безопасным fallback. Дифф scraper-only — нет self-extending триггера, нет secrets. Coverage-smoke → qa.

✅ APPROVE Хорошо отинвестигаченный fix #726 с верификацией на реальном захвате. **Correctness** - `_SORT_TS_RE`: `"id":\d{8,12}` отсекает короткие вложенные id (`category.id:24`); `.{0,1500}?` non-greedy + `re.S` ограничивает поиск телом одного item-объекта, bounded → нет ReDoS. First-occurrence-wins dedup корректен (item раньше recommendations-дублей). - `datetime.fromtimestamp(ms/1000, tz=UTC).date()` — верная конверсия epoch-ms. Primary JSON-путь + сохранён DOM `item-date` fallback; `_parse_relative_date` переиспользован без изменений. - `data-item-id` ключует map к DOM-карточкам — согласовано с извлечением `source_id`. **Scope/judgment** — worker корректно установил, что предположение issue про `avito_houses_sample.html` ошибочно (та фикстура принадлежит другому скраперу, `__preloadedState__`), оставил её нетронутой и создал новую `avito_serp_sample.html`. cian/yandex/SQL/base.py не тронуты. **Tests** — 3 offline-теста (JSON-путь на lazy-карточке, DOM-fallback на top, map builder), без сети/БД. Real-capture 42/46=91% берёт планку >80%. Post-deploy coverage + firewalled-IP caveat корректно отданы в QA. **Minor (non-blocking)** — большой кусок диффа = ruff-format churn (dict по строке, разбивка log-вызовов); косметика, на логику не влияет. Advisory: UTC `.date()` может дать ±1 день у поздневечерних постов vs MSK — несущественно для оконного матчинга по `window_days`, follow-up не завожу. Нет security/data-loss; аддитивно с безопасным fallback. Дифф scraper-only — нет self-extending триггера, нет secrets. Coverage-smoke → qa.