gendesign/tradein-mvp/backend/app/services/scrapers
bot-backend d6dca24ea2 fix(scrapers): avito listing_date from sortTimeStamp JSON (#726)
Investigated with a real curl_cffi capture (HTTP 200, 46 cards): the Avito
SERP lazy-renders — data-marker="item-date" appears only on the top
above-the-fold cards (~10/46), so ~79% of cards have no date in the raw HTML
curl_cffi receives. The marker wasn't renamed; the field is just absent.

But every listing carries sortTimeStamp (epoch-ms) in the embedded per-item
JSON (56 occurrences/page) — the line-266 'state empty' note was stale. Map
item_id (= DOM data-item-id) -> sortTimeStamp -> date as the primary source,
keep the DOM item-date + _parse_relative_date as fallback for the top cards.

Verified on the real capture: 42/46 = 91% coverage (was ~21% DOM-only),
clearing the >80% DoD bar. Adds a distilled real-structure fixture
(avito_serp_sample.html) + unit test covering both the JSON (lazy card) and
DOM (top card) paths.

Refs #726
2026-05-30 17:09:41 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) 2026-05-30 17:09:41 +03:00
avito_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
avito_exceptions.py fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487) 2026-05-23 20:08:40 +00:00
avito_houses.py fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566) 2026-05-28 15:58:19 +05:00
avito_imv.py fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565) 2026-05-28 16:29:05 +05:00
base.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
cian.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
cian_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
cian_newbuilding.py feat(tradein): houses.cian_zhk_url column + backfill (#568) (#627) 2026-05-28 13:57:43 +00:00
cian_state_parser.py feat(tradein): cian_state_parser shared utility + ScrapedLot Cian fields (#447) 2026-05-23 12:35:01 +00:00
cian_valuation.py fix(tradein/cian): sanitize valuation cache writes (047 UNIQUE + 026/044 cols) (#523) 2026-05-24 13:44:39 +00:00
n1.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
repair_state_normalizer.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(tradein): scraper_settings live-config + Yandex admin trigger endpoints (#484) 2026-05-23 15:28:34 +00:00
yandex_realty.py feat(tradein): infer repair_state from listing description text (#622) (#632) 2026-05-28 15:37:34 +00:00
yandex_valuation.py fix(yandex-valuation): off-by-one cell index in _parse_row_cells (#552) 2026-05-24 19:57:17 +00:00