gendesign/tradein-mvp/backend/app/services
bot-backend d6dca24ea2 fix(scrapers): avito listing_date from sortTimeStamp JSON (#726)
Investigated with a real curl_cffi capture (HTTP 200, 46 cards): the Avito
SERP lazy-renders — data-marker="item-date" appears only on the top
above-the-fold cards (~10/46), so ~79% of cards have no date in the raw HTML
curl_cffi receives. The marker wasn't renamed; the field is just absent.

But every listing carries sortTimeStamp (epoch-ms) in the embedded per-item
JSON (56 occurrences/page) — the line-266 'state empty' note was stale. Map
item_id (= DOM data-item-id) -> sortTimeStamp -> date as the primary source,
keep the DOM item-date + _parse_relative_date as fallback for the top cards.

Verified on the real capture: 42/46 = 91% coverage (was ~21% DOM-only),
clearing the >80% DoD bar. Adds a distilled real-structure fixture
(avito_serp_sample.html) + unit test covering both the JSON (lazy card) and
DOM (top card) paths.

Refs #726
2026-05-30 17:09:41 +03:00
..
exporters chore(tradein): close pre-commit ruff gap + add CI pytest gate (#666) (#675) 2026-05-29 19:18:48 +00:00
matching feat(tradein): same-building матчинг аналогов по house_id (#6) 2026-05-28 13:55:51 +05:00
scrapers fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) 2026-05-30 17:09:41 +03:00
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
account_quota.py feat(tradein): configurable estimate quota + new exhausted-quota copy (#658) (#671) 2026-05-29 16:07:02 +00:00
brand.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
cache.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
cian_session.py fix(tradein): CIAN_REQUIRED_COOKIES — add real auth cookies (DMIR_AUTH + Cian session) (#480) 2026-05-23 14:52:24 +00:00
dadata.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
estimator.py fix(tradein): coarse-geo downgrade no longer shielded by Tier C anchor (#693) (#720) 2026-05-30 12:34:11 +00:00
geocoder.py feat(tradein): DaData /suggest tier в /suggest endpoint (PR Q2) 2026-05-27 19:08:58 +05:00
house_metadata.py feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) 2026-05-24 08:57:30 +00:00
image_sanitizer.py fix(tradein-photos): sanitize uploaded images via Pillow re-encode (#511) 2026-05-24 11:32:04 +00:00
scheduler.py feat(tradein): daily asking→sold ratio refresh (#648 S4) (#663) 2026-05-29 13:49:36 +00:00
scrape_pipeline.py feat(scheduler): dormant Yandex Realty city-sweep (#561) (#642) 2026-05-29 08:50:10 +00:00
scrape_runs.py fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487) 2026-05-23 20:08:40 +00:00
scraper_settings.py feat(tradein): global scraper delay setting (applies across all scrapers) (#485) 2026-05-23 15:43:28 +00:00
search_query.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00