Commit graph

4 commits

Author SHA1 Message Date
d6dca24ea2 fix(scrapers): avito listing_date from sortTimeStamp JSON (#726)
Investigated with a real curl_cffi capture (HTTP 200, 46 cards): the Avito
SERP lazy-renders — data-marker="item-date" appears only on the top
above-the-fold cards (~10/46), so ~79% of cards have no date in the raw HTML
curl_cffi receives. The marker wasn't renamed; the field is just absent.

But every listing carries sortTimeStamp (epoch-ms) in the embedded per-item
JSON (56 occurrences/page) — the line-266 'state empty' note was stale. Map
item_id (= DOM data-item-id) -> sortTimeStamp -> date as the primary source,
keep the DOM item-date + _parse_relative_date as fallback for the top cards.

Verified on the real capture: 42/46 = 91% coverage (was ~21% DOM-only),
clearing the >80% DoD bar. Adds a distilled real-structure fixture
(avito_serp_sample.html) + unit test covering both the JSON (lazy card) and
DOM (top card) paths.

Refs #726
2026-05-30 17:09:41 +03:00
81bf7e6def fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565)
Add warm-up GET /evaluation/realty + production browser header set (Sec-Fetch,
X-Requested-With, 25s timeout) mirroring avito.py, чтобы защитить 3-step IMV flow
от server-IP анти-бот 403. Refactor step B/C парсинга в pure helpers
(_parse_geo_position/_parse_price/_parse_placement_history/_parse_suggestions)
+ fixture-driven offline тесты из живых JSON-captures.

Честный диагноз: контракт (geoFieldsHash, price-блок, categoryId=24) НЕ менялся —
проверено живым capture'ом с этой машины (A→B→C, recommended 7.39M). Реальная
причина #565 — скорее анти-бот по PROD-IP, не код → если #565 останется после
deploy, настоящий фикс = ротация egress IP воркера (devops, отдельно).

code-reviewer:  APPROVE no critical — warm-up non-fatal, injected-session path
unchanged, refactor preserves shapes, graceful-None сохранён, fixtures без PII.
Tests: 12 new (27 total). Ruff clean.

Closes #565
2026-05-28 16:29:05 +05:00
a3f45a6722 fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566)
Avito switched the houses-catalog state blob from JSON.parse("<js-escaped>")
to a URL-encoded string literal, wrapped the 10 widgets in
placeholders[0].widgets, и re-nested developmentData под props.housePage.
JS-unescape path silently failed → houses rows перестали наполняться
(starve'ит #6 same-building + #569 deal centroids).

Extract decode в pure _extract_preloaded_state (urllib.parse.unquote,
JSON.parse fallback для старой формы), add _flatten_widgets, и сделать
_parse_house_page tolerant к обеим developmentData/ratingPreview формам.
Real-HTML fixture (Yumasheva-6 EKB) + 4 regression-теста.

code-reviewer:  APPROVE no critical (regex anchoring verified, decode
safe — no eval/unicode_escape, backward-compat реальный, delisted-house
envelope degrade'ит gracefully, fixture без PII/secrets). Tests: 22 passed.

Closes #566
2026-05-28 15:58:19 +05:00
02ad9a1f58 feat(tradein): Phase 1 of #582 — address mismatch audit infrastructure
Stratified 200-house Yandex reverse-geocode audit comparing our DB's
(address, lat/lon) pair against what Yandex.Карты returns for those
coordinates. Лежит фундамент данных для Phases 2-5 (canonical = Yandex
per user decision).

What ships:
- Migration 066: address_mismatch_audit table + FDW foreign table
  gendesign_ekb_districts_geom для stratified sampling.
- Sampling SQL: 8 EKB districts × 25 houses = 200.
- Driver scripts/audit_address_mismatch.py с двумя режимами:
  API (если YANDEX_GEOCODER_API_KEY есть) и Playwright (CAPTCHA-aware,
  персистентный контекст). Resumable по --batch.
- Report SQL: p50/p75/p95 distance, top-20 outliers, per-district breakdown.
- 19 unit-тестов: normalize, distance bind, API parser, resume idempotency,
  captcha → status=blocked.

Devops note: после деплоя миграции — выдать tradein_fdw_reader SELECT на
gendesign.public.ekb_districts_geom перед первым запуском audit.

Refs #582 (Phase 1 of 5). Phase 3 backfill заблокирован до Yandex API
key (#402).
2026-05-25 12:50:30 +05:00