Commit graph

5 commits

Author SHA1 Message Date
f7deed096b fix(scrapers): avito per-card parse_failures counter + logging (Refs #823) (#830)
Some checks failed
Deploy Trade-In / changes (push) Successful in 4s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 25s
Deploy Trade-In / build-backend (push) Successful in 1m11s
Deploy Trade-In / deploy (push) Has been cancelled
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 20:01:50 +00:00
e94929159b fix(scrapers): avito listing_date from sortTimeStamp JSON (#726) (#729)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 24s
Deploy Trade-In / build-backend (push) Successful in 39s
Deploy Trade-In / deploy (push) Successful in 34s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-30 14:13:38 +00:00
81bf7e6def fix(tradein-imv): harden Avito IMV anti-bot path + offline contract tests (#565)
Add warm-up GET /evaluation/realty + production browser header set (Sec-Fetch,
X-Requested-With, 25s timeout) mirroring avito.py, чтобы защитить 3-step IMV flow
от server-IP анти-бот 403. Refactor step B/C парсинга в pure helpers
(_parse_geo_position/_parse_price/_parse_placement_history/_parse_suggestions)
+ fixture-driven offline тесты из живых JSON-captures.

Честный диагноз: контракт (geoFieldsHash, price-блок, categoryId=24) НЕ менялся —
проверено живым capture'ом с этой машины (A→B→C, recommended 7.39M). Реальная
причина #565 — скорее анти-бот по PROD-IP, не код → если #565 останется после
deploy, настоящий фикс = ротация egress IP воркера (devops, отдельно).

code-reviewer:  APPROVE no critical — warm-up non-fatal, injected-session path
unchanged, refactor preserves shapes, graceful-None сохранён, fixtures без PII.
Tests: 12 new (27 total). Ruff clean.

Closes #565
2026-05-28 16:29:05 +05:00
a3f45a6722 fix(tradein): decode Avito houses URL-encoded __preloadedState__ (#566)
Avito switched the houses-catalog state blob from JSON.parse("<js-escaped>")
to a URL-encoded string literal, wrapped the 10 widgets in
placeholders[0].widgets, и re-nested developmentData под props.housePage.
JS-unescape path silently failed → houses rows перестали наполняться
(starve'ит #6 same-building + #569 deal centroids).

Extract decode в pure _extract_preloaded_state (urllib.parse.unquote,
JSON.parse fallback для старой формы), add _flatten_widgets, и сделать
_parse_house_page tolerant к обеим developmentData/ratingPreview формам.
Real-HTML fixture (Yumasheva-6 EKB) + 4 regression-теста.

code-reviewer:  APPROVE no critical (regex anchoring verified, decode
safe — no eval/unicode_escape, backward-compat реальный, delisted-house
envelope degrade'ит gracefully, fixture без PII/secrets). Tests: 22 passed.

Closes #566
2026-05-28 15:58:19 +05:00
02ad9a1f58 feat(tradein): Phase 1 of #582 — address mismatch audit infrastructure
Stratified 200-house Yandex reverse-geocode audit comparing our DB's
(address, lat/lon) pair against what Yandex.Карты returns for those
coordinates. Лежит фундамент данных для Phases 2-5 (canonical = Yandex
per user decision).

What ships:
- Migration 066: address_mismatch_audit table + FDW foreign table
  gendesign_ekb_districts_geom для stratified sampling.
- Sampling SQL: 8 EKB districts × 25 houses = 200.
- Driver scripts/audit_address_mismatch.py с двумя режимами:
  API (если YANDEX_GEOCODER_API_KEY есть) и Playwright (CAPTCHA-aware,
  персистентный контекст). Resumable по --batch.
- Report SQL: p50/p75/p95 distance, top-20 outliers, per-district breakdown.
- 19 unit-тестов: normalize, distance bind, API parser, resume idempotency,
  captcha → status=blocked.

Devops note: после деплоя миграции — выдать tradein_fdw_reader SELECT на
gendesign.public.ekb_districts_geom перед первым запуском audit.

Refs #582 (Phase 1 of 5). Phase 3 backfill заблокирован до Yandex API
key (#402).
2026-05-25 12:50:30 +05:00