Commit graph

14 commits

Author SHA1 Message Date
04550bc94f fix(tradein): drop non-EKB avito detail coords on ingest (audit #1871 P2)
All checks were successful
CI / changes (pull_request) Successful in 5s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
avito_detail.parse_detail_html писал data-map-lat/lon прямо в listings
UPDATE COALESCE без bbox-валидации — в отличие от geocoder.py, который
фильтрует везде. Итог: 406 active avito-листингов с координатами
Питер/Тюмень/Уфа просочились в ЕКБ-датасет (ложные аналоги через Tier A
address-match оценщика). Prod-verified 2026-06-23.

- geocoder: DRY-хелпер is_within_ekb_bbox(lat, lon, bbox) + именованные
  EKB_BBOX_TIGHT (geocoder-фильтр, числа не изменены) и EKB_BBOX_WIDE
  (ingest-guard); 2 inline tight-проверки → хелпер (поведение identical).
- avito_detail: после извлечения lat/lon → reset None + logger.warning
  если вне wide-bbox; листинг падает в geocode-cron путь (COALESCE NULL
  не затирает прежнее хорошее значение).
- tests: bbox-хелпер (ЕКБ True, Питер/Тюмень/Уфа False, inclusive,
  WIDE⊇TIGHT инвариант) + parse_detail_html guard end-to-end.

WIDE строго содержит TIGHT — guard физически не режет ничего, что
geocoder сам бы пропустил. NULL lat (14935, geocode backlog) — отдельная
проблема, не в scope. One-time cleanup существующих 406 — отдельно
(database-expert, prod data mutation).

Refs #1871
2026-06-23 14:33:10 +05:00
37fc50ad08 fix(scrapers): avito_detail_backfill — снапшот ЕКБ+active, detail browser-only без curl-fallback на бан (#1814)
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-06-19 21:58:35 +03:00
d65a88f704 fix(avito): reconnect-retry on detail 429 under backconnect proxy
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Detail backfill (run 217) hit 5 consecutive HTTP 429s and aborted at 19 enriched
because the detail fetcher only had a same-session 429 short-retry (no reconnect).
429s are persistent: keep-alive reuses one backconnect exit IP across sequential
detail fetches -> avito rate-limits it. Mirror the merged SERP 429 fix (#1769):
widen short-retry 4->8 and add up to 3 ephemeral-session reconnects (fresh exit IP)
on 429 exhaustion before raising. Caller session never touched (as with 403 path).
2026-06-18 23:29:47 +03:00
c993243bc7 fix(avito): reconnect-retry on detail 403/firewall under backconnect proxy
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
fetch_detail raised on the first 403/429, aborting the detail backfill (run 191
enriched only 48 then zombied) -> avito detail coverage stuck at 14% of active.
Mirror the merged SERP fix (#1765): on 403/firewall under the backconnect proxy,
rebuild an ephemeral session (fresh exit IP) and retry up to N before raising;
short-retry 429. Never close/rebuild the caller-provided shared session. Also
add the missing HTTP-200 firewall-interstitial check on the curl detail path.
2026-06-18 22:12:03 +03:00
22a7a3f086 fix(avito): classify novostroyki by item-development-name marker; map переуступка sale_type
All checks were successful
CI / changes (pull_request) Successful in 6s
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
2026-06-17 22:24:13 +03:00
5285a03c98 fix(tradein): avito detail parser — добавить living_area_m2/ceiling_height_m/finishing
All checks were successful
CI / frontend-tests (push) Has been skipped
CI / openapi-codegen-check (push) Has been skipped
CI / changes (push) Successful in 7s
CI / backend-tests (push) Has been skipped
STEP 0 (живая проверка 2026-06-16): три страницы EKB квартир вернули HTTP 200.
Подтверждены метки: "Жилая площадь" (напр. "12.6 м²"), "Высота потолков"
("2.7 м"), "Отделка" (напр. "чистовая"). "Стоимость ремонта" — не обнаружена.

- RUS_FIELD_MAP: добавлены три записи.
- DetailEnrichment: новые поля living_area_m2/ceiling_height_m/finishing
  (в dataclass attrs и в locals-блоке parse_detail_html).
- parse_detail_html: ветки elif для трёх полей; helper _parse_height_m().
- save_detail_enrichment: COALESCE-строки + params-dict для новых полей.
- migration 111: ADD COLUMN IF NOT EXISTS ceiling_height_m, finishing
  (living_area_m2 уже существовала, migration не нужна).
- Тесты: 5 новых кейсов в test_avito_detail_parse.py + 5 unit-тестов _parse_height_m.
  Полный suite: 1795 passed, 2 deselected.
2026-06-16 10:19:09 +03:00
92f44ac968 feat(tradein): route avito pipeline SERP+detail+houses through BrowserFetcher (#915) (#916)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 46s
Deploy Trade-In / deploy (push) Successful in 37s
Deploy Trade-In / build-browser (push) Has been skipped
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 18:10:38 +00:00
70838c7dfe fix(tradein): avito house-link из detail SSR — SERP-карточка JS-only (#871) (#873)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 30s
Deploy Trade-In / build-backend (push) Successful in 41s
Deploy Trade-In / deploy (push) Successful in 36s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-05-31 08:27:47 +00:00
4494866574 fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 28s
Deploy Trade-In / deploy (push) Successful in 35s
Deploy Trade-In / build-backend (push) Successful in 40s
Co-authored-by: bot-backend <bot-backend@gendsgn.local>
Co-committed-by: bot-backend <bot-backend@gendsgn.local>
2026-05-31 07:04:31 +00:00
312df8536a feat(tradein): infer repair_state from listing description text (#622) (#632)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / deploy (push) Successful in 38s
Deploy Trade-In / build-backend (push) Successful in 48s
2026-05-28 15:37:34 +00:00
d842811390 refactor(tradein): normalize repair_state raw→enum at ingest (#621) (#629)
All checks were successful
Deploy Trade-In / changes (push) Successful in 5s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 49s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-28 15:18:11 +00:00
8fc309d9ad fix(tradein-avito): strip Emotion CSS from listings.address (#502)
Some checks failed
Deploy Trade-In / build-backend (push) Blocked by required conditions
Deploy Trade-In / build-frontend (push) Blocked by required conditions
Deploy Trade-In / deploy (push) Blocked by required conditions
Deploy Trade-In / changes (push) Has been cancelled
2026-05-24 10:52:50 +00:00
222389cba9 fix(tradein/avito): anti-bot hardening — sleep + abort + shared session (#487)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 40s
Deploy Trade-In / deploy (push) Successful in 36s
2026-05-23 20:08:40 +00:00
7cdde30b13 feat(tradein): avito_detail.py — detail page parser (30+ fields incl. domoteka) (#443)
All checks were successful
Deploy Trade-In / changes (push) Successful in 6s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-backend (push) Successful in 38s
Deploy Trade-In / deploy (push) Successful in 17s
2026-05-23 12:17:15 +00:00