gendesign/tradein-mvp/backend/app/services/scrapers
bot-backend a1ee06ef17 feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007)
house_type-нормализация = реальный фикс estimator soft-penalty: yandex SERP
писал building.buildingType как есть (SCREAMING — MONOLIT/BRICK/PANEL/...),
а estimator штрафует аналог с house_type != target. 'MONOLIT' никогда не равен
каноничному 'monolith' → ~70% yandex-аналогов получали ложный штраф и фактически
выпадали из скоринга. Новый модуль house_type_normalizer.normalize_house_type
приводит yandex SCREAMING и cian camelCase (reuse в #2008) к канону
{panel, brick, monolith, monolith_brick, block, wood}; unknown/'other'/'' → None
(NULL нейтрально для estimator, не штрафуется). Backfill существующих yandex-строк
миграцией 140 (идемпотентный UPDATE под тот же маппинг).

kitchen_area_m2 / ceiling_height_m promote из raw_payload в колонки — для
coverage-дашборда и matching, НЕ valuation-фича (estimator их пока не читает —
отдельный follow-up). ceiling пишется в ОБЕ колонки (ceiling_height 019 — её
читает coverage-дашборд admin.py + yandex_detail/cian_detail; ceiling_height_m
111 — живая avito-колонка) через COALESCE в ON CONFLICT, чтобы SERP re-scrape
без полей не затирал detail-enrichment (avito поведение сохранено).
2026-06-27 20:30:27 +03:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py fix(avito): post-merge polish — unify fetch_mode guard + empty-render guard (#1811) (#1909) 2026-06-26 07:44:22 +00:00
avito_detail.py fix(tradein): drop non-EKB avito detail coords on ingest (audit #1871 P2) 2026-06-23 14:33:10 +05:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(scrapers): avito ban-triggered IP rotation (#1790) — changeip on block, bounded 2026-06-20 21:24:12 +03:00
avito_imv.py feat(tradein-imv): route IMV backfill через BrowserFetcher.fetch_json за dormant-флагом (#915 Stage 3) (#1916) 2026-06-26 09:54:28 +00:00
base.py feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007) 2026-06-27 20:30:27 +03:00
browser_fetcher.py feat(tradein-imv): route IMV backfill через BrowserFetcher.fetch_json за dormant-флагом (#915 Stage 3) (#1916) 2026-06-26 09:54:28 +00:00
cian.py fix(cian): recover new-build house via cadastral anchor + listings (source,source_id) dedup (#1773) (#1912) 2026-06-26 07:45:22 +00:00
cian_detail.py fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) 2026-06-19 19:57:18 +03:00
cian_newbuilding.py feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) 2026-06-18 09:01:50 +03:00
cian_state_parser.py fix(tradein): cian _cianConfig concat-формат + auth из header-frontend (#639) (#819) 2026-05-30 19:08:01 +00:00
cian_valuation.py fix(tradein): Mera-audit fix-1 — Cian valuation price sanity bounds 2026-06-21 09:02:32 +03:00
domclick.py feat(tradein/domclick): enrich Layer A SERP parser — description, repair-inference, minhash, seller 2026-06-27 16:35:25 +03:00
domclick_exceptions.py fix(tradein/domclick): rewrite dead scraper to BFF JSON API + honest status (#1846, #1968) 2026-06-27 13:57:07 +03:00
ekb_geoportal_client.py fix(tradein): verify=False for EKB geoportal WFS (Russian-CA cert) 2026-06-19 18:19:03 +03:00
house_type_normalizer.py feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007) 2026-06-27 20:30:27 +03:00
price_brackets.py perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city) 2026-06-18 17:55:55 +03:00
repair_state_normalizer.py fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) 2026-06-19 19:57:18 +03:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py fix(scrapers): yandex_detail — area/ceiling из структурного блока, покрытие area 63%->~95% (#1792) 2026-06-19 19:37:51 +03:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) 2026-06-18 09:01:50 +03:00
yandex_realty.py feat(tradein/yandex): normalize house_type + promote kitchen/ceiling to columns (#2007) 2026-06-27 20:30:27 +03:00
yandex_valuation.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00