gendesign/tradein-mvp/backend/app/services/scrapers
Light1YT 282243fb97
All checks were successful
CI / changes (pull_request) Successful in 11s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
fix(cian): recover new-build house number via cadastral anchor + listings (source,source_id) dedup (#1773)
PART 1 — cian-новостройки теряли номер дома: geo.address[] для ЖК часто без
части {type:"house"}, адрес выходил house-less («…улица Евгения Савкова» без
«29») и раздробленные корпуса схлопывались в catch-all → same-building anchor
рвался. Теперь для новостроек без house восстанавливаем номер по приоритету:
(a) building.houseNumber / building.address, (b) building_cadastral_number как
anchor-ключ, (c) хвостовой дом-токен из newbuilding.name — без catch-all
фабрикации. building_cadastral_number предпочтён как house anchor
(house_source=cian_building_cad) → Tier 0 cadastr_exact схлопывает корпуса;
nb_id остаётся fallback'ом.

PART 2 — дубль-листинги: миграция 133 дедупит listings по (source, source_id)
(keeper = house_id_fk → активность → свежесть → id), безопасно перепривязывает
детей (price_history/snapshots/listing_sources/external_valuations/merged_into)
collision-safe и вешает частичный UNIQUE (source, source_id) WHERE source_id IS
NOT NULL. Идемпотентно, недеструктивно для keeper. save_listings оставлен на
ON CONFLICT (dedup_hash) — арбитр согласован с новым constraint, source_id=NULL
(yandex/url-only) не ломается.

Refs #1773
2026-06-26 12:29:25 +05:00
..
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
avito.py fix(avito): graceful sidecar HTTP 500/503 in browser SERP fetch (#1790) 2026-06-25 06:47:41 +00:00
avito_detail.py fix(tradein): drop non-EKB avito detail coords on ingest (audit #1871 P2) 2026-06-23 14:33:10 +05:00
avito_exceptions.py fix(tradein): avito content-block (HTTP-200) не маскируется под end-of-pagination (Refs #754) (#779) 2026-05-30 16:28:23 +00:00
avito_houses.py fix(scrapers): avito ban-triggered IP rotation (#1790) — changeip on block, bounded 2026-06-20 21:24:12 +03:00
avito_imv.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00
base.py fix(cian): recover new-build house number via cadastral anchor + listings (source,source_id) dedup (#1773) 2026-06-26 12:29:25 +05:00
browser_fetcher.py fix(tradein/yandex): инкрементальный per-combo save + bounded gate-timeout (анти-зомби city_sweep) 2026-06-20 20:57:41 +03:00
cian.py fix(cian): recover new-build house number via cadastral anchor + listings (source,source_id) dedup (#1773) 2026-06-26 12:29:25 +05:00
cian_detail.py fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) 2026-06-19 19:57:18 +03:00
cian_newbuilding.py feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) 2026-06-18 09:01:50 +03:00
cian_state_parser.py fix(tradein): cian _cianConfig concat-формат + auth из header-frontend (#639) (#819) 2026-05-30 19:08:01 +00:00
cian_valuation.py fix(tradein): Mera-audit fix-1 — Cian valuation price sanity bounds 2026-06-21 09:02:32 +03:00
domclick.py feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) 2026-06-18 09:01:50 +03:00
domclick_exceptions.py fix(tradein): domklik source name, scraper delay, DataDome block detect (#796 fixup) 2026-05-31 17:21:30 +03:00
ekb_geoportal_client.py fix(tradein): verify=False for EKB geoportal WFS (Russian-CA cert) 2026-06-19 18:19:03 +03:00
price_brackets.py perf(tradein): shared EKB seed price-brackets across avito/cian/yandex (city-dict ready for multi-city) 2026-06-18 17:55:55 +03:00
repair_state_normalizer.py fix(scrapers): cian_detail — ceiling/repair актуальные пути state (#1791) 2026-06-19 19:57:18 +03:00
snapshot_writer.py fix(tradein): implement listings_snapshots + offer_price_history writers (#567) (#624) 2026-05-28 13:49:52 +00:00
yandex_detail.py fix(scrapers): yandex_detail — area/ceiling из структурного блока, покрытие area 63%->~95% (#1792) 2026-06-19 19:37:51 +03:00
yandex_helpers.py fix(tradein): Yandex listing_date — keyword + absolute-no-year (#602) 2026-05-27 13:11:22 +00:00
yandex_newbuilding.py feat(browser): per-source proxy pool behind FEATURE_BROWSER_POOL_ENABLED (Phase 1) 2026-06-18 09:01:50 +03:00
yandex_realty.py fix(tradein/yandex): инкрементальный per-combo save + bounded gate-timeout (анти-зомби city_sweep) 2026-06-20 20:57:41 +03:00
yandex_valuation.py fix(tradein): route yandex/valuation/imv/newbuilding scrapers через scraper_proxy_url (#860) 2026-05-31 07:04:31 +00:00