gendesign/tradein-mvp/backend/app/services
bot-backend eb564cfc01 feat(tradein/scraper-kit): migrate backfill-task imports to kit, Group C (#2310)
Migrates legacy app.services.scrapers.* imports to scraper_kit equivalents for
house_imv_backfill.py, avito_detail_backfill.py, cian_history_backfill.py,
ekb_geoportal_ingest.py, and yandex_detail_backfill.py, proving parity via
tests/support/parity.assert_parity per the epic's gate (#2304).

newbuilding_enrich_backfill.py and yandex_newbuilding_sweep.py are left fully
on legacy imports: both call into scraper_kit.providers.{cian,yandex}.newbuilding,
which construct BrowserFetcher(source=...) without the now-mandatory endpoint=
kwarg (issue #2322, verified still open against the actual provider source, not
just issue status) -- no caller-side fix is possible, matching Group A's (#2305)
precedent for the same bug in admin.py.

Config-gated kit function footguns found and fixed (Group B #2306 pattern):
  - avito fetch_detail's backconnect-on-403 retry silently drops when config=
    is omitted -- now passes config=RealScraperConfig() explicitly, with a
    regression test proving the gate.
  - kit AvitoScraper's constructor now requires ScraperConfig positionally --
    wired via RealScraperConfig(), which _rotate_ip() reads for
    avito_proxy_rotate_url.
  - BrowserFetcher(source=...) call sites (house_imv_backfill, avito/cian
    detail backfills) now pass the mandatory endpoint=settings.browser_http_endpoint.

New footgun discovered (NOT #2322, flagged for follow-up): kit's
build_warmed_session() builds its curl_cffi session via _build_detail_session()
with no config parameter at all, unlike fetch_detail -- migrating it would
silently drop the sticky MGTS-proxy egress on avito_detail_backfill's
warm-batch path (the prod default). Left build_warmed_session/
_AVITO_WARM_SEARCH_URL on legacy imports, documented inline.

Refs #2310
2026-07-04 01:19:34 +03:00
..
exporters fix(tradein/pdf): sold-price consistency + актуальные допущения + свой дизайн отчёта (#2327) 2026-07-03 21:43:40 +00:00
matching fix(tradein/pii): удалить телефоны продавцов (NULL) + PII клиента (DROP) (#2217, #1969) 2026-07-03 23:00:42 +03:00
scrapers fix(tradein/pii): удалить телефоны продавцов (NULL) + PII клиента (DROP) (#2217, #1969) 2026-07-03 23:00:42 +03:00
__init__.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
account_quota.py fix(tradein): atomic conditional quota increment — close TOCTOU (#747) (#749) 2026-05-30 15:52:34 +00:00
brand.py feat: add tradein-mvp subproject (Trade-In Estimator под /trade-in) 2026-05-21 00:25:39 +03:00
buildings_query.py fix(tradein/sale-share): гард по этажности — отсечь near-neighbor mis-match 2026-06-29 00:31:09 +03:00
cache.py feat(tradein): Phase 3.2 — /api/v1/search endpoint + Redis cache (#479) 2026-05-23 14:44:22 +00:00
cian_price_history.py fix(tradein/scrapers): migrate cian_price_history/cian_session/yandex_price_history imports to scraper_kit (#2306) 2026-07-04 00:20:13 +03:00
cian_session.py fix(tradein/scrapers): migrate cian_price_history/cian_session/yandex_price_history imports to scraper_kit (#2306) 2026-07-04 00:20:13 +03:00
dadata.py fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») 2026-06-15 20:21:11 +05:00
estimator.py fix(tradein/estimator): дедуп кросс-source дублей в anchor-пути (#2265 follow-up) (#2311) 2026-07-03 20:35:37 +00:00
gar_flats_loader.py feat(tradein): канонический матч ГАР↔houses (0%→41% yield) + tradein_canon_addr + --match-only (#2063) 2026-06-28 15:50:07 +00:00
geocoder.py feat(tradein/matching): fias_exact tier + estimate fias persist + suggest fias_id contract 2026-07-02 20:49:04 +03:00
house_dedup_merge.py fix(tradein/dedup): drop geo-guard in fias-pass — same-fias identity outranks proximity (#2187) 2026-07-02 21:35:20 +03:00
house_imv_backfill.py feat(tradein/scraper-kit): migrate backfill-task imports to kit, Group C (#2310) 2026-07-04 01:19:34 +03:00
house_metadata.py feat(tradein): postgres_fdw live read of gendesign.cad_buildings (replaces snapshot) (#493) 2026-05-24 08:57:30 +00:00
image_sanitizer.py fix(week-review): автофиксы код-ревью — 169 issue (label «week ревью 1») 2026-06-15 20:21:11 +05:00
location_coef.py feat(tradein): location-coef MVP через FDW-мост к OSM POI Птицы (#2045) 2026-07-03 23:44:55 +03:00
product_handlers.py feat(tradein): location-coef MVP через FDW-мост к OSM POI Птицы (#2045) 2026-07-03 23:44:55 +03:00
proxy_pool.py feat(proxy-pool): pool service (acquire/release/health) + healthcheck scheduler task (#2162) 2026-07-02 16:41:39 +00:00
rosreestr_poll.py feat(tradein): scheduled monthly Rosreestr new-quarter poll → ingest alert (#888) (#894) 2026-05-31 13:13:55 +00:00
sber_index.py fix(tradein): correct SberIndex hedonic/asking dataset-paths + 404 warning (#902) 2026-06-17 22:50:52 +03:00
scheduler.py feat(tradein): location-coef MVP через FDW-мост к OSM POI Птицы (#2045) 2026-07-03 23:44:55 +03:00
scrape_pipeline.py fix(tradein/scrapers): wire proxy_pool into Avito legacy pipeline, rotate on ban (#2202) (#2328) 2026-07-03 21:48:15 +00:00
scrape_runs.py fix(tradein): persist scrape_runs.total_seen/new_count from sweep counters (#1926) (#1928) 2026-06-26 18:09:17 +00:00
scraper_adapters.py feat(proxy-pool): P4 — browser dynamic pool-proxy (acquire→/fetch→relaunch) + orchestration wiring, ship-dark (#2164) 2026-07-02 19:20:29 +00:00
scraper_settings.py chore(tradein): деактивировать N1.ru scraper — источник переехал в Cian 2026-06-16 14:50:51 +03:00
search_query.py feat(tradein): segment=vtorichka|novostroyki|all query-param on listings (#1188) 2026-06-17 21:43:01 +03:00
yandex_address_backfill.py fix(tradein): yandex address-enrich — city-agnostic title regex (спутники ЕКБ) (#875) 2026-05-31 09:08:29 +00:00
yandex_price_history.py fix(tradein/scrapers): migrate cian_price_history/cian_session/yandex_price_history imports to scraper_kit (#2306) 2026-07-04 00:20:13 +03:00
zhkh_flats_loader.py fix(tradein): cadastre ЖКХ-fallback + COALESCE merge-key + median-floors guard 2026-06-29 01:32:48 +03:00