Commit graph

9 commits

Author SHA1 Message Date
0de22f4bc9 fix(tradein/scraper): диагноз бана перестаёт назначаться по умолчанию (#2764) (#2765)
All checks were successful
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m59s
Deploy Trade-In / build-backend (push) Successful in 2m1s
Deploy Trade-In / deploy (push) Successful in 2m14s
2026-08-06 23:17:59 +00:00
9e5e9fca08 fix(tradein/scraper): статус «забанен» перестаёт выдавать наш сбой за чужой (#2686) (#2711)
All checks were successful
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m58s
Deploy Trade-In / build-backend (push) Successful in 1m37s
Deploy Trade-In / deploy (push) Successful in 1m44s
2026-08-06 08:18:26 +00:00
cb64796b44 fix(tradein/scraper): дневной снимок узнаёт свой прогон (#2701) (#2707)
All checks were successful
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m55s
Deploy Trade-In / build-backend (push) Successful in 1m34s
Deploy Trade-In / deploy (push) Successful in 1m21s
2026-08-06 07:00:17 +00:00
59c072fc4c chore(tradein): удалить мёртвые mobileproxy env-переменные и rotate-ip (#2616 шаги 2-3) (#2650)
All checks were successful
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Successful in 2m13s
Deploy Trade-In / test (push) Successful in 2m45s
Deploy Trade-In / build-browser (push) Successful in 3m2s
Deploy Trade-In / build-backend (push) Successful in 1m38s
Deploy Trade-In / deploy (push) Successful in 2m29s
2026-08-05 09:35:54 +00:00
bot-backend
a95c38bdcc fix(tradein/scraper): подключить браузерный путь Авито к пулу прокси
All checks were successful
CI / changes (pull_request) Successful in 7s
CI Trade-In / changes (pull_request) Successful in 8s
CI / backend-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 2m40s
2026-08-02 17:31:00 +03:00
7c9319d3b9 fix(tradein/scraper): не помечать городом развёртки объявления соседних городов (#2626)
All checks were successful
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m36s
Deploy Trade-In / build-backend (push) Successful in 1m37s
Deploy Trade-In / deploy (push) Successful in 1m38s
2026-08-02 11:46:02 +00:00
bot-backend
0a4e126b30 fix(tradein/scraper): проставлять город объявления из контекста развёртки (#2594)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 9s
CI / changes (pull_request) Successful in 9s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 2m28s
Скрапер знает город в момент сбора (city_slug из CITY_LOCATIONS/CITY_ANCHORS,
scraper_kit.orchestration.pipeline), но раньше нигде его не записывал. Провайдеры
(avito/cian) часто отдают адрес БЕЗ города в тексте ("ул. Победы, 30" вместо
"Нижний Тагил, ул. Победы, 30" — cian даже явно вырезает location-часть перед
записью, providers/cian/serp.py _format_address skip_types={"location",...}).
Без города такой адрес при геокодинге считался "город не назван" и коллизировал
с одноимённой екатеринбургской улицей (Ленина/Победы/Тенистая — сотни совпадений
в ЕКБ-реестрах) → объявление получало координаты Екатеринбурга.

Fix: отдельная колонка listings.city (196_listings_city.sql), проставляется из
sweep-контекста через save_listings(..., city=...) — НЕ парсингом/дописыванием
в address. Раздельная колонка не портит исходный текст адреса: downstream
text-парсеры (geocoder._parse_street_house/_names_non_ekb_city, estimator
house-matching) продолжают работать на исходном сыром тексте неизменёнными —
дописывание города в address ломало бы bare-form адреса без street-маркера
("Дружинина, 33" без "ул.") в этих же парсерах.

Симметрия: EKB-варианты city-sweep функций (city_slug=None) тоже получают
city="Екатеринбург" — resolve_city_name(None) даёт тот же ЕКБ-дефолт, что и
get_city_location/get_city_anchors. Проставлено во всех продовых write-путях:
run_avito_city_sweep/run_yandex_city_sweep/run_cian_city_sweep (city_slug-aware),
run_avito_newbuilding_sweep/run_cian_full_load/run_yandex_full_load/
run_avito_full_load (подтверждённо EKB-only по докстрингам), run_domclick_city_sweep
(EKB city_id, oblast B2 ещё не wired — честный None для неизвестного city_id).

Scope: только write-path для НОВЫХ листингов. Бэкфилл накопленных строк и
консультация city в geocode_missing_listings/backfill_coords_from_geoportal
(gate там пока text-only, _names_non_ekb_city) — geocoder.py намеренно не
тронут (#2582/#2580) — отдельные follow-up задачи.
2026-07-31 22:14:47 +03:00
bot-backend
79a77200d7 refactor(tradein/scrapers): удалить legacy scrape_pipeline.py + тест-хирургия (#2397 Part E1)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 10s
CI / changes (pull_request) Successful in 9s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 1m43s
app.services.scrape_pipeline (3734 строк) имел 0 runtime-импортёров — Part C убрал
последнего вызывающего (admin.py, scheduler_main.py уже на
scraper_kit.orchestration.pipeline). Удаление:

- backend/app/services/scrape_pipeline.py — удалён целиком

Тесты (18 файлов, импортировавших scrape_pipeline):

DELETE (чистая legacy-оркестрация, kit-эквивалент покрыт в
test_scraper_kit_pipeline_parity{,2}.py):
- test_scrape_pipeline.py, test_scrape_pipeline_proxy_pool.py (proxy-pool DB-lease
  вытеснен RealProxyProvider DI-адаптером, покрытым test_scraper_adapters_contracts.py)
- test_anchor_watchdog.py, test_ban_rotation_1790.py, test_provider_rotation_1848.py
- test_scraper_resilience_1949_1950.py, test_sweep_drain.py

CONVERT (parity-сравнение убрано, kit-only regression оставлен/усилен):
- test_scraper_kit_pipeline_parity.py, test_scraper_kit_pipeline_parity2.py —
  _drive_old убран, _drive_new стал единственной веткой; ассёрты на конкретные
  значения counters сохранены (это теперь основное regression-покрытие kit
  orchestration)
- test_sweep_imv_phase.py — 6 IMV-phase тестов (enrich_imv true/false/no-touched/
  crash/failed-counter/cancel) переведены на run_avito_city_sweep (kit) с DI
  (config/matcher/enrichment); enrichment.process_houses_imv_batch — инжектируемый
  Protocol-метод вместо прямого импорта app.services.house_imv_backfill

PARTIAL-EDIT (легаси sweep-behavior вырезан, admin/kit-DI/несвязанные тесты
оставлены):
- test_city_sweep.py — только retarget 4 импортов на scraper_kit.orchestration.pipeline
- test_cian_city_sweep.py, test_yandex_city_sweep.py — sweep-behavior секции удалены,
  admin endpoint тесты (уже kit-DI) + anchor-timeout/watchdog формулы (retarget import)
  оставлены
- test_domclick_sweep.py — sweep-wiring тесты удалены, Counters + _map_item оставлены
- test_avito_newbuilding_sweep.py — sweep-wiring тест #5 удалён, retarget
  NewbuildingSweepCounters
- test_pipeline_browser_routing.py — 3 run_avito_pipeline теста удалены, browser-routing
  на уровне fetch_detail/fetch_house_catalog/AvitoScraper не тронут
- test_scraper_proxy.py, tests/scrapers/test_avito_anti_bot.py — _avito_proxies()
  переведён на kit (DI-параметр вместо settings-patch); pipeline-level тесты удалены

Stale-комментарии (scrape_pipeline.py как текущая реальность) поправлены в
admin.py, avito_detail_backfill.py, house_imv_backfill.py, yandex_price_history.py,
scrapers/avito_detail.py + kit-пакете (orchestration/__init__.py, orchestration
/pipeline.py, providers/avito/detail.py — strangler завершён, легаси удалён).

grep -rn scrape_pipeline app/ scripts/ packages/ — только past-tense упоминания
("удалён в Part E1", "перед его удалением") + один historical footnote
(contracts.py:138, provenance-заметка "собраны grep'ом по ... .py").

Backend suite: 3088 passed, 6 skipped (известный flake test_search_cache_hit #2208 —
не регрессия). ruff check чист на всех изменённых файлах.
2026-07-04 13:56:06 +03:00
8a177df0e2 feat(scraper-kit): copy sweep pipeline orchestration (avito) with protocol injection, strangler (#2135)
All checks were successful
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m35s
Deploy Trade-In / build-backend (push) Successful in 1m24s
Deploy Trade-In / deploy (push) Successful in 1m0s
2026-07-02 17:08:40 +00:00