Commit graph

14 commits

Author SHA1 Message Date
272abac4d2 fix(tradein/matching): город развёртки доезжает до стража Tier-2a — межгородские склейки домов (#2777) (#2808)
All checks were successful
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 3m6s
Deploy Trade-In / build-backend (push) Successful in 1m37s
Deploy Trade-In / deploy (push) Successful in 2m3s
2026-08-10 08:51:14 +00:00
bot-backend
1a577fe748 fix(tradein/matching): снять слияние по ГАР-GUID, починить приёмник кадастра и keeper (#2674)
All checks were successful
CI Trade-In / changes (pull_request) Successful in 8s
CI / changes (pull_request) Successful in 7s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / backend-tests (pull_request) Successful in 2m54s
Ревью PR #2688 нашло, что расширение ключа дедупа было неверным. Снимаю его
полностью и добавляю три правки, которых не хватало.

СНЯТО: слияние 781 дома по COALESCE(house_fias_id, gar_house_guid).
Аргумент «общий UUID здания есть независимая идентичность» оказался круговым.
gar_flats_loader проставляет gar_house_guid предикатом
  WHERE tradein_canon_addr(COALESCE(h.short_address, h.full_address, h.address)) = gp.canon
— левая часть побайтово равна ключу канон-прохода, то есть guid является
детерминированной функцией канон-адреса, а не вторым наблюдением. Проход шёл
с выключенным гео-стражем, значит #2187 обходился боковой дверью: канон-проход
отказывается слить два дома в 6 км, а этот сливал их же за «общий UUID»,
выданный за тот же адрес. Плюс gar_pick берёт DISTINCT ON (canon) — одна
ГАР-строка на канон, а 20.3% канонов накрывают несколько зданий, и ЕКБ-фильтр
стоит только на стороне ГАР. Кедровка/Советская 17 уехала бы в ЕКБ. Нужен
ключ, независимый от канона, либо включённый гео-страж — это другая задача.

Приёмник кадастра сужен до кадастра ЗДАНИЯ. Параметр cadastral_number (кадастр
КВАРТИРЫ) убран из match_or_create_house, Protocol HouseMatcher,
RealMatcherAdapter и обоих вызывающих; `cad` больше не падает на него фолбэком.
Мина была отложенной: начни Циан отдавать offer["cadastralNumber"], который
парсер уже читает, — у каждой квартиры свой номер, Tier 0 не сматчил бы
никогда, New-house INSERT записал бы номер квартиры в houses.cadastral_number
и попутно снял P1-страж «безномерный адрес без кадастра не создаём». Две
квартиры одного дома дали бы два дома — то самое дробление. В listings оба
поля пишутся как раньше.

Keeper: listing_cnt DESC NULLS LAST. Счётчик приходит из LEFT JOIN, у дома без
объявлений он NULL, а DESC в Postgres — NULLS FIRST, поэтому пустая запись
обгоняла запись со 192 объявлениями вопреки задокументированному правилу.
Дефект предсуществующий и живой для канон-прохода.

Сторож границы вызова для живого ФИАС-тира. Прежние проверки были
структурными — видели имя параметра в сигнатуре. Уберут аргумент на настоящей
границе (estimator.estimate_quality -> match_house_readonly) — сигнатура цела,
тесты зелёные, тир снова мёртв. Новый тест смотрит на сам вызов. Заявление
«тест ловит неуловимый класс» из прошлого описания снято как преувеличение:
структурная проверка ловит подслучай, и building_cadastral_number её проходит
при нуле срабатываний из 49 502.

Остаётся из первого захода: снятый фильтр поиска has_kadastr, разделение
ФИАС-тира (удалён в пути создания, оставлен в read-only), поправка ложного
утверждения в шапке cadastral_geo_match.py.

Refs #2674
2026-08-06 05:11:47 +05:00
d362b16d7c fix(tradein/proxy): доводить сигнал бана площадки до пула (#2600 п.1) (#2653)
All checks were successful
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m39s
Deploy Trade-In / build-backend (push) Successful in 1m35s
Deploy Trade-In / deploy (push) Successful in 1m40s
2026-08-05 11:37:35 +00:00
59c072fc4c chore(tradein): удалить мёртвые mobileproxy env-переменные и rotate-ip (#2616 шаги 2-3) (#2650)
All checks were successful
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Successful in 2m13s
Deploy Trade-In / test (push) Successful in 2m45s
Deploy Trade-In / build-browser (push) Successful in 3m2s
Deploy Trade-In / build-backend (push) Successful in 1m38s
Deploy Trade-In / deploy (push) Successful in 2m29s
2026-08-05 09:35:54 +00:00
ef2b07ee9c fix(tradein/proxy): один прокси на сессию браузера вместо смены на каждом запросе (#2640)
All checks were successful
Deploy Trade-In / changes (push) Successful in 9s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 2m36s
Deploy Trade-In / build-backend (push) Successful in 1m33s
Deploy Trade-In / deploy (push) Successful in 1m26s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-08-04 18:45:37 +00:00
7d13e93792 fix(tradein/scraper): не ходить через мёртвый запасной прокси (#2634)
All checks were successful
Deploy Trade-In / changes (push) Successful in 10s
Deploy Trade-In / build-frontend (push) Successful in 38s
Deploy Trade-In / build-browser (push) Successful in 2m18s
Deploy Trade-In / test (push) Successful in 2m44s
Deploy Trade-In / build-backend (push) Successful in 1m37s
Deploy Trade-In / deploy (push) Successful in 2m27s
Co-authored-by: lekss361 <lekss361@gendsgn.local>
Co-committed-by: lekss361 <lekss361@gendsgn.local>
2026-08-04 18:27:49 +00:00
bot-backend
9b4cdf8ec6 feat(tradein/scraper-kit): shared base/utils для providers/* (#2358, Foundation)
Аддитивный Foundation-модуль (эпик #2277 Group F4): scraper_kit/providers/_base.py
извлекает 3 реально повторяющихся паттерна, найденных чтением avito/cian/yandex/
domclick providers (~13k строк):

- build_browser_fetcher(config: ScraperConfig, source, ...) — config MANDATORY,
  устраняет footgun-класс #2322/#2330 (BrowserFetcher конструировался вручную в
  6+ местах, endpoint/proxy-pool wiring легко забыть/потерять).
- build_curl_cffi_session / build_document_session + http_proxies() — общий
  curl_cffi AsyncSession(impersonate=chrome120, proxies=..., headers=...)
  boilerplate, дублированный в avito/serp.py, avito/detail.py, avito/imv.py,
  cian/detail.py, cian/newbuilding.py (x2), cian/session.py.
- DOCUMENT_HEADERS — идентичный "document" headers dict, дублированный трижды.

Retry/backoff-циклы (yandex tarpit-retry, avito page-retry, tenacity в
BaseScraper) сознательно НЕ обобщены — разная семантика ошибок/действий между
попытками, обобщение дало бы leaky abstraction (см. docstring модуля).

Existing providers/{avito,cian,yandex,domclick}/*.py НЕ тронуты — миграция
call-site'ов на новый модуль это отдельные future issues (F4a-d).

Заодно: актуализирован устаревший докстринг app/services/scraper_adapters.py
("Пока НИКТО их не использует") — адаптеры wired в прод с #2192 через
app.scheduler_main._run_kit_scheduler.
2026-07-04 03:50:55 +03:00
ca473a0012 feat(proxy-pool): P4 — browser dynamic pool-proxy (acquire→/fetch→relaunch) + orchestration wiring, ship-dark (#2164)
All checks were successful
Deploy Trade-In / changes (push) Successful in 13s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m32s
Deploy Trade-In / build-browser (push) Successful in 2m13s
Deploy Trade-In / build-backend (push) Successful in 1m18s
Deploy Trade-In / deploy (push) Successful in 2m29s
2026-07-02 19:20:29 +00:00
0937da4260 feat(proxy-pool): P3 — kit curl-paths acquire/release from pool behind USE_PROXY_POOL_CURL, ship-dark (#2163)
All checks were successful
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m26s
Deploy Trade-In / build-backend (push) Successful in 1m18s
Deploy Trade-In / deploy (push) Successful in 1m22s
2026-07-02 18:56:36 +00:00
bot-backend
0eb033d77d fix(tradein/matching): P1 extended — never create a house from a numberless address (no cadastral)
All checks were successful
CI / changes (pull_request) Successful in 14s
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
2026-07-02 21:33:52 +03:00
abdb413a99 feat(scraper-kit): copy remaining sweep orchestrators (yandex/cian/domclick/full-load), strangler (#2135 F2)
All checks were successful
Deploy Trade-In / changes (push) Successful in 7s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m27s
Deploy Trade-In / build-backend (push) Successful in 1m32s
Deploy Trade-In / deploy (push) Successful in 59s
2026-07-02 17:29:05 +00:00
8a177df0e2 feat(scraper-kit): copy sweep pipeline orchestration (avito) with protocol injection, strangler (#2135)
All checks were successful
Deploy Trade-In / changes (push) Successful in 8s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m35s
Deploy Trade-In / build-backend (push) Successful in 1m24s
Deploy Trade-In / deploy (push) Successful in 1m0s
2026-07-02 17:08:40 +00:00
f7063d49b5 fix(scraper-kit): sync RealScraperConfig adapter with ScraperConfig protocol (cookie_encryption_key)
All checks were successful
Deploy Trade-In / changes (push) Successful in 26s
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / build-frontend (push) Successful in 50s
Deploy Trade-In / test (push) Successful in 1m56s
Deploy Trade-In / build-backend (push) Successful in 1m29s
Deploy Trade-In / deploy (push) Successful in 2m45s
2026-07-02 16:46:05 +00:00
6391587572 feat(scraper-kit): protocol boundaries (matcher/config/session/jobs) + backend adapters (#2131)
All checks were successful
Deploy Trade-In / changes (push) Successful in 11s
Deploy Trade-In / build-frontend (push) Has been skipped
Deploy Trade-In / build-browser (push) Has been skipped
Deploy Trade-In / test (push) Successful in 1m27s
Deploy Trade-In / build-backend (push) Successful in 1m26s
Deploy Trade-In / deploy (push) Successful in 1m0s
2026-07-02 15:25:38 +00:00