refactor(tradein/avito): avito_detail_backfill off scrape_pipeline helpers (#2397 slice B) #2399

Merged

lekss361 merged 1 commit from refactor/tradein-extract-avito-scrape-helpers into main

2026-07-04 09:01:41 +00:00

lekss361 commented

2026-07-04 08:59:21 +00:00

Owner

Summary

Umbrella #2397 slice B (эпик #2277 decommission legacy orchestration). Отвязывает app/tasks/avito_detail_backfill.py от legacy scrape_pipeline.py, чтобы последний можно было удалить в Part E без поломки этого таска.

Было: from app.services.scrape_pipeline import _CHROME_HEADERS, _avito_proxies
Стало: from scraper_kit.providers._base import DOCUMENT_HEADERS, http_proxies (kit Foundation #2358 — byte-identical эквиваленты)
Call-site: headers=DOCUMENT_HEADERS, proxies=http_proxies(settings.scraper_proxy_url)
Инлайн AsyncSession(...) СОХРАНЁН (не переключён на kit build_document_session()) — намеренно, чтобы не сломать mock-seam app.tasks.avito_detail_backfill.AsyncSession в тестах. Поведение идентично.
scrape_pipeline.py НЕ тронут.

Review

code-reviewer — ✅ APPROVE. Byte-equivalence подтверждена key-for-key (_CHROME_HEADERS≡DOCUMENT_HEADERS, _avito_proxies≡http_proxies, тот же settings-singleton, та же proxy-резолюция). Mock-seam-обоснование признано верным.

Test plan

tests/tasks/test_avito_detail_backfill.py: 14 passed
Full suite: 3276 passed, 6 skipped, 1 pre-existing unrelated fail (#2208)
ruff check/format clean; grep — 0 импортов scrape_pipeline в файле

Refs #2397, #2277

## Summary Umbrella #2397 slice B (эпик #2277 decommission legacy orchestration). Отвязывает `app/tasks/avito_detail_backfill.py` от legacy `scrape_pipeline.py`, чтобы последний можно было удалить в Part E без поломки этого таска. - Было: `from app.services.scrape_pipeline import _CHROME_HEADERS, _avito_proxies` - Стало: `from scraper_kit.providers._base import DOCUMENT_HEADERS, http_proxies` (kit Foundation #2358 — byte-identical эквиваленты) - Call-site: `headers=DOCUMENT_HEADERS`, `proxies=http_proxies(settings.scraper_proxy_url)` - Инлайн `AsyncSession(...)` СОХРАНЁН (не переключён на kit `build_document_session()`) — намеренно, чтобы не сломать mock-seam `app.tasks.avito_detail_backfill.AsyncSession` в тестах. Поведение идентично. - `scrape_pipeline.py` НЕ тронут. ## Review `code-reviewer` — ✅ APPROVE. Byte-equivalence подтверждена key-for-key (`_CHROME_HEADERS`≡`DOCUMENT_HEADERS`, `_avito_proxies`≡`http_proxies`, тот же settings-singleton, та же proxy-резолюция). Mock-seam-обоснование признано верным. ## Test plan - [x] `tests/tasks/test_avito_detail_backfill.py`: 14 passed - [x] Full suite: 3276 passed, 6 skipped, 1 pre-existing unrelated fail (#2208) - [x] `ruff check`/`format` clean; grep — 0 импортов scrape_pipeline в файле Refs #2397, #2277

lekss361 added 1 commit 2026-07-04 08:59:21 +00:00

refactor(tradein/avito): avito_detail_backfill off scrape_pipeline helpers (#2397 slice B)

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI Trade-In / changes (pull_request) Successful in 7s

Details

CI / changes (pull_request) Successful in 7s

Details

CI Trade-In / backend-tests (pull_request) Successful in 1m38s

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

8ca529ece0

app/tasks/avito_detail_backfill.py imported _CHROME_HEADERS/_avito_proxies from
the legacy app/services/scrape_pipeline.py (the only two symbols it needed from
that module). scrape_pipeline.py is slated for wholesale deletion in Part E of
epic #2277 -- this severs the last dependency so that deletion won't break the
backfill task.

Source chosen: kit-reuse, not relocate. scraper_kit/providers/_base.py (#2358
Foundation) already extracted the identical building blocks while deduplicating
providers/avito/{serp,detail,imv}.py:
- DOCUMENT_HEADERS -- byte-identical dict to _CHROME_HEADERS (same 8 keys/values,
  Accept/Accept-Language/Cache-Control/Sec-Fetch-*/Upgrade-Insecure-Requests).
- http_proxies(proxy_url) -- same formula as _avito_proxies():
  {"http": url, "https": url} if url else None, just parameterized instead of
  reading settings.scraper_proxy_url internally (both resolve the identical
  settings singleton in production, so behavior is unchanged).

Kept the AsyncSession(...) construction inline (didn't switch to kit's
build_document_session helper) to preserve the existing
`app.tasks.avito_detail_backfill.AsyncSession` mock seam used by
tests/tasks/test_avito_detail_backfill.py -- swapping to the helper would call
curl_cffi's AsyncSession via scraper_kit.providers._base instead, silently
bypassing that patch target.

scrape_pipeline.py itself is untouched (kept its own copies, per Part E plan).

Full backend suite: 3276 passed, 1 known-unrelated fail (test_search_cache_hit,
#2208), 6 skipped.