refactor(tradein/scraper-kit): yandex providers on shared _base.py (#2363) #2373

Closed

lekss361 wants to merge 1 commit from feat/tradein-scraper-kit-f4c-yandex into main

lekss361 commented

2026-07-04 06:30:12 +00:00

Owner

F4c-yandex (epic #2277 Group F, parent #2352): migrate the SERP scraper's BrowserFetcher construction onto build_browser_fetcher() from the Foundation module (#2358, providers/_base.py) instead of local ad-hoc construction.

What changed

yandex/serp.py::__aenter__ now calls build_browser_fetcher(config, source=, proxy_provider=, fetch_timeout_s=) instead of constructing BrowserFetcher(...) directly. Preserves the existing fetch_timeout_s=30, endpoint/proxy_provider/use_pool resolution byte-for-byte. The retry/tarpit logic in fetch_around() (rotate_ip + sleep on status==0/JSON-error) is untouched, per the Foundation module's documented exclusion.
test_kit_serp_proxy_pool.py's 3 yandex-specific tests: patch target moved from module-local yandex_serp.BrowserFetcher to scraper_kit.providers._base.BrowserFetcher (the migrated call path no longer references the name directly). Every assertion is unchanged.

Deliberately left unmigrated (not oversights)

yandex/serp.py::_rotate_ip's curl_cffi AsyncSession(timeout=30) — hits the mobile-proxy provider's own changeip API, not Yandex; intentionally carries no impersonate/headers/proxy (mirrors avito/serp.py). Not one of _base.py's documented extraction sites.
yandex/detail.py — no BrowserFetcher/curl_cffi construction exists (uses BaseScraper's httpx-based _http_get). Nothing to migrate.
yandex/newbuilding.py — both BrowserFetcher(...) call sites take an optional config: ScraperConfig | None, a genuinely different contract from build_browser_fetcher's mandatory config. Migrating would require making config mandatory, breaking admin.py:1511, yandex_newbuilding_sweep.py:353, and test_scraper_kit_newbuilding_endpoint.py's "without config, endpoint=None" regression case. Out of scope for a call-site swap; left untouched.

Review

code-reviewer verdict: approve. Confirmed diff touches exactly the 2 files claimed; both scope-narrowing findings above independently verified against callers/tests. Reran targeted parity tests (43 passed) and the full backend suite (1 failed / 3278 passed / 6 skipped — the single failure, test_search_cache_hit, reproduces identically on main HEAD in a disposable worktree, pre-existing and unrelated). ruff check + ruff format --check clean on both changed files.

Re-verified after rebase onto latest main (post #2372 domclick migration, no file overlap): targeted tests green, ruff clean.

Closes #2363.

F4c-yandex (epic #2277 Group F, parent #2352): migrate the SERP scraper's BrowserFetcher construction onto `build_browser_fetcher()` from the Foundation module (#2358, `providers/_base.py`) instead of local ad-hoc construction. ## What changed - `yandex/serp.py::__aenter__` now calls `build_browser_fetcher(config, source=, proxy_provider=, fetch_timeout_s=)` instead of constructing `BrowserFetcher(...)` directly. Preserves the existing `fetch_timeout_s=30`, endpoint/proxy_provider/use_pool resolution byte-for-byte. The retry/tarpit logic in `fetch_around()` (rotate_ip + sleep on status==0/JSON-error) is untouched, per the Foundation module's documented exclusion. - `test_kit_serp_proxy_pool.py`'s 3 yandex-specific tests: patch target moved from module-local `yandex_serp.BrowserFetcher` to `scraper_kit.providers._base.BrowserFetcher` (the migrated call path no longer references the name directly). Every assertion is unchanged. ## Deliberately left unmigrated (not oversights) - `yandex/serp.py::_rotate_ip`'s curl_cffi `AsyncSession(timeout=30)` — hits the mobile-proxy provider's own changeip API, not Yandex; intentionally carries no impersonate/headers/proxy (mirrors avito/serp.py). Not one of `_base.py`'s documented extraction sites. - `yandex/detail.py` — no BrowserFetcher/curl_cffi construction exists (uses BaseScraper's httpx-based `_http_get`). Nothing to migrate. - `yandex/newbuilding.py` — both `BrowserFetcher(...)` call sites take an **optional** `config: ScraperConfig | None`, a genuinely different contract from `build_browser_fetcher`'s mandatory `config`. Migrating would require making config mandatory, breaking `admin.py:1511`, `yandex_newbuilding_sweep.py:353`, and `test_scraper_kit_newbuilding_endpoint.py`'s "without config, endpoint=None" regression case. Out of scope for a call-site swap; left untouched. ## Review `code-reviewer` verdict: **approve**. Confirmed diff touches exactly the 2 files claimed; both scope-narrowing findings above independently verified against callers/tests. Reran targeted parity tests (43 passed) and the full backend suite (1 failed / 3278 passed / 6 skipped — the single failure, `test_search_cache_hit`, reproduces identically on `main` HEAD in a disposable worktree, pre-existing and unrelated). `ruff check` + `ruff format --check` clean on both changed files. Re-verified after rebase onto latest `main` (post #2372 domclick migration, no file overlap): targeted tests green, ruff clean. Closes #2363.

lekss361 added 1 commit 2026-07-04 06:30:13 +00:00

refactor(tradein/scraper-kit): yandex providers on shared _base.py (#2363 )

CI Trade-In / changes (pull_request) Successful in 10s

Details

CI / changes (pull_request) Successful in 12s

Details

CI Trade-In / frontend-checks (pull_request) Has been skipped

Details

CI / backend-tests (pull_request) Has been skipped

Details

CI / frontend-tests (pull_request) Has been skipped

Details

CI / openapi-codegen-check (pull_request) Has been skipped

Details

CI Trade-In / backend-tests (pull_request) Successful in 1m45s

Details

802b114e0d

F4c-yandex (epic #2277 Group F, parent #2352): migrate the SERP scraper's
BrowserFetcher construction onto build_browser_fetcher() from the Foundation
module (#2358, providers/_base.py) instead of local ad-hoc construction.

- yandex/serp.py::__aenter__ now calls build_browser_fetcher(config, source=,
  proxy_provider=, fetch_timeout_s=) instead of constructing BrowserFetcher(...)
  directly. Preserves the existing fetch_timeout_s=30 (see caveat below),
  endpoint/proxy_provider/use_pool resolution byte-for-byte. The retry/tarpit
  logic in fetch_around() (rotate_ip + sleep on status==0/JSON-error) is
  untouched, per the Foundation module's documented exclusion.

Left unchanged (deliberate, not an oversight):
- yandex/serp.py::_rotate_ip's curl_cffi AsyncSession(timeout=30) — this hits
  the mobile-proxy provider's own changeip API, not Yandex; it intentionally
  carries no impersonate/headers/proxy (same as the mirrored avito/serp.py
  rotate-ip session). Not listed among the _base.py docstring's documented
  extraction sites -- consistent exclusion, not a new one.
- yandex/detail.py -- no BrowserFetcher/curl_cffi construction exists in this
  file (uses BaseScraper's httpx-based _http_get). Nothing to migrate.
- yandex/newbuilding.py -- both BrowserFetcher(...) call sites take an
  OPTIONAL `config: ScraperConfig | None`, a genuinely different contract from
  build_browser_fetcher's mandatory `config`. Migrating would require making
  config mandatory (breaking admin.py:1511, yandex_newbuilding_sweep.py:353,
  and several tests that call without config -- out of scope for a call-site
  swap) and breaks test_scraper_kit_newbuilding_endpoint.py's "without config,
  endpoint=None" regression case (its SimpleNamespace mock lacks
  use_proxy_pool_browser, and the mock config path can never satisfy a
  mandatory-config helper). Left untouched; the class-bug #2322/#2330 this
  file already fixed (config now threaded, just not via the shared helper)
  remains fixed.

Also updates test_kit_serp_proxy_pool.py's 3 yandex-specific tests: they
monkeypatched the module-local `yandex_serp.BrowserFetcher` name, which the
migrated call path no longer references directly (it now goes through
scraper_kit.providers._base.build_browser_fetcher -> _base's own BrowserFetcher
import). Patch target moved to scraper_kit.providers._base.BrowserFetcher;
every assertion is unchanged -- this adapts the mock to the new (intentional)
call path, it does not weaken what the test verifies. Cian's tests in the same
file are untouched (cian/serp.py is not migrated yet, separate F4b issue).

lekss361 referenced this pull request

2026-07-04 06:40:53 +00:00

refactor(tradein/scraper-kit): yandex providers on shared _base.py (#2363) #2384

lekss361 commented

2026-07-04 06:41:08 +00:00

Author

Owner

Заменён на PR #2384 (тот же коммит-контент, ребейзнут на актуальный main с разрешённым 3-way merge докстринга в test_kit_serp_proxy_pool.py). Этот PR упёрся в non-fast-forward после ребейза; вместо force-push (запрещён без явного одобрения) пересоздана ветка feat/tradein-scraper-kit-f4c-yandex-v2. Закрываю этот, мерж идёт через #2384.

Заменён на PR #2384 (тот же коммит-контент, ребейзнут на актуальный main с разрешённым 3-way merge докстринга в `test_kit_serp_proxy_pool.py`). Этот PR упёрся в non-fast-forward после ребейза; вместо force-push (запрещён без явного одобрения) пересоздана ветка `feat/tradein-scraper-kit-f4c-yandex-v2`. Закрываю этот, мерж идёт через #2384.

lekss361 closed this pull request

2026-07-04 06:41:27 +00:00