fix(msk-collector): узкий ретрай page.content() на гонке перенавигации
All checks were successful
CI / changes (pull_request) Successful in 9s
CI Trade-In / browser-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 8s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
All checks were successful
CI / changes (pull_request) Successful in 9s
CI Trade-In / browser-tests (pull_request) Has been skipped
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI Trade-In / changes (pull_request) Successful in 8s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Авито дорисовывает выдачу после domcontentloaded, и content() иногда попадает ровно в смену документа — Playwright бросает "page is navigating and changing the content". Это не отказ площадки, но прогон падал целиком: на 1110-й карточке потеряно 100 несброшенных строк. Ретрай узкий — только на этот текст ошибки и только 4 попытки; любая другая ошибка поднимается как есть, гварды блокировки не ослаблены. runs/ (планы и CSV прогонов) в .gitignore.
This commit is contained in:
parent
e2045582ab
commit
3daf09c7e4
2 changed files with 24 additions and 1 deletions
1
tradein-mvp/scripts/local-avito-msk/.gitignore
vendored
Normal file
1
tradein-mvp/scripts/local-avito-msk/.gitignore
vendored
Normal file
|
|
@ -0,0 +1 @@
|
|||
runs/
|
||||
|
|
@ -270,10 +270,32 @@ class Loader:
|
|||
await self._page.wait_for_selector('[data-marker="item"]', timeout=7_000)
|
||||
except Exception: # noqa: BLE001 — пустая/блочная страница разбирается ниже
|
||||
pass
|
||||
html = await self._page.content()
|
||||
html = await self._read_content()
|
||||
_guard(html, status)
|
||||
return html, status
|
||||
|
||||
async def _read_content(self, attempts: int = 4) -> str:
|
||||
"""page.content() с узким ретраем на гонку клиентской перенавигации.
|
||||
|
||||
Авито дорисовывает выдачу после domcontentloaded, и content() иногда
|
||||
попадает ровно в момент смены документа: "Unable to retrieve content
|
||||
because the page is navigating and changing the content". Это НЕ отказ
|
||||
площадки — гвардов не касается, поэтому ретраим только эту ошибку и
|
||||
только её, а любую другую поднимаем как есть.
|
||||
"""
|
||||
last: Exception | None = None
|
||||
for i in range(attempts):
|
||||
try:
|
||||
return await self._page.content()
|
||||
except Exception as exc: # noqa: BLE001 — сузили проверкой текста ниже
|
||||
if "page is navigating" not in str(exc):
|
||||
raise
|
||||
last = exc
|
||||
print(f" content() поймал перенавигацию, попытка {i + 2}/{attempts}",
|
||||
flush=True)
|
||||
await asyncio.sleep(1.5)
|
||||
raise RuntimeError(f"page.content() не отдал документ за {attempts} попыток") from last
|
||||
|
||||
|
||||
# --- заливка в msk_raw -----------------------------------------------------
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue