fix(msk-collector): ретрай заливки батча на обрыве ssh
All checks were successful
CI Trade-In / changes (pull_request) Successful in 7s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / browser-tests (pull_request) Has been skipped
CI / changes (pull_request) Successful in 10s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
All checks were successful
CI Trade-In / changes (pull_request) Successful in 7s
CI Trade-In / backend-tests (pull_request) Has been skipped
CI Trade-In / browser-tests (pull_request) Has been skipped
CI / changes (pull_request) Successful in 10s
CI Trade-In / frontend-checks (pull_request) Has been skipped
CI / backend-tests (pull_request) Has been skipped
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Has been skipped
Прогон длится часами, и ssh рвётся: живьём поймано Connection reset by peer (ssh 255) прямо посреди заливки — умирал весь прогон, несброшенный батч (до 1000 карточек) терялся, resume перечитывал страницы заново. Ретрай безопасен: SQL идемпотентен целиком — batch через ON CONFLICT DO NOTHING, карточки через ON CONFLICT (source_id,batch_id,kind) DO NOTHING. Ретраится ТОЛЬКО транспорт (код 255); ошибка самого psql под ON_ERROR_STOP поднимается сразу — это дефект данных или SQL, повтор его не лечит.
This commit is contained in:
parent
62b9d1d4a7
commit
ae70e7a39b
1 changed files with 24 additions and 7 deletions
|
|
@ -434,18 +434,35 @@ def build_finalize_sql(batch_id: str, query: str, notes: str) -> str:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def run_psql(sql: str, ssh_host: str, container: str, db_user: str, db_name: str) -> None:
|
def run_psql(sql: str, ssh_host: str, container: str, db_user: str, db_name: str,
|
||||||
|
attempts: int = 4) -> None:
|
||||||
|
"""Заливка батча через ssh с ретраем на обрыв транспорта.
|
||||||
|
|
||||||
|
Прогон длится часами, и ssh рвётся: живьём поймано «Connection reset by peer»
|
||||||
|
(ssh возвращает 255) прямо посреди заливки — весь прогон умирал, а несброшенный
|
||||||
|
батч терялся. Ретраить безопасно: SQL идемпотентен (batch через ON CONFLICT DO
|
||||||
|
NOTHING, карточки через ON CONFLICT (source_id,batch_id,kind) DO NOTHING).
|
||||||
|
|
||||||
|
Ретраится ТОЛЬКО транспорт (ssh 255). Ошибка самого psql (ON_ERROR_STOP, любой
|
||||||
|
другой код) — это дефект данных или SQL, её повтор не лечит: поднимаем сразу.
|
||||||
|
"""
|
||||||
cmd = [
|
cmd = [
|
||||||
"ssh", ssh_host,
|
"ssh", ssh_host,
|
||||||
f"docker exec -i {container} psql -U {db_user} -d {db_name} "
|
f"docker exec -i {container} psql -U {db_user} -d {db_name} "
|
||||||
"-v ON_ERROR_STOP=1 -f -",
|
"-v ON_ERROR_STOP=1 -f -",
|
||||||
]
|
]
|
||||||
proc = subprocess.run(cmd, input=sql.encode("utf-8"), capture_output=True)
|
for i in range(attempts):
|
||||||
out = (proc.stdout + proc.stderr).decode("utf-8", "replace").strip()
|
proc = subprocess.run(cmd, input=sql.encode("utf-8"), capture_output=True)
|
||||||
if proc.returncode != 0:
|
out = (proc.stdout + proc.stderr).decode("utf-8", "replace").strip()
|
||||||
raise RuntimeError(f"psql через ssh вернул {proc.returncode}:\n{out}")
|
if proc.returncode == 0:
|
||||||
if out:
|
if out:
|
||||||
print(f" psql: {out}", flush=True)
|
print(f" psql: {out}", flush=True)
|
||||||
|
return
|
||||||
|
if proc.returncode != 255 or i == attempts - 1:
|
||||||
|
raise RuntimeError(f"psql через ssh вернул {proc.returncode}:\n{out}")
|
||||||
|
tail = out.splitlines()[-1] if out else "без вывода"
|
||||||
|
print(f" ssh оборвался ({tail}), повтор заливки {i + 2}/{attempts}", flush=True)
|
||||||
|
time.sleep(15.0 * (i + 1))
|
||||||
|
|
||||||
|
|
||||||
# --- накопитель карточек ---------------------------------------------------
|
# --- накопитель карточек ---------------------------------------------------
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue