fix(deploy): trigger workflow on all data/sql/** changes, not only .sql #320

Merged

lekss361 merged 1 commit from fix/deploy-paths-data-sql-py into main

2026-05-17 18:51:29 +00:00

lekss361 commented

2026-05-17 18:48:40 +00:00

Owner

Bug

PR #318 (fix(objective parser): wrap bytes-iter in file-like reader for ijson) merged как bb9a003 в 17:52:55Z — но deploy.yml workflow НЕ запустился.

Подтверждение: Forgejo Actions list — последний run #109 на 3978aa5 (TI-2 PDF), нет run для bb9a003. Worker контейнер всё ещё на коде 389e141 (PR #315 streaming с broken ijson.items(stream, prefix) без _IterBytesReader).

Glitchtip BACKEND-1K (release=3978aa5, count=4, lastSeen 17:54:44Z) — ValueError: too many values to unpack (expected 3, got 65536) повторяется на КАЖДОМ live trigger.

Root cause

.forgejo/workflows/deploy.yml:16 top-level path filter:

on:
  push:
    branches: [main]
    paths:
      - "backend/**"
      - "frontend/**"
      - "docker-compose.prod.yml"
      - "Caddyfile"
      - ".forgejo/workflows/deploy.yml"
      - "data/sql/*.sql"     # ← ловит только .sql

PR #318 изменил только data/sql/70_parse_objective_raw.py (Python parser, исторически лежит в data/sql/). Top-level фильтр data/sql/*.sql не сматчился → workflow skip → deploy не выполнился.

Mismatch: внутренний dorny/paths-filter на строке 47 уже ловит все changes в data/sql/:

filters: |
  backend:
    - 'backend/**'
    - 'data/sql/**'    # ← internal filter правильный

Но trigger filter ограничен .sql.

Fix

Одна строка:

-      - "data/sql/*.sql"
+      - "data/sql/**"

data/sql/** glob ловит все файлы в этой папке (SQL + Python parsers + etc).

Cascade после merge

Этот PR смержится → path filter совпадёт на .forgejo/workflows/deploy.yml → workflow стартанёт
Deploy.yml делает git reset --hard origin/main → подтянет включая bb9a003 (ijson fix)
Worker контейнер recreated --force-recreate → новый код с _IterBytesReader
Следующий manual trigger /admin/scrape/objective/sync-our → lots > 0 (ijson корректно стримит)

Acceptance

Forgejo Actions: deploy.yml run для этого PR's merge commit появляется
Через ~5-8 мин после deploy: trigger sync → objective_lots.raw_id IS NOT NULL count растёт
Glitchtip BACKEND-1K не повторяется на release > merge SHA

Risk

Минимальный — однострочное расширение path filter pattern. Сам этот PR изменяет .forgejo/workflows/deploy.yml, что и так в trigger filter → workflow ЗАПУСТИТСЯ корректно на merge.

## Bug PR #318 (`fix(objective parser): wrap bytes-iter in file-like reader for ijson`) merged как `bb9a003` в 17:52:55Z — **но deploy.yml workflow НЕ запустился**. Подтверждение: Forgejo Actions list — последний run #109 на `3978aa5` (TI-2 PDF), нет run для `bb9a003`. Worker контейнер всё ещё на коде `389e141` (PR #315 streaming с broken `ijson.items(stream, prefix)` без `_IterBytesReader`). Glitchtip BACKEND-1K (release=`3978aa5`, count=4, lastSeen 17:54:44Z) — `ValueError: too many values to unpack (expected 3, got 65536)` повторяется на КАЖДОМ live trigger. ## Root cause `.forgejo/workflows/deploy.yml:16` top-level path filter: ```yaml on: push: branches: [main] paths: - "backend/**" - "frontend/**" - "docker-compose.prod.yml" - "Caddyfile" - ".forgejo/workflows/deploy.yml" - "data/sql/*.sql" # ← ловит только .sql ``` PR #318 изменил **только** `data/sql/70_parse_objective_raw.py` (Python parser, исторически лежит в data/sql/). Top-level фильтр `data/sql/*.sql` не сматчился → workflow skip → deploy не выполнился. Mismatch: внутренний `dorny/paths-filter` на строке 47 уже ловит все changes в data/sql/: ```yaml filters: | backend: - 'backend/**' - 'data/sql/**' # ← internal filter правильный ``` Но trigger filter ограничен `.sql`. ## Fix Одна строка: ```diff - - "data/sql/*.sql" + - "data/sql/**" ``` `data/sql/**` glob ловит все файлы в этой папке (SQL + Python parsers + etc). ## Cascade после merge 1. Этот PR смержится → path filter совпадёт на `.forgejo/workflows/deploy.yml` → workflow стартанёт 2. Deploy.yml делает `git reset --hard origin/main` → подтянет включая `bb9a003` (ijson fix) 3. Worker контейнер recreated `--force-recreate` → новый код с `_IterBytesReader` 4. Следующий manual trigger `/admin/scrape/objective/sync-our` → `lots > 0` (ijson корректно стримит) ## Acceptance - [ ] Forgejo Actions: deploy.yml run для этого PR's merge commit появляется - [ ] Через ~5-8 мин после deploy: trigger sync → `objective_lots.raw_id IS NOT NULL` count растёт - [ ] Glitchtip BACKEND-1K не повторяется на release > merge SHA ## Risk Минимальный — однострочное расширение path filter pattern. Сам этот PR изменяет `.forgejo/workflows/deploy.yml`, что и так в trigger filter → workflow ЗАПУСТИТСЯ корректно на merge.

lekss361 added 1 commit 2026-05-17 18:48:40 +00:00

fix(deploy): trigger workflow on all data/sql/** changes, not only .sql 1d4ef98707

PR #318 (ijson bytes-iter adapter fix) merged как bb9a003, но deploy
не запустился — top-level path filter ловил только data/sql/*.sql,
а изменения были в data/sql/70_parse_objective_raw.py (Python parser).

Worker контейнер до сих пор крутит код 389e141 → ValueError
'too many values to unpack' повторяется (Glitchtip BACKEND-1K count=4,
release=3978aa5).

Меняем pattern на data/sql/** — same behavior что у internal dorny/
paths-filter (строка 47), который ловит все changes в data/sql/.

lekss361 commented

2026-05-17 18:51:23 +00:00

Author

Owner

Deep Code Review — verdict APPROVE

Summary

Status: APPROVE — fast-track merge (prod-critical, Glitchtip BACKEND-1K still firing on live triggers)
Files reviewed: 1 (P2: .forgejo/workflows/deploy.yml)
Lines: +1 / -1
Scope: surgically minimal

Diff verification

-      - "data/sql/*.sql"
+      - "data/sql/**"

Diff confirmed via get_pr_diff — exactly one line in .forgejo/workflows/deploy.yml:16, nothing else snuck in (additions:1, deletions:1, changed_files:1).

Checks

Glob syntax — ** in paths: IS supported by Forgejo Actions (Gitea-compatible, mirrors GitHub Actions semantics). data/sql/** matches any file at any depth under data/sql/. Valid.
Internal dorny/paths-filter consistency — .forgejo/workflows/deploy.yml:43 already uses 'data/sql/**' in the backend: filter. Top-level filter now matches internal — no more mismatch where workflow triggers but internal filter says "no backend changes" or vice versa.
Cascade reasoning verified — this PR modifies .forgejo/workflows/deploy.yml which is itself in the trigger paths filter (line 15) → its own merge fires the workflow → git fetch origin main && git reset --hard origin/main (line 178-179) pulls in bb9a003 (ijson _IterBytesReader fix from PR #318) → docker compose ... up -d --force-recreate --no-deps backend worker beat (line 249-250) recreates worker with new code. Self-healing cascade works.
No unintended side effects — listed data/sql/ contents: only .sql migration files + .py parsers/loaders (11_load_domrf.py, 70_parse_objective_raw.py, etc.) + 02_load_all_quarters.sh + decode_to_file.py / upload_server.py utilities + __pycache__. All of these legitimately affect backend behavior — should trigger deploy. No README/docs/test-fixtures that would cause noise. __pycache__ is gitignored anyway. Clean.
SQL apply loop (line 223: for sql_file in $(ls -1 data/sql/*.sql ...)) — unchanged, still correctly filters to .sql only on the deploy side. So .py files in data/sql/ trigger the build/deploy but are NOT mistakenly fed to psql. Boundary preserved.

Cross-file impact

.forgejo/workflows/deploy.yml:43 internal filter already aligned — no other files need to change.
No GHA .github/workflows/deploy.yml divergence concern (Forgejo migration 2026-05-16 made Forgejo the source of truth).

Risk

Risk: Low — pure path-filter widening, can only ADD workflow runs (never block existing triggers).
Reversibility: Trivial (revert single line).
Merge window: Now (prod stuck on broken 389e141 worker code, Glitchtip errors live).

Merging.

## Deep Code Review — verdict APPROVE ### Summary - **Status**: APPROVE — fast-track merge (prod-critical, Glitchtip BACKEND-1K still firing on live triggers) - **Files reviewed**: 1 (P2: `.forgejo/workflows/deploy.yml`) - **Lines**: +1 / -1 - **Scope**: surgically minimal ### Diff verification ```diff - - "data/sql/*.sql" + - "data/sql/**" ``` Diff confirmed via `get_pr_diff` — exactly one line in `.forgejo/workflows/deploy.yml:16`, nothing else snuck in (`additions:1, deletions:1, changed_files:1`). ### Checks 1. **Glob syntax** — `**` in `paths:` IS supported by Forgejo Actions (Gitea-compatible, mirrors GitHub Actions semantics). `data/sql/**` matches any file at any depth under `data/sql/`. Valid. 2. **Internal `dorny/paths-filter` consistency** — `.forgejo/workflows/deploy.yml:43` already uses `'data/sql/**'` in the `backend:` filter. Top-level filter now matches internal — no more mismatch where workflow triggers but internal filter says "no backend changes" or vice versa. 3. **Cascade reasoning verified** — this PR modifies `.forgejo/workflows/deploy.yml` which is itself in the trigger paths filter (line 15) → its own merge fires the workflow → `git fetch origin main && git reset --hard origin/main` (line 178-179) pulls in `bb9a003` (ijson `_IterBytesReader` fix from PR #318) → `docker compose ... up -d --force-recreate --no-deps backend worker beat` (line 249-250) recreates worker with new code. Self-healing cascade works. 4. **No unintended side effects** — listed `data/sql/` contents: only `.sql` migration files + `.py` parsers/loaders (`11_load_domrf.py`, `70_parse_objective_raw.py`, etc.) + `02_load_all_quarters.sh` + `decode_to_file.py` / `upload_server.py` utilities + `__pycache__`. All of these legitimately affect backend behavior — should trigger deploy. No README/docs/test-fixtures that would cause noise. `__pycache__` is gitignored anyway. Clean. 5. **SQL apply loop** (line 223: `for sql_file in $(ls -1 data/sql/*.sql ...)`) — unchanged, still correctly filters to `.sql` only on the deploy side. So `.py` files in `data/sql/` trigger the build/deploy but are NOT mistakenly fed to `psql`. Boundary preserved. ### Cross-file impact - `.forgejo/workflows/deploy.yml:43` internal filter already aligned — no other files need to change. - No GHA `.github/workflows/deploy.yml` divergence concern (Forgejo migration 2026-05-16 made Forgejo the source of truth). ### Risk - **Risk**: Low — pure path-filter widening, can only ADD workflow runs (never block existing triggers). - **Reversibility**: Trivial (revert single line). - **Merge window**: Now (prod stuck on broken `389e141` worker code, Glitchtip errors live). Merging.