Commit graph

5 commits

Author SHA1 Message Date
c646a71001 fix(freshness): output-floor on cycle-SUM over fresh_days window (#1945, #1947)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 1m45s
CI / backend-tests (pull_request) Successful in 11m57s
Adversarial review found the single-newest last_success_work basis fragile:
a load cycle = MULTIPLE done-runs (objective per group_name, kn per region +
resume), so the newest run alone misrepresents the cycle. Replace with
SUM(work_col) FILTER(done) over the source's own fresh_days window.

- recent_output = SUM(COALESCE(work_col,0)) FILTER done in fresh_days window
  (make_interval secs => fresh_days*86400; no :: cast; NULL-counter -> 0)
- downgrade when status==ok and recent_output < min_output_rows (strict <)
- registry floors kept: kn_flats=50000 (healthy ~376k, broken sum <=3670),
  objective=1000 (SUM 7d ~946k); comments rewritten to cycle-sum basis +
  kn_flats zombie-resume KNOWN LIMITATION
- window=fresh_days keeps the #1947 aging-FP fix (7-8d run stays in window)
- tests: low-output-cycle->failed, above-floor->ok, boundary==floor->ok,
  weekly-aging->ok, age-stale precedence, registry floors

Verified on prod: domrf_kn_flats latest snapshot 2026-06-22=9 flats vs healthy
2026-05-17=376604 (broken 5+ weeks); objective SUM(rows_lots,7d)=946264.
2026-06-27 12:36:50 +05:00
bd1adc1f59 feat(freshness): zero-output check + kn_flats source (#1945)
All checks were successful
CI / changes (pull_request) Successful in 7s
CI / frontend-tests (pull_request) Has been skipped
CI / openapi-codegen-check (pull_request) Successful in 1m53s
CI / backend-tests (pull_request) Successful in 12m5s
The data-freshness monitor classified by run RECENCY only, so the domrf_kn
FLATS loader running status=done but extracting 0 flats for ~5 weeks went
undetected — and the kn source watched objects_count (healthy ~1548), not
flats_count (the broken =0 metric).

Add an opt-in zero-output check: an otherwise-fresh run-ledger source (recent
success, would-be fresh by age) that produced 0 work-rows in the 7d window is
downgraded to status="failed" (so scrape_freshness_check alerts), with an
additive "reason". Guards: alert_on_zero_output flag, run-ledger only
(timestamp_col is None), status=="ok" (age-stale/failed already covered), and
upd_7d==0 (SUM of the source's own work_col over done-runs).

Registry: new kn_flats source (kn_scrape_runs, work_col=flats_count, critical,
flag on) — watches the column that was broken; existing kn (objects_count)
unchanged. Flag also enabled on objective (rows_lots, critical). nspd/nspd_geo/
cadastre left unflagged (legitimate-0 / data-table).

JSON additive only (new nullable "reason" key; endpoint is dict[str,Any], no
frontend consumer / no codegen needed). 4 new tests (downgrade, no-false-
positive, age-precedence, registry). code-reviewer APPROVE.

Would have caught #1945 within ~8-14d instead of 5 weeks.
2026-06-27 11:52:29 +05:00
67d41df880 fix(freshness): repoint nspd source to live nspd_quarter_dumps (data-table mode)
All checks were successful
Deploy / changes (push) Successful in 7s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 4m34s
Deploy / build-worker (push) Successful in 5m23s
Deploy / deploy (push) Successful in 1m59s
nspd freshness watched a defunct ledger (nspd_scrape_runs, WAF-banned manual April runs) → false "no successful runs". Add data-table mode (timestamp_col) + repoint nspd to live nspd_quarter_dumps.fetched_at_utc (fresh<14/stale<30). Run-ledger sources untouched. Refs #73
2026-06-26 06:51:00 +00:00
2304e09e37 fix(freshness): calibrate kn staleness thresholds to weekly cadence
All checks were successful
Deploy / changes (push) Successful in 6s
Deploy / build-frontend (push) Has been skipped
Deploy / build-backend (push) Successful in 2m41s
Deploy / build-worker (push) Successful in 3m18s
Deploy / deploy (push) Successful in 1m28s
kn (DOM.РФ ЖК) runs weekly (Mon cron) but thresholds were 2/5 → false "stale" every Wed-Sun. Recalibrate to 8/14 (weekly, mirrors objective) + regression test. Surfaced by the freshness monitor's first prod run. Refs #73
2026-06-26 06:33:29 +00:00
d38417d8af feat(admin): freshness endpoint (#73) + README Forgejo refresh (#83) + migrations doc (#72)
All checks were successful
Deploy / build-frontend (push) Has been skipped
Deploy / changes (push) Successful in 5s
Deploy / build-backend (push) Successful in 2m39s
Deploy / build-worker (push) Successful in 7m20s
Deploy / deploy (push) Successful in 2m48s
#73: GET /admin/scrape/freshness — last_success/age/freshness/status для 5 источников
(kn/objective/nspd/nspd_geo/cadastre) + overall_status (worst по critical). Prod-verified
имена таблиц, FILTER/COALESCE корректны, CAST(:d AS interval). +4 теста.
#83: README — GitHub→Forgejo CI/CD, секция 6 subagents, ссылки перемаплены.
#72: docs/Schema_Migrations_vs_Alembic.md — _schema_migrations паттерн (не Alembic),
рекомендация снести пустой alembic-скелет (отдельный PR).

Closes #73
Closes #83
Closes #72
2026-06-13 18:25:32 +00:00