feat(ci): tested-tree fast path, image_tag reuse, timestamped Dokku output #13

Merged

john merged 1 commit from feat/promotion-fast-path into main

2026-09-29 09:22:40 +00:00

john commented

2026-09-29 09:18:29 +00:00

Owner

What

Three backward-compatible additions to the shared workflows. All new inputs are optional, and the defaults keep today's behaviour for every @main consumer.

Tested-tree fast path (detect-changes.yml + new tested-tree.yml)
- New inputs: fast_path_branch (default empty = off), fast_path_deploy_branch (main), fast_path_jobs, fast_path_workflow (ci.yml), fast_path_depth (30).
- New outputs: tested_tree, verified_sha.
- On a push to main it checks two things. First, HEAD^{tree} must equal the tree of a recent first-parent dev commit. Second, every named job must have its latest attempt at success in that commit's push run on dev.
- Per-job truth comes from /actions/tasks, not commit statuses. Forgejo posts a skipped job's commit status as success, so statuses cannot prove a job ran.
- Token order: the runner's github.token first, then the optional caller-passed secret fast_path_token.
- It fails closed: any doubt outputs false. The step has continue-on-error, so it can never fail a caller's run.
- tested-tree.yml carries the same step for repos that keep their own change filter (both FDA repos do). The two copies must stay byte-identical, and a test enforces it.
dokku-image-deploy.yml image_tag
- Empty or github.sha: build, push and deploy as before.
- Any other value skips build and push. docker manifest inspect then checks every image at that tag, with three tries. If one is missing, the job fails with a clear error before any SSH. Otherwise it deploys that tag.
- The running-identity assertion (alternate-tags label) is checked against the deployed tag, so reuse mode still verifies exactly what runs.
Streamed, timestamped Dokku output
- git:from-image output now streams live through tee into a temp file, with every line prefixed HH:MM:SS (UTC).
- The retry logic (exit code + last line) reads the raw, unstamped file.
- The exit code goes through a file, so the result is the same with or without pipefail. Tests run both ways.
- ps:rebuild is streamed the same way and still fails the step on error.
- Route resolution, config check, manifest check and ps:inspect are also stamped.
- forgejo-ci has no moreutils/ts, so it uses a while read + date -u loop.

README.md is new and documents the fast path and image_tag. CLAUDE.md gets a pointer to it.

Why

John approved this on 2026-09-29. Every FDA change runs three times (PR, dev push, main push). The main run repeats the dev run's tests and rebuilds the image dev already built. Current main runs take 4.5 min (API) and 6 min (site).

Validation

uv run --with pyyaml==6.0.3 python -m unittest discover -s tests: 33 tests pass. There are 13 new fast-path tests and 8 new deploy tests.
- The fast-path tests use real git repos, including a shallow checkout like actions/checkout's, with curl mocked.
- They cover: tree mismatch (direct push to main), and a job that was skipped, failed, still running, or retried into a failure.
- They also cover: a job that ran on the wrong event, branch or workflow; a refused token; a failed page 2; the page limit; non-newest-first ordering; a garbage body; the off switch; and a wrong ref or event.
- The deploy tests cover: tag and build selection, invalid tags, a present or missing manifest, timestamps on the streamed output, ps:rebuild failure propagation, and identity checked against the reused tag.
The deploy tests also pass with the runner's default bash -e (no pipefail).
Live, read-only: I ran the real step against the production task API.
- For forward-deploy-web main 360b148 it returned tested_tree=true verified_sha=19173d3….
- For haskos-academy main 13895d1 it returned true, 30766f3…. In that run, a bogus first token got a 401 and the fallback token was used.
- I checked the tree-equality premise on the last four promotions in both repos: every main merge tree equals its dev parent's tree.
actionlint 1.7.12 with shellcheck: no new findings. The 10 remaining are pre-existing and unchanged apart from line shifts. yamllint (relaxed, line-length off): clean. ruff check and ruff format pass on tests/.
Nothing was deployed.

Expected timings (dry-run reasoning)

The Dokku phase is now the floor. Site run #107 (retire wait already 10 s) spent 3m20s inside git:from-image, 09:08:13 to 09:11:32, with a cached 2 s build. The academy's streamed git:sync log from run #67 shows where the time goes:

Dokku phase	Time
Container start	~32 s
Network sync + nginx template, twice	~70 s
Postdeploy / schedule	~13 s
Source sync	~22 s
Build	~25 s

Run	Today	Fast path, expected
Site main	6.0 min	≈ 4.2 min: detect ∥ fast path ~0.4, gate ~0.2, deploy setup + manifest ~0.3, Dokku ~3.3
API main	4.5 min	≈ 3.0 min: no Backend/Docker Build, no host source sync or build; Dokku ~2.3

The 2.5 min (site) and 2 min (API) targets are not reachable from the pipeline side alone. Reaching them means cutting host-side Dokku time: the double network/nginx pass, container start, and possibly contabo CPU steal. The new timestamps will show exactly where that time goes on the first run after merge.

Spec Drift Callouts

Neither FDA repo uses detect-changes.yml. Both have inline change filters with different, deliberate semantics: the site excludes .claude/; the academy counts tests/fixtures/*.md as code. So the fast path also ships as tested-tree.yml, and consumers read needs.fast-path.outputs.*, not needs.detect-changes.outputs.*. detect-changes.yml still carries the same inputs for fleet repos that use it.
Commit statuses were rejected in favour of /actions/tasks. Statuses report skipped jobs as success.
feat/image-identity-marker is unrelated and already merged (7526610, July). It is the /etc/ci-image-id marker in the CI image. The identity check that mattered is the deploy's running-sha assertion, which now compares against image_tag.
forgejo-ci had no README. I created one.
forgejo-ci/CLAUDE.md says "push directly to main". This change went through a PR as instructed. The runner facts in that file (capacities, registry IP) are stale; I did not touch them.
Token scope is unverified. I could not confirm that Forgejo 15's automatic github.token may read /actions/tasks. Consumers pass CI_FORGEJO_TOKEN as the fallback, and its scope is also unverified. If both are refused, the log says task API not readable and the run takes the normal path.

Rollout order

This PR first. It is safe alone: nothing changes for any consumer until one passes the new inputs.
Then forward-deploy-web and haskos-academy, in either order relative to each other. Both reference tested-tree.yml and image_tag, which only exist once this is on main, so do not merge them before this one.

Not verifiable without a live run

Whether github.token and/or CI_FORGEJO_TOKEN can read /actions/tasks (the fallback is the slow path).
docker manifest inspect against registry-direct with publisher credentials inside a job container.
Nested workflow-call outputs reaching the caller (the same mechanism as haskydocs-v2's has_code, which works).

🤖 Generated with Claude Code

https://claude.ai/code/session_01CS99mH2YQv9t6iPuAbr21S

## What Three backward-compatible additions to the shared workflows. All new inputs are optional, and the defaults keep today's behaviour for every `@main` consumer. 1. **Tested-tree fast path** (`detect-changes.yml` + new `tested-tree.yml`) - New inputs: `fast_path_branch` (default empty = off), `fast_path_deploy_branch` (`main`), `fast_path_jobs`, `fast_path_workflow` (`ci.yml`), `fast_path_depth` (`30`). - New outputs: `tested_tree`, `verified_sha`. - On a push to `main` it checks two things. First, `HEAD^{tree}` must equal the tree of a recent first-parent `dev` commit. Second, every named job must have its **latest** attempt at `success` in that commit's `push` run on `dev`. - Per-job truth comes from `/actions/tasks`, not commit statuses. Forgejo posts a skipped job's commit status as `success`, so statuses cannot prove a job ran. - Token order: the runner's `github.token` first, then the optional caller-passed secret `fast_path_token`. - It fails closed: any doubt outputs `false`. The step has `continue-on-error`, so it can never fail a caller's run. - `tested-tree.yml` carries the same step for repos that keep their own change filter (both FDA repos do). The two copies must stay byte-identical, and a test enforces it. 2. **`dokku-image-deploy.yml` `image_tag`** - Empty or `github.sha`: build, push and deploy as before. - Any other value skips build and push. `docker manifest inspect` then checks every image at that tag, with three tries. If one is missing, the job fails with a clear error before any SSH. Otherwise it deploys that tag. - The running-identity assertion (`alternate-tags` label) is checked against the deployed tag, so reuse mode still verifies exactly what runs. 3. **Streamed, timestamped Dokku output** - `git:from-image` output now streams live through `tee` into a temp file, with every line prefixed `HH:MM:SS` (UTC). - The retry logic (exit code + last line) reads the raw, unstamped file. - The exit code goes through a file, so the result is the same with or without `pipefail`. Tests run both ways. - `ps:rebuild` is streamed the same way and still fails the step on error. - Route resolution, config check, manifest check and `ps:inspect` are also stamped. - `forgejo-ci` has no `moreutils`/`ts`, so it uses a `while read` + `date -u` loop. `README.md` is new and documents the fast path and `image_tag`. `CLAUDE.md` gets a pointer to it. ## Why John approved this on 2026-09-29. Every FDA change runs three times (PR, dev push, main push). The main run repeats the dev run's tests and rebuilds the image dev already built. Current main runs take 4.5 min (API) and 6 min (site). ## Validation - `uv run --with pyyaml==6.0.3 python -m unittest discover -s tests`: 33 tests pass. There are 13 new fast-path tests and 8 new deploy tests. - The fast-path tests use real git repos, including a shallow checkout like `actions/checkout`'s, with `curl` mocked. - They cover: tree mismatch (direct push to main), and a job that was skipped, failed, still running, or retried into a failure. - They also cover: a job that ran on the wrong event, branch or workflow; a refused token; a failed page 2; the page limit; non-newest-first ordering; a garbage body; the off switch; and a wrong ref or event. - The deploy tests cover: tag and build selection, invalid tags, a present or missing manifest, timestamps on the streamed output, `ps:rebuild` failure propagation, and identity checked against the reused tag. - The deploy tests also pass with the runner's default `bash -e` (no `pipefail`). - **Live, read-only:** I ran the real step against the production task API. - For forward-deploy-web `main` 360b148 it returned `tested_tree=true verified_sha=19173d3…`. - For haskos-academy `main` 13895d1 it returned `true, 30766f3…`. In that run, a bogus first token got a 401 and the fallback token was used. - I checked the tree-equality premise on the last four promotions in both repos: every `main` merge tree equals its `dev` parent's tree. - `actionlint` 1.7.12 with shellcheck: no new findings. The 10 remaining are pre-existing and unchanged apart from line shifts. `yamllint` (relaxed, line-length off): clean. `ruff check` and `ruff format` pass on `tests/`. - **Nothing was deployed.** ## Expected timings (dry-run reasoning) The Dokku phase is now the floor. Site run #107 (retire wait already 10 s) spent **3m20s inside `git:from-image`**, 09:08:13 to 09:11:32, with a cached 2 s build. The academy's streamed `git:sync` log from run #67 shows where the time goes: | Dokku phase | Time | |---|---| | Container start | ~32 s | | Network sync + nginx template, twice | ~70 s | | Postdeploy / schedule | ~13 s | | Source sync | ~22 s | | Build | ~25 s | | Run | Today | Fast path, expected | |---|---|---| | Site main | 6.0 min | ≈ 4.2 min: detect ∥ fast path ~0.4, gate ~0.2, deploy setup + manifest ~0.3, Dokku ~3.3 | | API main | 4.5 min | ≈ 3.0 min: no Backend/Docker Build, no host source sync or build; Dokku ~2.3 | **The 2.5 min (site) and 2 min (API) targets are not reachable from the pipeline side alone.** Reaching them means cutting host-side Dokku time: the double network/nginx pass, container start, and possibly contabo CPU steal. The new timestamps will show exactly where that time goes on the first run after merge. ## Spec Drift Callouts - **Neither FDA repo uses `detect-changes.yml`.** Both have inline change filters with different, deliberate semantics: the site excludes `.claude/`; the academy counts `tests/fixtures/*.md` as code. So the fast path also ships as `tested-tree.yml`, and consumers read `needs.fast-path.outputs.*`, not `needs.detect-changes.outputs.*`. `detect-changes.yml` still carries the same inputs for fleet repos that use it. - **Commit statuses were rejected in favour of `/actions/tasks`.** Statuses report skipped jobs as `success`. - **`feat/image-identity-marker` is unrelated and already merged** (7526610, July). It is the `/etc/ci-image-id` marker in the CI image. The identity check that mattered is the deploy's running-sha assertion, which now compares against `image_tag`. - **`forgejo-ci` had no README.** I created one. - **`forgejo-ci/CLAUDE.md` says "push directly to main".** This change went through a PR as instructed. The runner facts in that file (capacities, registry IP) are stale; I did not touch them. - **Token scope is unverified.** I could not confirm that Forgejo 15's automatic `github.token` may read `/actions/tasks`. Consumers pass `CI_FORGEJO_TOKEN` as the fallback, and its scope is also unverified. If both are refused, the log says `task API not readable` and the run takes the normal path. ## Rollout order 1. **This PR first.** It is safe alone: nothing changes for any consumer until one passes the new inputs. 2. Then forward-deploy-web and haskos-academy, in either order relative to each other. Both reference `tested-tree.yml` and `image_tag`, which only exist once this is on `main`, so **do not merge them before this one**. ## Not verifiable without a live run - Whether `github.token` and/or `CI_FORGEJO_TOKEN` can read `/actions/tasks` (the fallback is the slow path). - `docker manifest inspect` against `registry-direct` with publisher credentials inside a job container. - Nested workflow-call outputs reaching the caller (the same mechanism as haskydocs-v2's `has_code`, which works). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CS99mH2YQv9t6iPuAbr21S

john added 1 commit

2026-09-29 09:18:30 +00:00

feat(ci): tested-tree fast path, image_tag reuse, timestamped Dokku output

Workflow Tests / shell-regression (pull_request) Successful in 11s

Details

release-checkpoint Human-approved exact production head

e11bd80daa

- detect-changes.yml: optional fast_path_* inputs and tested_tree /
  verified_sha outputs. On a push to main whose tree equals a recent dev
  commit whose push run passed the named jobs (task API, latest attempt,
  not skipped), outputs tested_tree=true. Off by default; fails closed;
  continue-on-error so it never fails the caller.
- tested-tree.yml: the same step as its own reusable workflow for repos
  that keep their own change filter (byte-identical, enforced by tests).
- dokku-image-deploy.yml: optional image_tag. Non-empty and not
  github.sha skips build+push, requires the tag in the registry
  (docker manifest inspect, clear error), deploys it and asserts the
  running container against that tag. Dokku output now streams live
  through tee with a UTC timestamp per line; retry logic reads the raw
  file. Route, config check and ps:inspect phases are stamped too.
- README.md documents the fast path and image_tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CS99mH2YQv9t6iPuAbr21S