Skip to content
Trazum

Gating on cost, in whatever CI you already run

Documentation / CLI setup · Gateway · Schedules

Trazum is one binary with meaningful exit codes. That is the whole integration story, and it is deliberate: a tool that only gates on one vendor's CI is a tool half its readers cannot use, and every vendor-shaped convenience is a second code path that rots.

Exit codes are the contract.

CodeMeans
0Every gate that ran, passed — or was waived, on the record.
1A gate failed, or the run could not be completed.

Commands that gate: trazum check (token budgets and the baseline), trazum profile (money budgets, from flags or spend in the config), and trazum verify --gate (a plan's promises).

Everything else exits 0 and reports. trazum doctor surveys and never gates — its model recommendation is a keyword heuristic, and gating a build on a keyword heuristic teaches people to re-run until green, which costs more than the tool saves.

GitHub Actions#

The repository ships an Action, which additionally comments on the pull request:

- uses: Davmunrey/Trazum@<commit-sha>  # pin to a commit, never a tag
  with:
    target: prompts/          # a file, or a directory using the config's budgets
    max-tokens: '900'

The input is target. file is a deprecated alias kept so existing workflows keep working. There is no path input, and giving one fails the run with "Set the 'target' input to the prompt file or directory to check, or 'usage-log' to gate the spend" — the Action refuses rather than checking something nobody asked about.

Or use the binary directly, exactly as below.

What the change does to each prompt's monthly cost#

On a pull request the Action also reads the base branch's prompts and puts a table in the same report, and so in the same comment: for each prompt the change touches, its tokens on the base and on the head, what it costs on each, and the difference. It is trazum check <target> --base <sha>, with the pull request's base commit, and it adds no second comment: the report is still one comment per comment-key, replaced on every push.

permissions:
  contents: read
  pull-requests: write
steps:
  - uses: actions/checkout@<commit-sha>
    with:
      fetch-depth: 0          # the base commit has to be in the clone
      persist-credentials: false
  - uses: Davmunrey/Trazum@<commit-sha>
    with:
      target: prompts/
      comment: 'true'
      github-token: ${{ secrets.GITHUB_TOKEN }}
      max-monthly-increase-usd: '25'   # optional; off unless set

What each figure is, and what it is not:

  • The base comes from the clone, never from the network. It is read with git show, and a base commit the clone does not have is said, with the fix (fetch-depth: 0), rather than fetched or stood in for by the working tree. actions/checkout fetches one commit by default, so without that line the Action does not compare and the report is what it was before this existed. With max-monthly-increase-usd set, which cannot be judged without the base, it compares anyway and fails with that fix.

  • A month needs a volume somebody stated. It is volumes in trazum.config.json, calls a month for a label that labels maps to the prompt file, the same map trazum plan --edits reads:

    {
      "labels": { "support": "prompts/support.txt" },
      "volumes": { "support": 30000 }
    }

    A changed prompt with no stated volume is priced per call, and the report says how many are. Several labels sending one file add up; one of them without a volume leaves the file per call. A prompt marked inside a source file is always per call, because a label maps a file and not a prompt in it.

  • Input tokens at the input price, both sides at the same prices. The model is usage.model, or Trazum's default, and the report says which. Output is left out, because a prompt edit does not set how much the model writes back. Every row names its price per million input tokens and the day that price was read from the provider's page: the catalogue's review date for that provider, or a local overlay's own date. Both sides use the same price list, so a provider repricing cannot read as a change the pull request made.

  • max-monthly-increase-usd is the one gate, and it is off unless set. It fails the build when the prompts with a stated volume cost more than that many dollars a month more on the head than on the base, and it names how many changed prompts it could not judge. Set off a pull request, or with usage-log, it is refused; when the base cannot be read it fails, because a gate that judged nothing is not a pass.

Locally, against any ref the clone has:

trazum check prompts/ --base origin/main --max-monthly-increase-usd 25

A weekly summary in the repository#

The same Action can write the week down and post it, so nobody has to remember to run bill. It reads a usage log the job can see (committed, or fetched by an earlier step), writes trazum weekly's summary, and posts it to the one place you name. It is off unless weekly-summary is set.

on:
  schedule:
    - cron: '41 7 * * 1'
permissions:
  contents: read
  issues: write
jobs:
  weekly:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: Davmunrey/Trazum@<commit-sha>  # pin to a commit, never a tag
        with:
          usage-log: logs/usage/
          weekly-summary: comment
          weekly-summary-issue: '12'
          github-token: ${{ secrets.GITHUB_TOKEN }}

weekly-summary is the owner's choice and there are two answers. issue opens one issue for the week, titled with its dates. comment adds one comment for the week on the standing issue weekly-summary-issue names, which is the only place it comments. Any other value stops the run. The repository is always the one the workflow runs in: no input and nothing in the event moves it.

  • A week is posted once. The summary ends in a marker naming the week's last day, and a re-run of the same week replaces its own post instead of adding another; the next week gets its own. Only a post made by a bot counts as its own, so an issue or comment somebody else wrote is never edited, even one that quotes the marker. That recognition needs the workflow's own GITHUB_TOKEN: with a personal access token the author is a user, and a re-run of the same week would post again.
  • It sets nothing else. No label, assignee or milestone: those are yours.
  • The endpoints are the four it needs: list the repository's issues or the standing issue's comments, create an issue or a comment, and edit the one it created. The token is the workflow's, handed to one script and never to the CLI. It needs issues: write and nothing more.
  • It never fails the build for not posting. A token too narrow, an issue that is not there or an API that refuses each leave a notice, and the summary is on the run's summary page either way. A summary that could not be written does fail the run, since nothing was said.
  • It is a mode of its own. With weekly-summary the run takes usage-log (and optionally until, label, config, locale) and refuses target and every gate (max-usd, against, max-growth-usd, max-cache-loss-usd, since): one run says one thing. Run a gate in its own step.

A plan that opens its own pull request#

Some of what a plan says to do is an edit Trazum can already write: a prompt trimmed by the safe rules. With plan-pull-request: 'true' the Action makes the plan with trazum plan --edits, makes those edits on a new branch, and opens one pull request whose body is the plan: each change, the measured figure it is expected against, and how trazum verify checks it. It is off unless asked for, and it is the one step of the Action that writes to the repository, so give it a job of its own and the two permissions there:

on:
  schedule:
    - cron: '23 6 * * 1'
permissions:
  contents: read
jobs:
  plan:
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v7
      - uses: Davmunrey/Trazum@<commit-sha>  # pin to a commit, never a tag
        with:
          usage-log: logs/usage/
          plan-pull-request: 'true'
          github-token: ${{ secrets.GITHUB_TOKEN }}

The files come from labels in trazum.config.json, which says which prompt file each workload sends. Only a file git tracks is edited, never a source file, and only by the safe rules: no model writes any of it.

  • The branch carries the plan's edits and nothing else. Each new text is made again from the default branch's file, which has to be the file the plan was made against (by SHA-256), and has to come out as the text the plan promised (by SHA-256). One file that moved on refuses the whole pull request, because a body listing a change the branch does not make would not be the plan. A file's mode is kept.
  • No pull request with no change. A plan with no edit, or one whose edits are all on the default branch already, creates nothing: no branch, no commit.
  • One pull request per plan. The branch is trazum/plan- and a hash of the plan's edits, so the same edits made into a plan next week are the same plan. An open pull request from it is reused; a closed or merged one is the owner's answer and is not opened again; a branch of that name with no pull request is left alone.
  • Never the default branch, never a merge. The one ref it creates is the new branch, once. It calls no merge endpoint, and sets no label, reviewer or assignee. The repository is the one the workflow runs in and the base is its default branch: no input and nothing in the event moves either.
  • No saving is projected. The figure each edit is held to is what the workload cost in the log, measured. The file's tokens before and after are the estimator's and are labelled estimated. trazum verify plan.json --against <the newer log> says afterwards whether the edit is in the file and whether the calls got shorter.
  • It never fails the build for not opening. A token without the two permissions, a file that moved on or an API that refuses each leave a notice, and the plan is on the run's summary page either way. A plan that could not be made does fail the run.
  • It is a mode of its own. It takes usage-log (and optionally config and locale) and refuses target, weekly-summary and every gate and window: it proposes, over the whole log, and gates nothing.

GitLab CI#

prompt-cost:
  image: node:20
  script:
    - npx --yes @trazum/cli check prompts/ --max-tokens 900
    - npx --yes @trazum/cli profile logs/ --max-usd 400
  artifacts:
    when: always
    paths: [trazum-summary.md]

--markdown-out trazum-summary.md on either command writes the same report as a Markdown file, which GitLab renders as a job artifact.

Jenkins#

stage('Prompt cost') {
  steps {
    sh 'npx --yes @trazum/cli check prompts/ --max-tokens 900'
    sh 'npx --yes @trazum/cli profile logs/ --max-usd 400 --markdown-out trazum.md'
  }
  post {
    always { archiveArtifacts artifacts: 'trazum.md', allowEmptyArchive: true }
  }
}

A failing gate throws on the sh step, which is what marks the stage failed. Nothing Jenkins-specific is needed and nothing Jenkins-specific is offered.

CircleCI#

jobs:
  prompt-cost:
    docker: [{ image: cimg/node:20.11 }]
    steps:
      - checkout
      - run: npx --yes @trazum/cli check prompts/ --max-tokens 900
      - run: npx --yes @trazum/cli profile logs/ --max-usd 400

The sign-off hook, for contributors to this repository#

.github/workflows/dco.yml requires every non-merge commit in a pull request to carry a Signed-off-by trailer, and it is a required check. Forgetting git commit -s once is expensive out of all proportion to the mistake: adding the trailer to a later commit does not help, and neither does reverting, because the workflow walks base..head and the unsigned commit is still in that range. The only fix is rewriting history and force-pushing.

So install the hook and stop thinking about it:

ln -s ../../scripts/prepare-commit-msg .git/hooks/prepare-commit-msg

It adds the trailer from the identity git is already about to record as the author, which is the same one git commit -s would use. It never adds a second one, it skips merges because the workflow does, and TRAZUM_SIGNOFF_HOOK=0 turns it off. Nothing installs it for you: a repository that adds git hooks during npm install is a repository that runs code somebody did not ask for.

signoff-hook.test.js reads the pattern out of the workflow and asserts the hook's own output matches it, so the two cannot drift on what a trailer looks like — a hook writing a trailer the check rejects would be worse than no hook, because it would look like the problem was solved.

A pre-commit hook#

The one place where speed matters more than completeness, so gate on the files this commit touches and nothing else — one pipe, no shell loop:

#!/bin/sh
# .git/hooks/pre-commit
git diff --cached --name-only | npx --yes @trazum/cli check --files-from - || {
  echo "Prompt over budget. Commit with --no-verify if you mean it."
  exit 1
}

--files-from - reads the list from stdin — exactly the shape git diff --name-only produces. Paths that are not prompt files, paths the config ignores, and deletions are dropped and counted out loud; a commit that touches no prompts passes without ceremony. Budgets apply per file from the config as always. The baseline gate is deliberately skipped here — it compares the whole repository against the committed record, and a partial list would read as removals — so keep trazum check . in CI for it.

--no-verify is named on purpose. A hook somebody cannot get past is a hook somebody uninstalls, and an uninstalled hook gates nothing.

Verifying a plan in CI#

The plan is the one output meant to be committed (format). Once it is in the repository, a broken promise is as visible as a broken test:

trazum plan logs/ -o plan.json          # once, reviewed in a pull request
trazum verify plan.json --against logs/ --gate

--gate fails on a promise that did not arrive, and on fields-stopped — a team that degraded its own log must not pass on the strength of the silence. A workload that genuinely vanished, or a tier the log never recorded, reports cannot-tell and fails nothing. Three outcomes, never two.

Gating performance without gating the runner#

A wall-clock budget in CI fails on weather: shared runners lie about time. The bench's gate holds a ratio instead — each workload timed against a calibration loop in the same process, so the runner's speed cancels out of the number being judged:

trazum bench --record trazum.bench.json                  # once, committed
trazum bench --against trazum.bench.json --max-ratio 3   # every build

Exits 1 when any workload is past its recorded ratio times the stated factor, on stderr, the way check gates. The factor is yours: 3× is a tripwire for the quadratic-regex class of regression; a tighter factor buys sensitivity at the price of noise. A baseline whose version the binary does not know is a loud error naming --record, never a best-effort read — the file is committed, so it crosses upgrades. This repository runs exactly this step on its own CI.

Waivers, and their record#

A gate failure a team has looked at and decided to live with goes in the config with a reason and an end date:

{
  "spend": { "maxUsd": 400 },
  "waive": [
    { "gate": "maxUsd", "reason": "the vendor migration lands in March", "until": "2026-04-01" }
  ]
}

All three fields are required. A waiver with no end date is a finding deleted with extra steps, and a reasonless one is a silence nobody can audit. The failure still prints — waived is shown as waived, never hidden — and only the exit code goes quiet.

Every use is written down. When a waiver silences a gate, Trazum appends a dated line to .trazum/waivers.jsonl: the gate, the reason and expiry as they stood at that moment, the commit when CI exported one, and the figures the gate judged. trazum history reads those lines back and can finally say that a finding has been waived nine times across four months under one unchanging sentence whose deadline moved three times — which is a decision nobody is revisiting, and something no config could ever have told you.

Two rules make that record trustworthy:

  • Nothing is back-filled. The history begins the day recording began, and says so. A past reconstructed from today's config would be a guess presented as evidence.
  • A waiver nobody's build has hit is dead config, not a habit, and the report keeps the two apart.

Commit .trazum/waivers.jsonl if you want the record shared, or leave it out of version control if you only want it locally. Trazum never rewrites it and offers no command to clear it: a record of decisions the tool can erase is a record nobody can rely on. Delete the file yourself if you mean to.

What is deliberately not here#

No vendor plugin, no marketplace listing, no wrapper action for each CI. They would each be a second code path with its own bugs, its own release cadence and its own way of drifting from the exit codes above. One binary, documented recipes, and any limitation stated rather than papered over.