New trazum eval command and evaluate() API: run both prompt versions over a
set of inputs and report whether the optimisation changed the answers. Every
other number Trazum reports is arithmetic; this is the one question arithmetic
cannot answer.
The design decision that makes it worth anything. A model asked the same question twice does not answer identically, so "the optimised prompt diverged on 3 of 10 cases" means nothing on its own — it might be better than the original manages against itself. The original therefore runs twice per case first, and that self-agreement is the yardstick the rewrite is judged against. It costs a third call per case and it is the only reason the verdict means anything.
- Four verdicts:
indistinguishable,within-noise,diverges, andinconclusive— the last for when the original cannot agree with itself often enough to judge anything against. A confident verdict off an inconsistent baseline would be worse than admitting the test does not work. - Exits 1 on
diverges, so it can gate a pull request the same waytrazum checkgates a token budget. - Prints the call count before spending anything.
- A template gets its first placeholder filled rather than the input appended: appending would test a prompt nobody runs.
- Cases come from a file, one per line (
#comments and blanks ignored) or a JSON array. A file that merely starts with[falls back to line mode rather than erroring. - Bounded concurrency, default 3. The baseline pair stays sequential within a case: issuing both at once invites a provider to serve one from cache and report a variance of zero.
- Tests grow from 179 to 196.