06 · the referee
Guignet
guignetpre-1.0 · from sourceGuignet is the Sothis suite’s referee. It mines your own git history into replayable, validity-checked tasks, runs any model, agent or config against them in isolated worktrees, and hands you the only leaderboard that matters: one measured on your code. Nothing is uploaded.
Why it exists
Public benchmarks are saturated and measured on other people’s repositories; eval SaaS wants you to upload your private code first. Neither answers the only question you actually have — which model, agent and config is best on *this* codebase. Being local-first is what makes that question askable at all.
What it does
mine reconstructs task-shaped commits — a fix plus the tests that prove it — into a prompt, a base commit and a held-out verifier, with a leak firewall enforced by a CI boundary check.
gate admits a task only after replaying it: the verifier must fail at base and pass at the real fix, k times each. Flaky tasks are discarded rather than patched, and the soundness rate is published.
run executes attempts in disposable worktrees behind a hard run-wide spend cap, with cost parsed from the harness’s own transcript instead of self-reported.
score is binary and judge-free against the held-out verifier, with contamination controls: a training-cutoff split and regurgitation flags.
report renders one self-contained HTML file that opens offline, plus a --json twin of every number in it.
Install and run
From source
$bun installClone the repo first — there is no published release yet. Needs Bun ≥ 1.3 and git. Windows: use WSL2, which gets the full POSIX treatment; native Windows is experimental and guignet doctor says so.
Run the pipeline
$guignet mineThen gate, run, score and report. Every stage is idempotent and resumable, and every command takes --json.
Installing the whole suite at once is a different path —the plugins first, then every CLI in one shot.