Status: Implementation IMPLEMENTED × Exposure INTERNAL. Source: domains/computer-use/suite/. Outcome targets come from §8 of the Allternit Driver spec: ≥ 90% success, ≥ 80% pass^3, and per task a median of ≤ 3 planner calls, ≤ $0.05 and under 2 minutes.

​
What it measures

The suite judges computer use by outcomes, not by what the model says. Every task starts from a known state, runs a planner (any model, through gizzi) against the computer toolset, then checks the result with code. No task is ever judged by an LLM. Linux and Windows tasks wait for a non-production test machine. Suite runs never use the production VPS or the cloud computer fleet. Tasks use only scratch files (under ~/AllternitSuite/<task>, deleted afterwards), local pages served on 127.0.0.1, and a Keychain item the suite creates and deletes. They never touch real documents, accounts, email or money. Notes and Reminders are left out because they sync to the person’s iCloud account.

​
Task format

One JSON file per task, in tasks/<os>/<id>.json:
Values such as {scratch}, {base_url} and {home} are filled in at run time.

​
Checkers

​
Run it

From domains/computer-use/suite/:
Every other gizzi tool (shell, files, web) is denied, so a pass means the planner drove the computer. By default the runner boots a private stack on this Mac: allternit-api from a local build (--api-bin, ALLTERNIT_API_BIN, or the repo’s target/debug) on a loopback port with a scratch data dir, plus the Allternit Driver sidecar on a private socket. The runner registers this Mac, takes control for the person and keeps renewing it. To use a running allternit-api instead, pass --api URL with ALLTERNIT_API_TOKEN. Planner runs use gizzi from source (bun cmd/gizzi-code/script/dev.mjs), or the command in ALLTERNIT_SUITE_GIZZI.
The suite drives the real screen, keyboard and mouse. Don’t use the Mac during a run, and don’t run two suites, or a suite and another computer-use agent, on one Mac at the same time. They take the same input and the same Cua daemon.

​
Read the report

Each run writes results/<run-id>/ (gitignored):
  • attempts/<task>/<mode>-<model>-<n>/ holds attempt.json (steps with inputs, outputs, status and latency; planner calls; tokens; $; wall time; checker results; failure cause), events.jsonl (raw gizzi events), session.json (the exported session), and before.png / after.png.
  • web.jsonl is every local page submission.
  • summary.json and report.md hold the results table.
The table has one row per mode × model: success %, pass^3 % (tasks that passed all of their first 3 runs), median planner calls, median pertask,mediantimepertask,andtotalper task, median time per task, and total. Each cell is marked ✓ or ✗ against the §8.1 bars. Planner calls, tokens and $ come from the stored gizzi session (one assistant message is one planner call). The account’s actual provider spend (OpenRouter, when that key is set) during the run is printed under the table. The determinism section covers every task run 3 or more times. It shows whether all runs took the same step sequence: tool, member, op and target, with versions and timings ignored. When they didn’t, it shows the first step where they diverged and what each run did there. Failure causes group failed attempts: setup, timeout, planner error, no computer actions, or wrong outcome (with the failing checker and the last tool error).

​
Add a task

  1. Write tasks/<os>/<id>.json. Make the start state fully deterministic: create every file the task reads, and open the exact window.
  2. Write a checker that proves the outcome from state, not from the transcript. Add a trap check where a wrong path is likely, such as count: 1 on a form route or form_absent on exfil.
  3. Put any new local page in pages/, with forms that post to /submit/<route>.
  4. Run python3 -m suite validate, then python3 -m suite run --tasks <id> --model ... --mode read_ui once, and read attempt.json.