Implementation: IMPLEMENTED (manifest binding, go-live gate, canary) · Exposure: INTERNAL. See status labels. A decision is only allowed to act when its confidence is calibrated: when a stated 0.9 is right about 90% of the time on data the component has not seen. Calibration is measured per primitive and recorded in a manifest.

​
Architecture contract

​
The calibration manifest

DecisionCalibrationManifestV1 is the deployed unit of calibration. It is bound to the exact component revision, tokenizer, quantization, serving runtime, question, candidate schema, candidate set and threshold profile, and carries a fingerprint of that scope. If anything in the scope changes, the manifest no longer binds and the decision falls back.

​
The go-live gate

S1 stays in shadow until calibrated. For each primitive, going live requires:
  • a calibration artifact bound to the exact scope above
  • expected calibration error (ECE) of at most 0.05 on held-out data
  • at most 5% observed error on the subset the decision would act on automatically
  • a statistically meaningful held-out sample, reported with confidence bounds
  • reversible, low-consequence decisions only at first
Hard policy and deterministic verification stay authoritative. Outside the calibrated coverage region the decision abstains or escalates. Each primitive’s authority expands only after its own evaluation passes. Agreement with another model is not a go-live criterion. Two models agreeing says nothing about whether either is right. Tracked metrics: accuracy, F1, Brier score, negative log-likelihood, ECE, coverage at risk, and sensitivity to flipping or reordering candidates. Why: a decision that acts automatically is only as safe as its confidence is honest. A per-primitive gate on held-out error lets S1 take on work one question at a time, with evidence.

​
Current implementation

On main, the decision runtime has manifests with scope fingerprints and exact-binding checks, the go-live gate as code (including minimum sample floors, bounds on the auto-act error, and the reversible-only rule), and the metrics above. The gate cannot take agreement with another model as an input.

​
Passing the gate

The gate on main is Q26. Held-out data the component was not trained on is split in two: one split tunes the confidence threshold, the other certifies it. A primitive passes when the Clopper-Pearson upper bound on its auto-act error over the certification split stays under its error budget (5% at 95% confidence by default). Permission and vendor-consequential primitives are never eligible to act automatically, whatever their scores. As of 2026-10-02, one primitive has passed: judge.first_pass.tool (is a tool call safe to run on the first pass), with an auto-act error upper bound of 0.58% at full coverage on 2,937 held-out decisions. The route, model-routing, error-classification and node-judge primitives did not pass and stay in shadow.

​
Canary

Passing the gate does not switch a primitive on. It makes it eligible for a canary, the only way a Q26 manifest acts:
  • Exposure budget. At most N live decisions per day per primitive (20 to start). Past the budget, decisions go back to shadow until the next day.
  • Audits. Every live decision is audited, plus a random 2–5% slice of shadow decisions (3% by default).
  • Automatic rollback. Audited outcomes feed a Bernoulli CUSUM. If the observed error drifts above the budget, the primitive rolls back to shadow on its own.
  • Growth. After 50 clean audits the budget can be doubled.
Canary state is kept per primitive in ~/.allternit/system-one/canary.json and managed with system-one canary <status|enable|grow|sync|rollback>. enable refuses a primitive that the Q26 report does not mark eligible.

​
Live mode

S1 consults the canary only in live mode. In Allternit Desktop, live mode is a saved System One setting (liveMode in ~/.allternit/system-one/settings.json). It is off by default and can also be set with ALLTERNIT_S1_MODE=live. Changing it restarts the System One process so it applies. While live mode is on, Desktop runs system-one canary sync about a minute after S1 starts and every 15 minutes after that, so audited outcomes reach the rollback check. With live mode off, everything stays shadow-only, including primitives in canary: S1 records what it would have decided and takes no action.

​
Availability

​
Developer relevance

If you want a decision primitive to act, you need labelled held-out data for that exact question and component, a passing Q26 report, a canary, and live mode on. Until then it runs in shadow and its results are recorded, not used.