Allternit Driver

​

Overview

allternit-api executor (lease, approval, audit, scaling)
        │  JSON-RPC, one JSON object per line
        │  this device: private unix socket / loopback TCP + token
        │  guests: guest exec channel -> allternit-driver-rpc -> the socket
Allternit Driver (one sidecar)
  ├─ element map: our ids + map versions
  ├─ router: per op × app × OS, learned from latency and failures
  ├─ locks: one reader and one input owner per window
  ├─ arc engine (forked arc-driver, macOS, in process)
  ├─ Cua engine (Cua Driver 0.34, every OS it is installed on)
  └─ native engine (guest images): AT-SPI + xdotool/scrot on Linux,
     UI Automation + user32 on Windows

​

Engines

​

Operations

​

Element ids and map versions

• An element’s id is a hash of its role, its name and its tree path. A path segment is role:n, where n counts only earlier siblings with the same role and name, so a new row with another name doesn’t shift anyone’s id. Identical twins under one parent also mix in their bbox, measured from the window’s corner and rounded to 8 pt, so moving the window doesn’t change ids.
• Ids are the same whichever engine read the window, and the same across reads while the element is there. A batch can therefore use ids from before its first step after earlier steps have changed the window.
• crops: true adds crop to each element: an 8×8 average hash of its pixels, taken from one window capture (macOS). The vision fallback and replay use it to recognise an element whose tree data changed.
• Each window keeps its last 8 map versions. A version changes only when something a model can see changed: ids, names, values, bounds, enabled or focus. since answers with added, changed and removed from that version, or the whole map with reset: true when it has aged out.
• An action carries the version it was planned on. If the window has moved on, the action is refused with stale_version and the fresh map, the same freshness guard arc applies to its own snapshots.
• Before an act or run_batch checks its version, the map is brought up to date with every notification so far, so a change the app announced a moment ago refuses an action planned before it. The refusal carries the current map (no extra walk).

​

Live element map

• One subscriber per window. On macOS one AXObserver per app runs on one run-loop thread (live/macos.py, the AXHub). It feeds arc’s change journal and settle counter (drop-in adapters installed with arc’s app_factory hook), each window’s walk cache, and the live map. Upstream arc ran three observers per app, each with a thread polling every 20–50 ms; the hub sleeps until an app posts. Both engines, the decision runtime and every caller read the same map.
• Notifications (registered on the app): value, title, created, destroyed, focused element, focused/main window, window created/moved/resized/minimized, layout, row count, rows expanded/collapsed, selected children/rows/text/cells, scroll position, menu opened/closed, sheet/drawer created, app hidden/shown. Web areas also get their own subscription, since a page posts on its web area.
• Patching. A notification marks the window dirty and wakes the patch worker. A patch is arc’s incremental walk: only the elements the notifications named are read again, plus a child-list check of containers that catches rows added without a notification. The map is rebuilt from that walk without further accessibility calls, and its version goes up only when something visible changed. A read of a dirty window patches first, so a read never answers older than the last notification.
• Full walks happen only on first sight, on a window-level change (a window, sheet or menu came or went, or focus moved to another window), when more than 40 structural notifications arrive in one burst, on fresh: true, or when the staleness probe fails. The probe runs every 2 s per watched window and costs two accessibility calls: the window’s child count and the app’s focused element.
• Re-reads cost no accessibility call. A clean window is answered from the map (cached: true); the app’s current window is also remembered until a window-level notification, so read_ui with only an app makes no call either.
• Fallback (live.state: degraded). When an app accepts no notifications, its observer fails, or the probe catches a change no notification announced twice, the window is read on a timer: the map is reused for at most 0.5 s, then read again (narrowed to 200 elements on the Cua engine), and every answer says degraded with the reason. On macOS every read is arc’s. Where the router picks Cua (Windows, Linux, or an app arc can’t read), Cua’s get_window_state runs only on first sight, after a notification, on a forced refresh (fresh: true), or on a degraded window’s timer.
• Lifecycle. Windows are watched on demand, when first read or acted on (and the prewarmed frontmost app, which is patched only once a caller reads it). A window is unwatched after 5 minutes without use; at most 8 are watched at once (the least recently used goes); each keeps 8 versions. Background patching runs only for windows a caller read in the last minute, at most every 100 ms per window; others are patched when read.
• Safety. The observer callback makes no accessibility call: it counts, queues and returns inside a try. Role lookups run on a separate dispatch thread with a 0.25 s messaging timeout, and watched apps get a 1 s timeout instead of the 6 s default, so a hung app can’t stall the others. Nothing on this path touches arc’s SkyLight input code (the D2 press crash). When an app quits, the probe or the next patch drops its watches and releases its arc state; a relaunch is a new pid, watched on its next read.
• macOS run loop. The sidecar’s main thread now runs the Cocoa run loop (requests are served on a thread). Without it, NSWorkspace never refreshed its list of running apps, so an app launched after the sidecar started answered app_not_running.
• Linux and Windows (D1b: the guest-image engines). The native engines run the structured members on guest images: engines/atspi.py walks the AT-SPI tree with pyatspi (roles mapped to the same AX-style vocabulary arc and Cua emit, so element ids and router decisions are comparable across OSes), acts through AT-SPI action/text/value interfaces, and posts keys and pixels through xdotool; engines/uia.py walks UI Automation with comtypes on one MTA thread, acts through invoke/value/selection/toggle patterns, and posts input through user32. Reads stay behind the same live-map contract: the observer (above) decides when a read is needed, the engine answers the tree. Both observers and engines were written to the macOS design and run for the first time on these images — validate-image.sh / validate-windows-image.ps1 prove hello and a populated read_ui before an image ships. Without pyatspi/comtypes or an accessibility bus, every window reports degraded and the API fails structured calls as driver_unavailable, never silently. Chromium and Electron apps (Chrome, the Allternit app) only publish an AT-SPI tree when accessibility is requested, so the Linux images set ACCESSIBILITY_ENABLED=1 (and GTK_MODULES=atk-bridge) for the desktop session, the headless app service and /etc/environment. Verified on the 1.1.6 cloud computer image (2026-10-10): read_ui on Chrome returns its tree through the AT-SPI engine.

​

Router

• Capability first. An engine only gets an op it can do. Cua Driver 0.34 has no run_actions, so macOS batches run on arc today and the decision says capability. zoom stays with Cua. Cua 0.34’s parse_visual_regions is not used: it needs Cua’s optional cua-perception extension, which is AGPL; the vision fallback proposes its own regions.
• Actions follow the map. An element action goes to the engine that produced the map its id came from, because only that engine holds the element’s handle.
• Learning. For every op and app the router keeps calls, failures, a latency EWMA and the last 10 outcomes. An engine that failed at least half of 3 or more recent calls hands the op to the other engine. When both have 5 or more timed calls and the other is at most half as slow, it takes over. Read-only ops (verify, screenshot, zoom) try the other engine on one call in 20 until it has enough samples to compare. Input never explores.
• Per user. The table is saved in the Desktop app’s data folder (computer-use/driver/router.json). Every routed op is written to computer-use/driver/audit.jsonl with its engine, reason, time and outcome.
• Degraded apps. An empty tree (or Cua’s degraded_reason) marks the app degraded, and reads say so (degraded: true, degraded_reason). The vision fallback takes over for those windows: the router decides per read whether elements come from the tree, from vision, or from both, and records it (source, source_reason, router.sources).

​

One reader and one input owner per window

• Only one tree walk of a window runs at a time. A reader that waited for another one reuses that walk if it finished in the last 250 ms.
• Input to a window has one owner at a time. Input to different windows runs side by side.
• Screen-wide pixel input (pixel_*) lands wherever the pointer or focus is, so it excludes all window input while it runs.

​

Packaging

• This device (Desktop). npm run prepare:allternit-driver (surfaces/allternit-desktop/scripts/prepare-allternit-driver.cjs) copies domains/computer-use/driver into the app’s resources and stages a relocatable CPython 3.12 per target from python-build-standalone, pinned by sha256. macOS targets also get the pinned PyObjC frameworks arc needs. A venv would point at the build machine’s Python, which users don’t have.
• The Desktop app starts the sidecar after Cua Driver, through spawnSidecar, so it and its cua-driver mcp child stop with the app. Python runs with -B -s -E: no bytecode written into the signed bundle, no user site-packages or PYTHON* variables.
• The this-device socket is computer-use/allternit-driver.sock in the app’s data folder, mode 0600 in a 0700 folder. Windows listens on 127.0.0.1 on a random port and every request carries a per-launch token; allternit-api connects there over loopback TCP (env ALLTERNIT_DRIVER_ENDPOINT + ALLTERNIT_DRIVER_TOKEN), so this-device read_ui/act/pixel input all go through the one driver on Windows too — the Cua CLI fallback remains only for Desktops too old to ship the sidecar. allternit-api gets the socket in ALLTERNIT_DRIVER_SOCKET and sends this Mac’s pointer and keyboard input through pixel_*. hold_key still posts CoreGraphics events itself. If an older Desktop app runs without the sidecar, input falls back to Cua Driver’s CLI.
• Guests (D1b). domains/computer-use/driver/packaging/ installs the driver into every image that ships to users: the Incus desktop/cloud/headless builds (cmd/allternit-computer-cloud/guest/build-*.sh and cloud-init.yaml), the sandbox container (core/sandbox/container/Dockerfile.driver), the tart VMs (infrastructure/tart-host/build-image.sh, as a session autostart entry), and Firecracker rootfses (ALLTERNIT_DRIVER_DIR in drivers/firecracker/src/rootfs.rs). Linux guests run it as allternit-driver.service on /run/allternit/driver.sock; Windows guests run it as a logon scheduled task on loopback TCP with a launch token; both expose the allternit-driver-rpc forwarder that allternit-api drives through the guest exec channel. xdotool/scrot and the PowerShell input helpers remain installed as the explicit fallback when the driver is down. validate-image.sh and validate-windows-image.ps1 check the service, the hello capability report and a real read_ui before an image ships.
• The socket is computer-use/allternit-driver.sock in the app’s data folder, mode 0600 in a 0700 folder. Windows listens on 127.0.0.1 on a random port and every request carries a per-launch token; allternit-api connects there over loopback TCP (env ALLTERNIT_DRIVER_ENDPOINT + ALLTERNIT_DRIVER_TOKEN), so this-device read_ui/act/pixel input all go through the one driver on Windows too — the Cua CLI fallback remains only for Desktops too old to ship the sidecar.
• allternit-api gets the socket in ALLTERNIT_DRIVER_SOCKET and sends this Mac’s pointer and keyboard input through pixel_*. hold_key still posts CoreGraphics events itself. If an older Desktop app runs without the sidecar, input falls back to Cua Driver’s CLI.

​

Measurements

• First read. It includes the one-time setup for an app: the notification watch, the window lookup and the walk. TextEdit misses 150 ms because its accessibility answers are slow (about 3 ms per element on this Mac; upstream arc measures the same). While computer use is active, and once at launch, the driver prepares whichever app comes to the front, so the read that follows a switch is a 4–5 ms cached answer. The live map (below) replaced these D1 figures for re-reads.
• Batch. The middle step is slower because Calculator posts no notification for +, so arc waits out its 0.6 s no-reaction window before it calls the step settled.
• Under heavy load (load average 30–200 from other work on the same Mac) walks were 3–10 times slower, and so was upstream arc. Re-reads stayed under 25 ms.

​

Live map (E4)

• A change made outside the driver (a pixel_click on Calculator’s “8”, typing into TextEdit with pixel_type_text): the first read_ui with since after the input already had the new version and the right diff (display 78; the text area’s new value), 0.7 ms and 9 ms after the input call returned.
• Stale version: act with the version from before a click came back stale_version with the current map in 5 ms.
• Idle: 0.2–0.4 % CPU over 30 s while watching 4–6 windows (one of them a terminal posting over a thousand notifications), measured in three runs.
• App quit and relaunch: after Calculator quit, read_ui answered app_not_running and its watch was gone within one probe; after the relaunch the first read took 28 ms and the re-read 0.45 ms. The sidecar kept running.

​

Baseline

​

Related

• Computer toolset: the contract and executor that call the driver
• Computer use