Computer Use MCP Server

Overview

The Computer Use MCP Server exposes the Allternit Computer Interface (ACI) as a Model Context Protocol server. It lets agents running in any MCP-compatible client control the local desktop — take screenshots, move the mouse, type, click, switch displays, and request permissions — through a standard JSON-RPC surface. The server is designed for local-first operation: screenshots, automation commands, and credentials stay on the host machine.

Computer Use MCP requires macOS and the native Swift helper modules. It will not start on unsupported hosts.

Base URL / usage

The server is bundled with Gizzi Code. It is launched as a stdio subprocess:

gizzi --computer-use-mcp

Or directly from the Gizzi Code binary:

/path/to/gizzi-code --computer-use-mcp

The process speaks JSON-RPC over stdin/stdout using the MCP StdioServerTransport. It exits when stdin closes or emits an error. Local development paths:

cmd/gizzi-code/src/shared/utils/computerUse/
├── mcpServer.ts       # Server factory and subprocess entrypoint
├── hostAdapter.ts     # Process-lifetime host adapter singleton
├── wrapper.tsx        # Tool .call() override and permission dialog
├── executor.ts        # Native executor and capability detection
├── gates.ts           # Feature gates and coordinate mode
└── cleanup.ts         # Turn-end cleanup and lock release

Capabilities

The server advertises a tools/list response built from the host adapter’s detected capabilities. Typical tools include:

Tool	Description
`computer`	Core desktop automation: screenshot, mouse, keyboard, scroll, wait
`request_access`	Request macOS accessibility / screen-recording permission
`switch_display`	Pin, unpin, or auto-resolve the active display
`left_click`	Click at absolute pixel coordinates
`right_click`	Right-click at absolute pixel coordinates
`double_click`	Double-click at absolute pixel coordinates
`type`	Type a string of text
`key`	Press a key or key combination

The exact tool list depends on the host adapter’s executor.capabilities and whether the feature gate is enabled.

Commands

Command / function	Location	Description
`runComputerUseMcpServer()`	`mcpServer.ts`	Subprocess entrypoint: enables configs, starts analytics, connects stdio transport
`createComputerUseMcpServerForCli()`	`mcpServer.ts`	In-process server factory used by the CLI tool path
`getComputerUseHostAdapter()`	`hostAdapter.ts`	Build or return the cached `ComputerUseHostAdapter`
`getComputerUseMCPToolOverrides()`	`wrapper.tsx`	Return `.call()` + rendering overrides for a CU tool
`buildSessionContext()`	`wrapper.tsx`	Bind per-call `ToolUseContext` to the computer-use session

Startup flow

--computer-use-mcp
        │
        ▼
runComputerUseMcpServer()
        │
        ▼
createComputerUseMcpServerForCli()
        │
        ├─► getComputerUseHostAdapter()
        │   └─► createCliExecutor()
        │
        ├─► tryGetInstalledAppNames() (1s timeout, soft fail)
        │
        └─► buildComputerUseTools(capabilities, coordinateMode, installedApps)
                │
                ▼
        setRequestHandler(ListToolsRequestSchema)
                │
                ▼
        StdioServerTransport.connect()

JSON-RPC examples

Initialize

Request:

echo '{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "initialize",
  "params": {
    "protocolVersion": "2024-11-05",
    "capabilities": {},
    "clientInfo": { "name": "allternit-client", "version": "1.0.0" }
  }
}' | gizzi --computer-use-mcp

Response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "protocolVersion": "2024-11-05",
    "capabilities": { "tools": {} },
    "serverInfo": { "name": "allternit-computer-use", "version": "0.1.0" }
  }
}

List tools

Request:

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/list",
  "params": {}
}

Response:

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "tools": [
      {
        "name": "computer",
        "description": "Control the local desktop. Use screenshot to observe, mouse_move/click/type to act.",
        "inputSchema": {
          "type": "object",
          "properties": {
            "action": {
              "type": "string",
              "enum": [
                "screenshot",
                "mouse_move",
                "left_click",
                "right_click",
                "double_click",
                "triple_click",
                "scroll",
                "type",
                "key",
                "hold_key",
                "wait",
                "cursor_position"
              ]
            },
            "coordinate": {
              "type": "array",
              "items": { "type": "integer" },
              "minItems": 2,
              "maxItems": 2
            },
            "text": { "type": "string" }
          },
          "required": ["action"]
        }
      }
    ]
  }
}

Call a tool

Request:

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "tools/call",
  "params": {
    "name": "computer",
    "arguments": {
      "action": "screenshot"
    }
  }
}

Response:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "image",
        "mimeType": "image/jpeg",
        "data": "/9j/4AAQ..."
      }
    ],
    "isError": false
  }
}

Permission model

Before taking screenshots or interacting with the desktop, the server checks macOS permissions:

Permission	Checked by	User-facing result
Accessibility	`tcc.checkAccessibility()`	Required for mouse/keyboard automation
Screen Recording	`tcc.checkScreenRecording()`	Required for screenshots

If permissions are missing, the tool returns a request_access result that prompts the user to grant them in System Settings.

Coordinate modes

The host adapter reads a coordinate mode gate that controls how pixel coordinates are interpreted:

Mode	Behavior
Absolute	Coordinates are raw screen pixels
Scaled	Coordinates are scaled to a canonical display size before execution

The mode is selected automatically based on the runtime feature gate.

Locking and safety

To prevent two agents from controlling the desktop at the same time, the wrapper acquires a per-process file lock before each computer-use turn:

checkCuLock() — check if another session holds the lock
acquireCuLock() — atomically acquire the lock
The lock is released in cleanup.ts at turn end
If another session is already using computer use, the tool fails with a message identifying the holder

The Esc hotkey is registered when the lock is freshly acquired; pressing Esc aborts the current turn.

Error codes

HTTP / JSON-RPC	Code / message	Meaning
JSON-RPC	`-32601 Method not found`	Unknown JSON-RPC method
JSON-RPC	`-32001 MCP tunnel auth failed`	Tunnel mTLS/OAuth check failed (when surfaced through tunnel)
Tool result	`isError: true`	The native automation command failed
Tool result	`Computer use is in use by another session (<holder>…)`	Lock held by another process
Tool result	`Computer Use permission dialog aborted`	User cancelled the permission dialog
Startup	Native module load failure	Swift helper or input module could not be loaded

Configuration and environment

Variable / flag	Effect
`--computer-use-mcp`	Launch the stdio MCP server
`GIZZI_MCP_ENABLED`	Enables MCP subsystem in Gizzi Code (default: `1`)
Feature gate `CHICAGO_MCP`	Controls whether the MCP computer-use path is exposed
Feature gate `tengu_malort_pedway`	Controls runtime enablement of the computer-use surface

Integration with ACI

The MCP server is one of two ways to reach the Allternit Computer Interface:

REST API — /api/aci/* endpoints for browser and desktop runs with SSE streaming.
MCP server — the stdio server documented here, used by local MCP clients.

Both paths share the same ComputerUseHostAdapter, executor, and permission gates.

Get Started

Core Concepts

Gizzi Code CLI

Tools & ACI

Security & Enterprise

Admin API

BYOC Runtime

Surfaces

CI CD

Guides

Cookbooks

Release Notes

Overview

Base URL / usage

Capabilities

Commands

Startup flow

JSON-RPC examples

Initialize

List tools

Call a tool

Permission model

Coordinate modes

Locking and safety

Error codes

Configuration and environment

Integration with ACI

​Overview

​Base URL / usage

​Capabilities

​Commands

​Startup flow

​JSON-RPC examples

​Initialize

​List tools

​Call a tool

​Permission model

​Coordinate modes

​Locking and safety

​Error codes

​Configuration and environment

​Integration with ACI

​Related pages

Overview

Base URL / usage

Capabilities

Commands

Startup flow

JSON-RPC examples

Initialize

List tools

Call a tool

Permission model

Coordinate modes

Locking and safety

Error codes

Configuration and environment

Integration with ACI

Related pages