Computer Use SDK

The Computer Use SDK (@allternit/computer-use) is a thin, type-safe HTTP client for the Allternit Computer Use Engine. It covers browser automation, desktop control, vision proposals, parallel runs, sessions, approvals, and real-time event streaming over SSE.

Overview

Use this SDK when you want to:

Drive browsers and desktop environments from TypeScript or Node.js.
Run tasks in intent, direct, or assist execution modes.
Stream execution events and handle approval gates programmatically.
Manage sessions, runs, and canonical environments through a single client.

Installation

npm install @allternit/computer-use

For Node.js environments, also install the eventsource peer dependency for SSE support:

npm install eventsource

Quick start

import { AllternitComputerUseClient } from '@allternit/computer-use';

const client = new AllternitComputerUseClient({
  endpoint: 'http://localhost:8080',
  timeout: 60000,
});

const result = await client.executeIntent('Fill out the contact form', {
  target_scope: 'browser',
  timeout_ms: 30000,
});

console.log(result.run_id, result.summary);

Client configuration

const client = new AllternitComputerUseClient({
  endpoint: 'http://localhost:8080', // Required
  timeout: 60000,                    // Optional
  apiKey: 'your-api-key',            // Optional
  headers: {                         // Optional
    'X-Custom-Header': 'value',
  },
});

Execution modes

Mode	Method	Use case
`intent`	`executeIntent(task, options)`	Natural language task.
`direct`	`executeDirect(actions, options)`	Explicit action list.
`assist`	`executeAssist(task, options)`	Requires explicit approval.

Intent mode

const result = await client.executeIntent(
  'Fill out the contact form with name John Doe',
  {
    target_scope: 'browser',
    timeout_ms: 30000,
    max_steps: 50,
  }
);

Direct mode

const result = await client.executeDirect(
  [
    { kind: 'click', target: { selector: '#contact-button' } },
    { kind: 'type', target: { selector: '#name' }, input: { text: 'John Doe' } },
  ],
  { target_scope: 'browser' }
);

Assist mode

const result = await client.executeAssist(
  'Delete all temporary files',
  { target_scope: 'desktop' }
);

if (result.status === 'needs_approval') {
  await client.approve(result.run_id, {
    decision: 'approve',
    comment: 'User confirmed',
  });
}

Core methods

Method	Engine endpoint	Description
`execute(request)`	`POST /computer-use/execute`	Canonical execution request.
`executeDirect(actions, options)`	`POST /computer-use/execute`	Direct mode shortcut.
`executeIntent(task, options)`	`POST /computer-use/execute`	Intent mode shortcut.
`executeAssist(task, options)`	`POST /computer-use/execute`	Assist mode shortcut.
`getRun(runId)`	`GET /computer-use/runs/{run_id}`	Get run status.
`listRuns(sessionId?, status?)`	`GET /computer-use/runs`	List runs, optionally filtered.
`getRunEvents(runId, afterIndex?)`	`GET /runs/{run_id}/events`	Get event history.
`cancelRun(runId, request?)`	`POST /runs/{run_id}/cancel`	Cancel a run.
`pauseRun(runId, request?)`	`POST /runs/{run_id}/pause`	Pause a run.
`resumeRun(runId, request?)`	`POST /runs/{run_id}/resume`	Resume a run.
`approve(runId, approval)`	`POST /computer-use/runs/{run_id}/approve`	Submit an approval.
`createSession(request?)`	`POST /computer-use/sessions`	Create a session.
`getSession(sessionId)`	`GET /computer-use/sessions/{session_id}`	Get session state.
`closeSession(sessionId, cleanup?)`	`DELETE /computer-use/sessions/{session_id}`	Close a session.
`listSessions()`	`GET /computer-use/sessions`	List sessions.
`subscribeToRun(runId, callback, options?)`	`GET /stream/{run_id}` (SSE)	Real-time event subscription.
`waitForRun(runId)`	SSE	Promise that resolves on terminal event.
`waitForApproval(runId)`	SSE	Promise that resolves on approval request.

Session management

const session = await client.createSession();

const result = await client.executeIntent('Task 1', {
  session_id: session.session_id,
});

const state = await client.getSession(session.session_id);
console.log(`Session has ${state.run_count} runs`);

await client.closeSession(session.session_id, true); // cleanup=true

Run control

await client.cancelRun(runId, {
  actor_id: 'user-123',
  comment: 'User requested cancellation',
});

await client.pauseRun(runId);
await client.resumeRun(runId);

const runs = await client.listRuns(sessionId, 'completed');

Event streaming

Subscribe to real-time events during execution:

import { EngineEvent } from '@allternit/computer-use';

const unsubscribe = client.subscribeToRun(
  result.run_id,
  (event: EngineEvent) => {
    console.log(`[${event.event_type}] ${event.message}`);

    if (event.event_type === 'action.completed') {
      console.log('Action completed:', event.data);
    }
  }
);

// Later
unsubscribe();

Common event types:

Event type	Meaning
`run.started`	Run has started.
`plan.created`	Engine produced a plan.
`action.started`	Action execution started.
`action.completed`	Action execution completed.
`approval.required`	Run is blocked waiting for approval.
`approval.received`	Approval decision received.
`run.completed`	Run finished successfully.
`run.failed`	Run failed.
`run.cancelled`	Run was cancelled.

Approval handling

Interactive approval

import { ApprovalHandler } from '@allternit/computer-use';

const handler = ApprovalHandler.createInteractive(client, async (request) => {
  return await confirm(`Approve: ${request.message}?`);
});

handler.watchRun(result.run_id);

Auto-approval with predicates

import { ApprovalHandler, ApprovalPredicates } from '@allternit/computer-use';

const handler = ApprovalHandler.createAutoApprove(
  client,
  ApprovalPredicates.readOnly()
);

handler.watchRun(result.run_id);

Built-in predicates

Predicate	Behavior
`always()`	Approve every request. Use with caution.
`never()`	Deny every request.
`matches(pattern)`	Approve if action summary matches a string or regex.
`safeActions(list)`	Approve if summary contains a whitelisted keyword.
`notDangerous(list)`	Deny if summary contains a blacklisted keyword.
`readOnly()`	Approve only actions that appear read-only.
`allowedAdapters(list)`	Approve only for specific adapter IDs.
`all(...predicates)`	AND combination.
`any(...predicates)`	OR combination.
`not(predicate)`	Negate a predicate.

Request and response example

Execute request

{
  "mode": "intent",
  "target_scope": "browser",
  "task": "Fill out the contact form with name John Doe",
  "options": {
    "timeout_ms": 30000,
    "max_steps": 50,
    "approvals": "on-risk"
  }
}

Execute response

{
  "run_id": "run_01hqx8x7x7x7x7x7x7x7x7x",
  "session_id": "sess_01hqx8x7x7x7x7x7x7x7x7x",
  "status": "completed",
  "mode": "intent",
  "target_scope": "browser",
  "selected_route": {
    "target_scope": "browser",
    "starting_layer": "semantic",
    "final_layer": "deterministic"
  },
  "summary": "Filled out the contact form.",
  "artifacts": [],
  "receipts": [],
  "counters": {
    "steps": 4,
    "actions": 3,
    "fallbacks": 0,
    "approvals": 0
  }
}

Error handling

import { AllternitComputerUseError } from '@allternit/computer-use';

try {
  await client.executeIntent('Task');
} catch (error) {
  if (error instanceof AllternitComputerUseError) {
    console.error(error.status, error.code, error.message);
  }
}

Error source	Description
`AllternitComputerUseError`	Structured error from the engine HTTP API.
Missing `eventsource`	Thrown when subscribing to SSE in Node.js without the peer dependency installed.

Types

All engine types are exported for TypeScript users:

import {
  EngineMode,
  TargetScope,
  EngineLayer,
  ExecutionStatus,
  EngineExecutionRequest,
  EngineExecutionResult,
  EngineEvent,
  EngineAction,
  EngineArtifact,
} from '@allternit/computer-use';

ACI overview — Allternit Computer Interface concepts
Tools / Computer Use — engine-level computer-use tools
MCP — attaching MCP servers to agents
Browser automation cookbook — end-to-end examples

​Computer Use SDK

​Overview

​Installation

​Quick start

​Client configuration

​Execution modes

​Intent mode

​Direct mode

​Assist mode

​Core methods

​Session management

​Run control

​Event streaming

​Approval handling

​Interactive approval

​Auto-approval with predicates

​Built-in predicates

​Request and response example

​Execute request

​Execute response

​Error handling

​Types

​Related pages