Skip to content
Fuse
Esc
↑↓navigate↵open⌘Jpreview

Drive the environment's desktop

Relay one computer-use action to the environment’s guest agent: move the mouse, click, type, scroll, screenshot, or zoom into a region. The response carries a base64 PNG screenshot on every action that implies one, which is what a computer-use agent loop feeds back to the model as the tool_result.

The action vocabulary is the guest agent’s (it matches Anthropic’s computer_20251124 tool): screenshot, left_click, right_click, middle_click, double_click, triple_click, left_click_drag, left_mouse_down, left_mouse_up, mouse_move, type, key, hold_key, scroll, wait, zoom, cursor_position. Fields this server has not learned yet pass through to the guest untouched, so a newer image’s actions work through an older control plane.

Requires an environment booted from a desktop image; on any other image the guest answers with 503 and a reason. Unlike exec this accepts API keys: an action here is a mouse click on a display the image chose to run, not a root shell.

While a human holds the environment (a takeover), or an ask stall is waiting to be acked, an action is refused with 423 locked before it reaches the guest, exactly as exec is. Wait on details.intervention_id, then send the action again.

Hitl gates. When the environment’s hitl.gates include computer gates, the action is checked against them before the guest sees it, by its action name and, for type, its text (text_contains). A gate in ask mode answers 202 with a HeldResponse; once a human approves, the orchestrator performs the action itself, with the body as sent (or the approver’s edited_text for a type), and stores the guest’s answer in the intervention’s result.computer. notify lets it through and records a terminal intervention; deny answers 403 forbidden with details.reason hitl_denied. A gate with a decide webhook asks it first: allow lets the action through, deny is the same 403 with the decider’s reason, and ask or any failure holds it for a human. A grant from an earlier approval lets the exact same action body through without a prompt.

POST/v1/environments/{vmId}/computer
Authorization
AuthorizationBearer token · headerrequired
`Authorization: Bearer <token>` — either the static master token (`ORCH_AUTH_TOKEN`) or a revocable API key issued via /v1/api-keys (API keys require Postgres; without a database only the master token is accepted). Master-only surfaces (exec, attach, API key management) refuse API keys with 403. An expired key is refused with 401 like an unknown one. A key minted with `scopes` reaches only the hitl routes (`/v1/interventions`, `/v1/hitl/*`, and POST /v1/api-keys to mint a narrower key). Any other route is 403 code `forbidden`, reason `env_scope`, except an `/v1/environments/{vmId}` route for an environment the key does not cover, which is 404 so the key cannot probe which environments exist. **Authentication can be disabled entirely.** When neither `ORCH_AUTH_TOKEN` nor an API key store (`DATABASE_URL`) is configured — the default local-development posture — the bearer middleware becomes a pass-through: every operation this document marks as secured is reachable with no credentials, requests carry no principal, and master-only surfaces therefore admit everyone. Never run that configuration on a reachable network; set `ORCH_REQUIRE_AUTH=true` to make the orchestrator refuse to start without a master token.
or
fuse_sessionAPI key · cookierequired
HttpOnly session cookie set by POST /login for browser callers. Consulted only when no Authorization header is present.
Path parameters
vmIdstringrequired
VM identifier assigned by the orchestrator.
Request body
requiredapplication/json
actionstringrequired
The action to perform, e.g. `screenshot`, `left_click`, `type`, `key`, `scroll`, `zoom`.
coordinateinteger[]
[x, y] target for pointer actions.
min items 2 · max items 2
start_coordinateinteger[]
[x, y] origin for left_click_drag.
min items 2 · max items 2
textstring
The payload for `type`, the keysym combo for `key` and `hold_key` (e.g. `ctrl+shift+t`), or modifiers held during a click action.
regioninteger[]
[x1, y1, x2, y2] crop for `zoom`.
min items 4 · max items 4
durationnumber
Seconds for `hold_key` and `wait`, capped guest-side at 100.
scroll_directionstring
Allowed:updownleftright
scroll_amountinteger
Wheel clicks for `scroll`, 1 to 100.
Responses
200The action's result, usually carrying a screenshot.
outputstring
Textual output, e.g. the pointer location for `cursor_position`.
screenshotstring
Base64 PNG of the display after the action settled. Present on every action that implies one; for `zoom` it is the cropped region at full resolution.
202A `computer` hitl gate holds the action. Nothing reached the guest yet. The body is a HeldResponse naming the pending intervention, and `Location` points at it. Wait on it with GET /v1/interventions/{interventionId}?wait=; once approved the orchestrator performs the action and stores the guest's answer in the intervention's `result.computer`.
intervention_idstringrequired
interventionInterventionrequired
Show properties
idstringrequired
env_idstringrequired
kindstringrequired
Allowed:asksecrettakeoverreviewexeclifecyclecomputerstallexpiry
gatestring
The gate that raised it. Empty for a request.
modestring
The gate's mode. Notify and deny records are born terminal.
Allowed:asknotifydeny
statestringrequired
Allowed:pendingapprovedrejectedexpiredcancelled
payloadobjectrequired
What was asked or attempted: `question` for an ask, `secret`, `takeover` or `review` for those requests, `exec`, `lifecycle` or `computer` for a gate, `signal` for a stall or expiry. `decider` is set when a gate's decider sent the action on to a human.
Show properties
questionstring
choicesstring[]
execExecPayload
A held exec.
Show properties
argvstring[]required
shellstring
The script when the caller sent `shell`; argv is then the sh -lc that runs it.
timeout_msinteger<int64>
lifecycleLifecyclePayload
A held environment action.
Show properties
actionstringrequired
Allowed:destroyforkmigratedrainsnapshotrotate-token
secretInterventionSecret
A secret request. It never holds the value.
Show properties
targetstringrequired
The declared target's `env` name or `file` path.
labelstringrequired
The target's label from the policy. Show it first: the agent writes `message`, so `message` alone could claim to be anything.
messagestring
The agent's own text.
takeoverInterventionTakeover
A takeover. `holder` and `started_at` are set while a human holds the environment, which is also when the agent's exec and computer calls get 423. The intervention stays `pending` until release.
Show properties
viastringrequired
Allowed:terminaldesktop
reasonstring
holderActor
Show properties
idstringrequired
A pinned or asserted actor id, `key:<key_id>`, `master`, `system` for fuse itself, or `decider:<gate>`.
namestring
sourcestringrequired
`pinned`: fixed on the key at mint time. `asserted`: named in `Fuse-Actor` by a `hitl:act-as` key or the master token, and only as true as that caller's own auth. `key`: derived from the credential. `system`: expiry and cancellation. `decider`: a gate's decider allowed or denied the action; the id is `decider:<gate>`.
Allowed:pinnedassertedkeysystemdecider
key_idstring
The key that authenticated the request, including the key that asserted an actor. Omitted for the master token.
started_atstring<date-time>
signalInterventionSignal
What fuse saw when it raised a stall or expiry. A stall carries `failures` and `last_exit_code`, or `quiet_seconds`. An expiry carries `reaper` and `reap_at`, and once approved `extended_until`.
Show properties
failuresinteger
The run of consecutive non-zero exec exits.
last_exit_codeinteger
The exit code that completed the run.
quiet_secondsinteger<int64>
How long the agent had made no exec or computer call.
reaperstring
What will tear the environment down.
Allowed:idle_timeoutmax_runtime
reap_atstring<date-time>
When. The expiry's own deadline is capped at this, with a 10s floor.
extended_untilstring<date-time>
Set on approval. The reapers leave the environment alone until then.
computerComputerPayload
A held computer action. On approval the orchestrator replays `input` to the guest, with the approver's `edited_text` in place of `text` when there is one.
Show properties
actionstringrequired
The computer action name.
textstring
The text of a `type` (or the keys of a `key`), as sent.
inputobjectrequired
The whole action body as the caller sent it (a ComputerActionRequest).
reviewInterventionReview
A review request. When `path` is set, `content` is that file as fuse read it at request time, so the human reviews what the agent submitted even if the file changed afterwards.
Show properties
pathstring
contentstringrequired
formatstringrequired
Allowed:textdiff
messagestring
The agent's note to the reviewer.
truncatedboolean
Set when the file was longer than 1 MiB and was cut off.
deciderInterventionDecider
What a gate's decider said about an action it sent on to a human: `verdict` `ask` with its `reason`, or the `error` that made fuse fall through to a human.
Show properties
verdictstringrequired
Allowed:ask
reasonstring
errorstring
Set when the decider call failed.
triggered_byActorrequired
Show properties
idstringrequired
A pinned or asserted actor id, `key:<key_id>`, `master`, `system` for fuse itself, or `decider:<gate>`.
namestring
sourcestringrequired
`pinned`: fixed on the key at mint time. `asserted`: named in `Fuse-Actor` by a `hitl:act-as` key or the master token, and only as true as that caller's own auth. `key`: derived from the credential. `system`: expiry and cancellation. `decider`: a gate's decider allowed or denied the action; the id is `decider:<gate>`.
Allowed:pinnedassertedkeysystemdecider
key_idstring
The key that authenticated the request, including the key that asserted an actor. Omitted for the master token.
deadlinestring<date-time>required
on_timeoutstringrequired
Allowed:rejectallow
decisionobject
Present once terminal. An expired intervention carries the on_timeout outcome and a cancelled one a reject, both decided by "system".
Show properties
outcomestringrequired
Allowed:approvereject
answerstring
reasonstring
decided_byActorrequired
Show properties
idstringrequired
A pinned or asserted actor id, `key:<key_id>`, `master`, `system` for fuse itself, or `decider:<gate>`.
namestring
sourcestringrequired
`pinned`: fixed on the key at mint time. `asserted`: named in `Fuse-Actor` by a `hitl:act-as` key or the master token, and only as true as that caller's own auth. `key`: derived from the credential. `system`: expiry and cancellation. `decider`: a gate's decider allowed or denied the action; the id is `decider:<gate>`.
Allowed:pinnedassertedkeysystemdecider
key_idstring
The key that authenticated the request, including the key that asserted an actor. Omitted for the master token.
decided_atstring<date-time>required
edited_argvstring[]
The argv the approver replaced a held exec's with.
edited_textstring
The text the approver replaced a held `type` action's with.
grantstring
How far an approval of a gated action reaches.
resultInterventionResult
What performing an approved gated action produced.
Show properties
execExecEnvironmentResponse
The outcome of a guest command. A non-zero `exit_code` arrives with **HTTP 200**: the command ran and failed, which is a successful answer to the question the caller asked. It is not an error and must not be raised by an SDK. HTTP errors mean the command could not be run at all. `stdout` and `stderr` are kept apart, and are plain strings rather than base64 — exec output is overwhelmingly text, and encoding it would make every caller pay a decoding tax for the rare binary case.
Show properties
exit_codeintegerrequired
The guest command's own exit status. Non-zero is a normal result, delivered with HTTP 200.
stdoutstringrequired
stderrstringrequired
computerComputerActionResponse
The guest's answer to one action.
Show properties
outputstring
Textual output, e.g. the pointer location for `cursor_position`.
screenshotstring
Base64 PNG of the display after the action settled. Present on every action that implies one; for `zoom` it is the cropped region at full resolution.
env_idstring
The environment a fork or migrate produced.
snapshot_idstring
The snapshot a snapshot action produced.
errorstring
Set when the approved action could not be performed. A non-zero exec exit code is a result in `exec`, not an error.
finished_atstring<date-time>required
created_atstring<date-time>required
updated_atstring<date-time>required
400Malformed request.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
403A `computer` hitl gate in `deny` mode, or a gate's decider, refused the action (code `forbidden`, `details.reason` `hitl_denied`, `details.gate` the gate's name, `details.intervention_id` the terminal record; the message is the gate's or the decider's reason).
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
404Resource does not exist. Note the distinct code `route_not_found`, returned when the URL matches no registered route at all (wrong host, port, or path prefix) rather than a known route with a missing resource.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
409The environment is not running.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
423A human holds the environment, so the agent's call was refused before it reached the guest (code `locked`). Either a takeover is held (`details.kind` `takeover`), or a stall in `ask` mode is waiting for a human to ack it (`details.kind` `stall`). `details.intervention_id` names the intervention. Long-poll it with GET /v1/interventions/{interventionId}?wait= until it is no longer `pending`, then send the call again; a new 423 means wait again. The lock never outlives the intervention's deadline, so this always ends. The SDKs wait and retry by default, and raise a `LockedError` carrying the intervention id and kind when told not to wait.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
500Unexpected server error.
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
501The operation is not supported by the backing provider (e.g. an exec or attach against an environment with no real guest).
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
503The environment has no display (not a desktop image, or the display is still starting).
errorobjectrequired
Show properties
codestringrequired
Stable machine-readable code. `route_not_found` (404) means the URL matches no route this server exposes — usually a wrong host, port, or path prefix — as opposed to `not_found`, where the route exists but the resource does not. `forbidden` is emitted by CIDR-allowlist rejections and by hitl access checks, which set `details.reason` (see the Forbidden response); master-only refusals (exec, attach, listing and revoking API keys) return 403 with code `unauthorized`.
Allowed:not_foundroute_not_foundconflictinvalid_argumentunauthorizedforbiddenunavailableinternalunimplementedlocked
messagestringrequired
Human-readable description.
detailsobject
Optional stable, non-sensitive metadata (e.g. ids, counts). Omitted when empty. A 403 `forbidden` from a hitl access check carries `reason` (`missing_scope`, `env_scope`, `self_approval`, `actor_required`, `actor_mismatch`, `actor_not_allowed`, `mint_escalation`, `hitl_denied`), plus `scope` for `missing_scope`, and `gate` and `intervention_id` for `hitl_denied`. A 423 `locked` carries `intervention_id` and `kind` (`takeover` or `stall`).
Request
curl -X POST "http://localhost:8080/v1/environments/string/computer" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "action": "left_click",
  "coordinate": [
    512,
    384
  ]
}'
Response
{
  "output": "string",
  "screenshot": "string"
}