interact — give your agent eyes and hands

The tool surface

Everything an agent needs to drive a UI

Real MCP tools, named so the model knows exactly what it's reaching for.

See

Grab pixels or read the DOM. Facts come back as text; how it looks comes back as a shot — vision only when you ask.

screenshotget_page_state

Act

The workhorse — click, type, scroll, drag, key-press, batched in one call, each step reporting what changed.

run_actions

Ground by vision

Click what's visible — "the Submit button" — instead of brittle CSS selectors that break on every redesign.

component model

Judge UIs

Not just see — critique. Find defects, PASS/FAIL your requirements, and measure exact WCAG contrast with no VLM spend.

review_uiverify_uimeasure_ui

Hear & watch

Understand sound and motion, not only stills — transcribe any audio or video, or ask what was said and what happened over time.

transcriberecord

Desktop + sandbox

Drive real OS windows — or launch an app into an isolated display the agent owns: non-intrusive, occlusion-proof, GPU-app safe.

launch_apptarget="nested"

Any MCP client

One interact install registers the server with Claude Code, Cursor, VS Code, Codex, Windsurf, Zed, Claude Desktop.

interact install

Text diffs, not screenshots

Every call returns what changed as compact text, so context stays tiny and cheap. Pixels are opt-in via query.

fast · cheap · legible

The code block is the demo

A call goes in. A text diff comes back.

No screenshot round-trips to burn your context — just what happened, in words.

agent ▸ interact (MCP)

navigate("https://news.ycombinator.com") Hacker News — 30 stories loaded. Top: "Show HN: a from-scratch autograd engine". 30 refs ready. run_actions([{click: "the top story link"}, {observe: "page title"}]) ✓ clicked ref#7 · navigated · title now "from-scratch autograd · GitHub" verify_ui(["header has a search box", "nav shows 4 tabs"]) PASS — search box present (top-right) FAIL — nav shows 3 tabs, not 4 record(query="did the menu slide in smoothly?") Yes — 220 ms ease-out, no dropped frames. transcribe("clip.mp4", query="how many speakers?") Two speakers; a short intro jingle at 0:03.

Three commands

Up and running in a minute

Installs the global interact CLI, config TUI, and MCP server in one.

Install

curl -LsSf https://raw\
.githubusercontent.com/AlanBlanchet\
/interact/main/install.sh | sh

Or any platform: uv tool install git+https://github.com/AlanBlanchet/interact

Connect your agent

interact install claude
# cursor · vscode · codex
# windsurf · zed · claude-desktop

Registers the MCP server with the client. interact doctor checks keys & providers.

Configure

interact          # config TUI
interact status   # what's bound
interact usage    # spend & tokens

Set models & keys in a terminal UI. Or leave models on auto — it picks a capable, cheaper-first model per job.

Capability	Linux	macOS	Windows
Browser · MCP server · CLI · TUI	✓	✓	✓
Desktop control (real windows)	✓	soon	soon
Isolated sandbox (Xephyr/Xvfb)	✓	—	—

Give your agent
eyes and hands.