Octopus — universal agentic hardware control

How it works

From “plugged in” to “agent-callable” — five steps.

Once you run the install command, the agent doesn't do anything magical. It does five things in order, each one writing a file the next step reads. If a step fails, the agent retries — and on retry it's allowed to rewrite the spec that told it what to do. Most software can't edit its own instructions. Ours can.

General pipeline

01 / PROBE

What's plugged in?

The agent scans the machine for everything connected — USB, Bluetooth, the local network — and writes down what it finds.

02 / IDENTIFY

What is each device?

For every device, the agent looks up what kind of thing it actually is, what it can do, and what library to talk to it with. It's allowed to web-search.

03 / INTERFACE

What functions should an agent see?

The agent designs the function signatures an AI client will need — one per thing the device can do. These are the “tools” in MCP.

04 / SERVE

Write the wrapper.

The agent writes the Python that actually talks to the hardware and exposes those functions over MCP.

05 / DEPLOY

Make it live.

Installs the Python dependencies, starts the server, and writes the snippets you'd paste into Claude Desktop or any MCP client to connect.

Two extra framework steps for our demo rig

These only run if the camera and the SO-ARM101 robot arm are plugged in. They give the agent extra context about hardware we demo on a lot, but they're optional — a fresh install on a machine without these never runs them.

· / PERCEIVE

Camera as the agent's eyes

Picks which camera to use and keeps a short visual memory — current frame, previous frame, a one-line summary of what changed.

· / ARM

Verify the arm by moving it

Actually moves the arm to verify the generated tool code is correct. If something fails, the agent rewrites the affected piece.

Features

The coding agent is the software.

What's in our repo is five markdown files and a runner. Everything else is generated. That's why one install command does so much, and why the same command does different things on different machines.

Model-agnostic

Pick any model that OpenRouter routes to — Gemini 3 Flash, Sonnet 4.6, Kimi K2, Opus 4.7, and ~6 more. One line in octopus.toml. Set OPENROUTER_API_KEY once, swap whenever.

Hardware-agnostic

If your OS can see the device, the agent figures out the rest. No driver registry, no per-device wiring code in our repo.

Prompts as protocol

The five spec files describe what to do, not how. Same files on Mac, Linux, and Windows (WSL) produce different code that does the same job.

↻

Self-healing daemon

A background daemon tails the server's logs and asks the agent to patch anything that breaks.

◉

Self-perception

When a camera is plugged in, the agent uses it as a second mirror — visually confirming the hardware actually did what it told it to. Self-watching at the hardware layer, paired with the daemon's self-watching at the software layer.

↤

MCP-compatible

Claude Desktop, ChatGPT desktop, OpenClaw, Claude Code, your local Ollama, our own bridge — anything that speaks MCP connects to the same URL.

Demo

From command line to controlling the arm.

Run one install command on a Pi with hardware plugged in. ~20 minutes later you have an MCP server tailored to whatever you plugged in, with no per-device code in our repo. The clip below runs through that whole flow on our reference rig — install, Claude Desktop picks up the server, Claude calls a tool, arm moves. The buttons in the next section drive this same rig live, right now.

~40 s walkthrough — Pi install completes, Claude Desktop connects to the public tunnel, arm physically responds.

How the system builds itself.

The five spec stages run in order on any platform. If the camera or the arm are detected, two extra framework steps run after — they give the agent extra context about hardware we demo on a lot. A background daemon then keeps the server healthy by tailing its logs and asking the agent to patch anything that breaks. A single octopus expose command publishes the whole thing over a Cloudflare quick-tunnel, so any MCP client on the public internet can reach it.

The full rig: SO-ARM101, motorized camera on curved rail, Raspberry Pi — **The rig.** 6-DoF SO-ARM101 robot arm (Feetech STS3215 servos), USB webcam on a motorized vertical post on a curved rail, Raspberry Pi 4. The camera can rotate around the arm for multiple viewing angles.

Closer view of the arm gripper with camera facing it — **Closed-loop eye.** The camera looks at the gripper. Every arm command triggers a frame capture, and the agent compares the new frame against the one before it to confirm the hardware actually did what it was told.

Arm in a different pose showing reach — **Reach.** Same arm, different pose. All 6 joints exposed as MCP tools (`arm_set_joint_angle`, `arm_home`, `arm_set_all_joints`, plus diagnostics). Zero hand-written hardware code in our repo.

Try it yourself

Control our hardware from anywhere.

You don't need a Pi or an arm. The buttons below talk to a real SO-ARM101 in our apartment over a public Cloudflare tunnel — every click is a real MCP tool call, every camera frame is a real capture. Three ways to connect, depending on what you're building.

01 / CURL / SHELL — the easiest path

REST bridge via Join39 / Railway

Just an HTTPS POST. Works from any language, any notebook, any shell. No MCP client needed. The buttons below hit the Pi directly.

(click any button — the response shows up below. Every arm command also captures a fresh camera frame, so you can see the arm actually moved.)

curl equivalents

# list available tools
curl -s -X POST https://mellow-miracle-production-b572.up.railway.app/tools/invoke \
  -H 'Content-Type: application/json' \
  -d '{"action":"list"}'

# move arm to rest pose
curl -s -X POST https://mellow-miracle-production-b572.up.railway.app/tools/invoke \
  -H 'Content-Type: application/json' \
  -d '{"action":"invoke","tool_name":"arm_home","parameters":{}}'

# grab a fresh camera frame
curl -s https://mellow-miracle-production-b572.up.railway.app/camera/latest.png \
  --output frame.png && open frame.png

02 / MCP CLIENT — the agent-native path

Claude Desktop, Inspector, any MCP client

Connect any client that speaks MCP — Claude Desktop, the MCP Inspector, your local Ollama setup, OpenClaw, Claude Code. Your agent gets the typed tool list and calls them directly.

Live tunnel URL (auto-updates from Pi): loading…

(click a button to copy a ready-to-paste config)

what you're pasting

// Claude Desktop → Settings → Connectors → Add custom
{ "name": "octopus-pi", "url": "LIVE_URL/mcp" }

// OpenClaw / any streamable-http MCP config
{ "transport": "streamable-http", "url": "LIVE_URL/mcp" }

// Local MCP Inspector (requires Node)
npx @modelcontextprotocol/inspector LIVE_URL/mcp

03 / JOIN39 — the embedded-in-an-agent path

Agents on join39.org

Join39 is a marketplace where AI agents discover apps. If you're building an agent there, add octopus-hardware to its app list and your agent can call the arm during conversation. Join39 forwards the call through our bridge to the Pi.

App: octopus-hardware

Note: the Cloudflare tunnel URL rotates whenever the Pi restarts. The Railway bridge URL (used by the buttons in lane 01) is stable — that's the safer link if you're sharing with someone else.

Philosophy

Four principles that everything else reduces to.

We landed on eight in our internal philosophy doc. These four are the ones that drove every architectural decision in the repo.

01 / Prompts as protocol & specifications

I = A(S, P)

The five markdown files in protocol/ aren't configuration — they're the program. An Agent reads the Specs and acts on the Platform in front of it. What comes out — the running, agent-callable hardware — is what we call Infrastructure.

I = A(S, P)

So the real infrastructure isn't shipped — it emerges from the agent's interaction with the specs and the machine, constrained by the scaffolding we provide (the orchestrator, RALPH, the principles below). Our repo is small. What it produces on your hardware is the actual product.

hover to expand

02 / RALPH (Read · Agent · Loop · Pass)

One control structure for the whole system

Every pipeline stage, every daemon tick, every self-heal action is the same four-step loop: read the current state, hand it to the agent, let the agent act, check whether it passed. No other control structures anywhere in the system. Adapted from Geoffrey Huntley's RALPH loop — his version is a coding agent re-prompted with the same goal until it converges; ours is the same spirit but split into four named steps that we re-use across the pipeline, the daemon, and heal.

hover to expand

03 / Self-reference, two layers

The system observes itself in two distinct ways

Software face: failed pipeline stages can edit the spec that governs them on retry; the daemon tails its own server's logs and patches what breaks.
Hardware face: the camera watches the arm the agent just moved — it grounds the system in the physical world by giving the agent a way to see the consequences of its own actions.

hover to expand

04 / Model calls in place of byzantine code

When the logic is fuzzy, replace the code with a model call

Anywhere we'd normally write tangled, edge-case-heavy logic — parsing a half-structured device descriptor, mapping a vendor ID through a registry of one-off names, deciding whether a tool's output looks like an error, classifying which kind of failure a log line represents — we ask the model instead. A short prompt with the input is more honest about the fuzziness than a 200-line if/else tree, and it keeps working when an edge case we never thought of shows up.

The narrower instance ("no per-device switch statements") is the most visible one. The general principle: when the logic you're about to write is dirty enough that your enumerated cases will leak, replace the code with a model call.

hover to expand