Jarvis

Ask how a project is doing. It gathers the truth from GitHub, your host and your VPS (open PRs, CI result, recent commits, deploy state, server health) in one turn, in parallel, then reads it back and shows the detail as cards in a HUD floating over your desktop. It never answers from memory. If it can’t reach something, it says so and names the fix.

It can also drive the machine: launch apps and Steam games, press real media keys, open a Discord channel, open a repo in VS Code, read your inbox, deploy, roll back, run a command on a server, or take the mouse and keyboard and work by looking at the screen.

The part that matters

Anything that writes, deploys, restarts or executes is confirmed by you first, every single time, with the exact command shown on screen. Not the transcript. Not a paraphrase. The literal command line that is about to run.

There is no approve all, no don’t ask again, and no bypass flag. A test asserts that none exists.

Every tool is classified in one registry file, and that file is the only authority; the JSON schemas the model sees carry no permissions at all.

An unknown tool name is classified as an action, never as a read. A tool advertised to the model but not registered fails the process at import rather than mid-conversation. An action tool with no plan step, and so nothing concrete to show you, fails the process too.

The execute half of an action tool receives the approved plan, not the model’s arguments. It is not structurally possible for what runs to differ from what you saw.

Empty input, a timeout, Escape, a closed browser tab, or an exception inside the gate itself: every one of them resolves to cancel. The base gate class wraps ask() in a try/except that turns any exception into a denial, because a broken UI must not mean “approved”.

Why the whitelist is the approval

There is no “run this program” tool. The model picks a label from your config; anything not listed cannot be launched at all. That is what makes app and game launches safe without a per-use prompt: they accept no path and no command string, and every one of them is undone by closing a window.

The same bargain covers the VPS. vps_read can only run commands already listed under that project’s read-only set. The model picks a label, never a command string. vps_tail refuses paths containing shell metacharacters instead of letting a “log path” smuggle in a second command. vps_run is gated and shows the full line. Unknown SSH host keys are refused rather than auto-accepted; silently trusting a new key would defeat the point of gating remote commands.

Websites are the one deliberate exception: web_open takes a raw URL, because the whitelist was never carrying weight there. A URL opens a tab and sends no clicks, and screen control can navigate anywhere the moment a browser is open. Refusing to type an address never bounded what Jarvis could reach, only how annoying it was to get there.

Screen control, and the tripwire

For everything with no API behind it, Jarvis can take the real mouse and keyboard and work by looking. A per-click prompt would make that useless; one prompt for forty clicks would make it a bypass. So it confirms twice: once for the task, with the resolved goal, screen size, step budget and any staged clipboard content shown in full, and again for every irreversible step it reaches while running.

That second gate does not rely on the model volunteering. A tripwire watches the actual action stream and fires on typing-then-Enter (how a message gets sent), destructive hotkeys, card-shaped runs of digits, and any input at all while a sensitive window title is in front. A screenshot between typing and Enter does not clear the trail; that would be a trivial bypass. The model is also told to ask first, and generally does, but instructions are a request and the tripwire is a check.

Three ways to stop it, none needing the agent’s cooperation: flick the mouse into any screen corner (pyautogui’s failsafe, never disabled, because your hand outranks the agent), the Stop button in the HUD, or the step and time budget.

Voice, locally

The wake word and transcription run entirely on your machine and cost nothing; audio never leaves the PC, so a false trigger is only expensive if its transcript gets sent, and Whisper’s silence hallucinations are filtered before the agent is ever called. Pausing genuinely closes the audio stream rather than quietly monitoring, so the microphone is released to whatever else wants it.

Voice approval adds a way to answer a confirmation; it never removes one. The modal still opens and still shows the resolved command. Only a whole utterance approves: “yes” does, “yes, but not the production one” does not, and never a substring match. A yes must be heard, not guessed: every answer is scored on the model’s own confidence, because Whisper hallucinates the word “yes” out of silence, and that is the check that stops the television approving a deploy. The no-list is deliberately wider than the yes-list.

Interfaces and honesty

A native click-through overlay HUD on Windows, a system tray app, a browser dashboard with no CDNs and no build step, and a CLI whose --dry-run resolves a plan and runs the gate without calling the API or executing anything.

Seven test suites, no API key, no network, no pytest. test_confirm proves anything that is not an explicit yes is a denial. test_server boots the real app over a real WebSocket with a scripted model and proves that approving executes, and that cancelling, disconnecting, or sending a stale request id executes nothing.

The README ends with a list of known limitations rather than a feature grid, because a tool that hides its edges is harder to trust.