RemoteCtrl

Delegate the task, not the browser.

RemoteCtrl is a task-scoped browser coworker: it lets a user delegate one approved task inside their own authenticated browser to an AI agent or remote operator, intervene when needed, and turn a successful session into a reusable workflow. It combines a visible browser workspace with an agent console, workflow editor, local browser profiles, visual guidance, and local Whisper speech-to-text.

This repository is a hackathon-ready development build. It is designed to make the main demo path easy to run locally while keeping credentials, browser profiles, and workflow data on the host machine.

Existing products solve pieces of the problem, but they generally optimize for a different unit of access: a whole machine, an agent-owned browser, a developer automation API, or a predefined workflow. RemoteCtrl is built around one approved browser task.

The user keeps the authenticated browser session. An agent performs the delegated work; the user can pause, approve an exception, complete MFA manually, or take over. Once the work succeeds, the interaction can be saved and replayed as a workflow.

This capability matrix is based on the source-linked market study in research/remotectrl-market and the architecture reference repositories in research/. A ✓ marks a documented, first-class focus; — means it is not the product’s core position in this comparison.

RemoteCtrl’s distinction is the combined row: task-scoped agent execution inside a user-controlled browser, with human intervention and workflow replay in the same product.

The policy layer can evaluate browser-specific execution context: the current browser session, the acting agent/operator, the open site, the requested capability, the action type, the declared task scope, and whether an approval is required. That is stronger than telling a model to “only do the approved task”; it gives the runtime a place to block or request approval before an agent action executes.

Current boundary: RemoteCtrl enforces scope for agent actions and provides human takeover. It does not claim that remote human mouse and keyboard input is scope-enforced.

RemoteCtrl is split across the Electron process boundary:

React renderer
  -> preload contextBridge (window.RemoteCtrlAPI)
  -> Electron IPC handlers
  -> main process services
       -> Playwright / Chrome CDP
       -> agent and workflow execution
       -> local JSON storage + OS credential encryption
       -> WebRTC signaling and extension bridge

• src/main/ owns Electron lifecycle, browser management, agent execution, workflow execution, IPC, storage, WebRTC, and the local Whisper runtime.
• src/preload/ exposes the narrow renderer-to-main API. Renderer code does not access Node APIs directly.
• src/renderer/ contains the React screens, panels, settings, browser surface, chat input, workflow editor, and Zustand stores.
• src/shared/ contains types, Zod schemas, provider defaults, and IPC contracts shared across processes.
• server/ contains the optional Socket.IO signaling server for remote sessions.
• extension/ contains the Manifest V3 Chrome extension and its localhost bridge client.
• demo-site/ contains the offline-friendly workflow target used for presentations.
• research/ contains architecture notes from browser-use and Stagehand and other reference material.

The main agent loop uses Vercel AI SDK tool calls, Playwright browser tools, compact DOM observations, selector heuristics, execution journals, checkpoints, and retry/resume paths. See docs/automation-agent.md and docs/architecture.md.

• Electron 42
• React 19 and TypeScript
• Vite and esbuild
• Playwright with Chromium/CDP
• Vercel AI SDK with provider adapters for OpenAI, Anthropic, Google Gemini, Google Vertex AI, Groq, DeepSeek, Nebius, and OpenRouter
• WebRTC and Socket.IO for remote sessions
• Zustand, Zod, Radix UI, Tailwind CSS, and Vitest
• Transformers.js with a pinned ONNX Whisper tiny English model for local speech-to-text

• Node.js with npm. Use a current LTS release; Node 20+ is recommended for the Electron and TypeScript toolchain.
• macOS, Windows, or Linux.
• A Chromium-compatible browser is installed automatically by Playwright during dependency installation. A system Chrome/Chromium/Edge installation is also supported for local browser mode.
• An API key for at least one configured AI provider if you want to run agent or recovery flows. Vertex AI can use Google Application Default Credentials instead of an API key.
• Microphone permission only if you want local Whisper speech-to-text.

macOS/Linux:

./scripts/setup.sh

Windows PowerShell:

Set-ExecutionPolicy -Scope Process Bypass
.\scripts\setup.ps1

The setup scripts verify Node/npm, install from package-lock.json with npm ci, and install Playwright's Chromium browser through the package postinstall hook. They do not overwrite an existing .env; this repository currently does not include a .env.example, so provider credentials should be entered in the app Settings screen.

npm run dev

This starts the Vite renderer and the Electron main process together. For focused development:

npm run dev:renderer   # renderer only at http://localhost:5173
npm run dev:main       # build main process and launch Electron

For a production-style local run after building:

npm run build
npm start

Open Settings inside RemoteCtrl and configure the provider/model you want to use. API keys are handled by the Electron main process and encrypted with safeStorage when the operating-system credential service is available; they are not exposed to the React renderer or committed to the repository.

Supported provider profiles and their default model families include:

Optional Vertex AI setup uses the environment variables recognized by the resolver (GOOGLE_VERTEX_PROJECT, GOOGLE_CLOUD_PROJECT, GCP_PROJECT, or GCLOUD_PROJECT) and GOOGLE_APPLICATION_CREDENTIALS, or local Google ADC via gcloud auth application-default login.

Other settings include browser mode/profile, headless mode, signaling URL, vision controls, theme, shortcut, and speech input mode. The default signaling URL is https://remotectrl-signaling.onrender.com.

Speech is local-only and optional. In Settings:

1. Start the local Whisper model download.
2. Wait for integrity verification to finish.
3. Enable microphone audio and grant the OS microphone permission.
4. Use push-to-talk or hands-free input according to the selected speech mode.

The app downloads a pinned onnx-community/whisper-tiny.en model into Electron user data, verifies its file sizes and SHA-256 digests, and runs inference locally through Transformers.js. Runtime configuration uses local model files only and disables remote model/cache access. Audio is not sent through the remote session or signaling server. The capture path currently limits a transcription segment to 60 seconds.

Remote host/controller sessions use the Socket.IO server. Run it locally when you want to test the full remote flow without the hosted endpoint:

npm run server

The server listens on port 3001 by default. Override it for a local session with:

PORT=3010 npm run server

On Windows PowerShell:

$env:PORT = '3010'; npm run server

Then set the signaling URL in Settings to http://localhost:3010 (or the reachable URL for the machine hosting the server). The signaling server uses short-lived PIN rooms, host approval, WebSocket transport, and basic failed-attempt rate limiting. It is a hackathon server and should not be exposed publicly without additional authentication, TLS, monitoring, and deployment hardening.

The extension connects to the desktop bridge at ws://127.0.0.1:45456.

1. Open chrome://extensions.
2. Enable Developer mode.
3. Choose Load unpacked.
4. Select the repository's extension/ directory.
5. Pin the extension and open it on a page.

The extension can delegate the active page and record interactions. Password values are masked before a recorded workflow is sent to the desktop bridge. See extension/README.md.

Run the full verification set before submitting a hackathon build:

npm test
npm run typecheck:all
npm run lint
npm run build
git diff --check

Useful focused commands:

npx vitest run src/main/automation src/main/ipc src/main/speech src/renderer/hooks
npm run build:renderer
npm run build:main

The app can be packaged for macOS, Windows, and Linux with:

npm run pack

Packaging requires the platform-specific Electron builder prerequisites and signing/notarization configuration for production distribution. The repository currently publishes no demo binary or test account.

• Settings, workflows, automation history, model metadata, and browser profiles are stored under Electron's app.getPath('userData').
• Typical locations are ~/Library/Application Support/RemoteCtrl/ on macOS, %APPDATA%\\RemoteCtrl\\ on Windows, and ~/.config/RemoteCtrl/ on Linux.
• API keys use Electron safeStorage (Keychain on macOS, DPAPI on Windows, and the platform secret service on Linux when available). The README does not claim encryption is guaranteed on every environment.
• Browser profiles may contain authenticated cookies and history. Treat the user-data directory as sensitive.
• Remote sessions stream browser frames and forward approved controller input. Use a trusted signaling endpoint and do not share session PINs publicly.
• Task policy can express capability, origin/domain/path, action-count, expiry, and approval requirements. Domain restrictions are optional and controlled by the task policy toggle; local sessions intentionally retain direct-control behavior.
• The demo workflow documents self-healing behavior, not comprehensive remote-input policy enforcement.
• The project is a hackathon build. Production deployment still needs signed installers, a hardened signaling deployment, stronger identity/access controls, observability, and a security review.

More detail: docs/storage-security.md, docs/webrtc-remote.md, and docs/workflows-replay.md.

src/main/automation/   agent loop, browser tools, workflow execution, retries
src/main/ipc/          validated renderer/main IPC handlers
src/main/speech/       local Whisper model manager and runtime
src/renderer/           React screens, stores, browser and workflow UI
src/shared/             types, schemas, defaults, IPC contracts
server/                 optional Socket.IO signaling server
extension/              Manifest V3 browser extension
demo-site/              offline workflow demo target
docs/                   architecture, security, workflow, and design notes
research/               browser-use and Stagehand architecture research
scripts/                macOS/Linux and Windows developer setup scripts

Codex was used as the engineering collaborator for product framing, architecture research, implementation, verification, and documentation. GPT-5.6/Codex work in this repository included:

• Product and UX decisions: turned the browser-control surface into a task-scoped delegation product, refined the policy model, reduced attention-heavy UI states, and documented the remote-control boundary honestly.
• Architecture research: studied browser-use and Stagehand agent loops, browser observation, retry, checkpoint, visual-grounding, and workflow patterns; the resulting notes live in research/.
• Subagent-driven development: used the subagent-driven-development skill to split bounded tasks across lower-capability Luna agents. Agents handled focused research, setup scripts, README groundwork, speech UI cleanup, and local Whisper implementation work; the main session coordinated scope, commits, and integration.
• Skill-guided execution: used the brainstorming skill to turn ambiguous product requests into clear design direction, and verification-before-completion to require fresh command output before claiming a change was complete.
• Implementation work: added and refined agent/workflow resilience, visual guidance, browser-control UX, custom desktop behavior, policy handling, and a local ONNX Whisper speech-to-text foundation.
• Testing and documentation: ran focused tests, type checks, builds, and diff checks where applicable; documented actual results and known limitations instead of asserting unsupported capabilities.

Codex was most useful where the work crossed layers: translating product intent into Electron main-process boundaries, renderer interactions, IPC contracts, testable persistence, and a clear hackathon story.

This documentation pass was informed by Codex session:

019f769b-288e-7da1-bdf6-9f11b22a5b7a

• Keep browser automation and credentials in the Electron main process; expose only typed, narrow IPC methods to the renderer.
• Prefer compact DOM observations and stable semantic selectors before visual fallback so agent actions remain inspectable and recoverable.
• Store workflows and history locally with atomic writes so an interrupted run is less likely to corrupt state.
• Keep remote transport separate from local automation and require an explicit host approval step before controller input is accepted.
• Make speech-to-text opt-in and local, with a pinned model manifest and integrity verification.
• Keep the hackathon demo deterministic through a static local site instead of relying on an external SaaS account.

• Add signed release artifacts and CI for macOS, Windows, and Linux.
• Harden signaling authentication, authorization, TLS, and abuse controls.
• Add richer automation observability and long-running soak tests.
• Improve model download resume UX and package the local inference path for all release targets.
• Add end-to-end tests for remote sessions, extension recording, and full workflow replay.

RemoteCtrl is licensed under the MIT License.