Gemini Robotics ER Playground

A browser-based playground for testing Google Gemini Robotics Embodied Reasoning — the AI model purpose-built for robotic manipulation. Run a full Sense-Plan-Act loop where Gemini ER analyzes a 3D scene and directs a 7-DOF Franka Panda arm to pick up objects, all in real-time physics simulation.

Instead of toy demos or static API calls, this puts Gemini's robotics model through an actual perception-to-manipulation pipeline — entirely in the browser, no robot hardware needed.

Try it online · Mirror · Bring your own Gemini API key

User Prompt ──→ Canvas Snapshot ──→ Gemini API ──→ 2D→3D Projection ──→ IK Solver ──→ Pick & Place
 "red cubes"      Base64 PNG        ER 1.5          Raycasting          7-DOF        State Machine

1. User types a target (e.g. "red cubes") and selects a detection mode
2. The app captures a top-down screenshot of the 3D scene
3. Gemini Robotics ER 1.5 analyzes the image and returns 2D object coordinates
4. Coordinates are projected from 2D screen space into 3D world space via raycasting
5. An analytical IK solver computes joint angles for the Franka Panda arm
6. A 15-step state machine executes the pick-and-place sequence

This project is designed as a practical testbed for Gemini's robotics capabilities:

git clone https://github.com/robotlearning123/gemini-robotics-er-playground.git
cd gemini-robotics-er-playground
npm install
cp .env.example .env.local   # add your Gemini API key
npm run dev

Or just use the online demo — paste your API key in the sidebar and start testing.

Interactive guides — click the previews to open:

Architecture Diagram	Study Guide
System architecture, data flow, IK solver	9-chapter walkthrough with code & quizzes

├── App.tsx                    # Main controller — orchestrates the AI loop
├── MujocoSim.ts               # Physics simulation engine (60fps loop)
├── RenderSystem.ts            # Three.js scene + 2D→3D projection
├── SequenceAnimator.ts        # 15-step pick-and-place state machine
├── FrankaAnalyticalIK.ts      # Analytical IK solver (He & Liu paper)
├── IkSystem.ts                # IK target management + redundancy resolution
├── RobotLoader.ts             # Downloads robot model from DeepMind Menagerie
├── types.ts                   # Shared TypeScript interfaces
├── components/
│   ├── UnifiedSidebar.tsx     # Control panel (prompt, detection mode, history)
│   ├── Toolbar.tsx            # Play/Pause, Reset, Dark Mode
│   └── RobotSelector.tsx      # Robot status overlay
├── rendering/
│   └── GeomBuilder.ts         # MuJoCo geometry → Three.js mesh factory
├── api/detect.ts              # Vercel serverless API proxy
├── functions/api/detect.ts    # Cloudflare Pages Function proxy
├── diagram.html               # Interactive architecture diagram
├── study.html                 # 9-chapter interactive study guide
└── setup.sh                   # One-command setup script

• API keys are never committed. .env* files are in .gitignore. Use .env.example as a template.
• Server proxy mode keeps keys server-side with IP-based rate limiting (30 req/IP/hour).
• Client-side mode stores the user's own key in localStorage only — never sent to our servers.

• MuJoCo physics engine (DeepMind)
• Franka Emika Panda model from MuJoCo Menagerie
• IK solver based on He & Liu (2020)

Apache-2.0