G1 + Quest 3 Teleoperation

Built on Unitree Robotics xr_teleoperate (Apache-2.0).

Status: Fully tested and working — Full-body teleoperation confirmed on G1 (29-DoF) via Quest 3, running entirely on the G1's Jetson Orin NX. Both Inspire RH56DFTP hands integrated via direct Modbus TCP.

Full-body teleoperation of a Unitree G1 (29-DoF) humanoid robot using a Meta Quest 3 VR headset. Everything runs directly on the G1's onboard Jetson Orin NX (PC2) — no external host PC needed.

Built on Unitree's xr_teleoperate v1.5 (Open-TeleVision framework, CoRL 2024). Uses browser-based WebXR — just open a URL on the Quest 3.

Repo Structure

This monorepo bundles all required components — no separate cloning needed:

robot-teleop/
├── xr_teleoperate/          # Teleop stack (IK, WebXR server, robot control)
│   ├── assets/              # URDF models, meshes for G1/H1/hands
│   └── teleop/              # Main scripts, televuer, utils
│       └── robot_control/
│           ├── robot_hand_inspire.py          # Original DDS-based Inspire driver
│           └── robot_hand_inspire_modbus.py   # Direct Modbus TCP driver (our addition)
├── teleimager/              # Camera streaming server
│   └── src/teleimager/      # image_server.py, image_client.py
├── unitree_sdk2_python/     # Unitree SDK (DDS comms, G1 modules)
│   └── unitree_sdk2py/      # Core SDK: g1/, comm/, idl/, etc.
└── README.md

Hardware

Component	Details
Robot	Unitree G1, 29-DoF arm config
VR Headset	Meta Quest 3
Robot onboard PC (PC2)	Jetson Orin NX 16GB, Ubuntu 20.04 ARM
Camera	Intel RealSense D430i (on G1, `/dev/video2`)
Dexterous hands	Inspire RH56DFTP × 2 (Ethernet Modbus TCP)
Unitree remote	Required for robot mode switching and emergency stop
WiFi router	Must be on `192.168.123.x` subnet (same as G1)

Quick Start

Once everything is set up, daily usage is three steps:

# SSH Session 1: Camera server
conda activate teleimager
cd ~/robot-teleop/teleimager
python -m teleimager.image_server

# SSH Session 2: Teleop server (with Inspire hands)
conda activate tv
cd ~/robot-teleop/xr_teleoperate/teleop
python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=hand --ee=inspire_ftp --display-mode=immersive

# Quest 3 browser:
# https://192.168.123.164:8012/?ws=wss://192.168.123.164:8012

# Terminal: r = start, s = record, q = quit
# Remote: L2+B = emergency stop

Setup

Part 1: Jetson Teleop Environment

All commands run via SSH into the G1.

1.1 Install Miniconda

The Jetson ships with Python 3.8 — too old. Miniconda provides Python 3.10:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh -O /tmp/miniconda.sh
bash /tmp/miniconda.sh -b -p ~/miniconda3
~/miniconda3/bin/conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
~/miniconda3/bin/conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
~/miniconda3/bin/conda init bash
source ~/.bashrc

1.2 Create the teleop environment

conda create -n tv python=3.10 pinocchio=3.1.0 numpy=1.26.4 -c conda-forge -y
conda activate tv

1.3 Clone this repo

cd ~
git clone https://github.com/<YOUR_ORG>/robot-teleop.git
cd robot-teleop

Everything is included — no submodules to init, no extra repos to clone.

1.4 Install packages from the repo

cd ~/robot-teleop

# Use the conda env's pip, not system pip
# Unitree SDK (includes MotionSwitcherClient, LocoClient, G1 modules)
/home/unitree/miniconda3/envs/tv/bin/pip install -e unitree_sdk2_python/

# Teleimager client (no-deps — server deps installed separately in Part 2)
/home/unitree/miniconda3/envs/tv/bin/pip install -e xr_teleoperate/teleop/teleimager/ --no-deps

# Televuer (WebXR interface)
/home/unitree/miniconda3/envs/tv/bin/pip install -e xr_teleoperate/teleop/televuer/

1.5 Install remaining dependencies

pip install 'vuer[all]==0.0.60'
pip install 'params-proto==2.13.2'
pip install meshcat==0.3.2
pip install 'rerun-sdk==0.21.0'
pip install sshkeyboard==2.3.1
pip install matplotlib==3.7.5
pip install git+https://github.com/unitreerobotics/logging-mp.git
pip install pymodbus       # Required for Inspire hands (Modbus TCP driver)
pip install numpy==1.26.4  # ALWAYS re-pin after installing anything

1.6 Install dex-retargeting (required for Inspire hands)

The dex-retargeting submodule handles finger pose retargeting from VR hand tracking to the Inspire hand's 6 DOFs. nlopt must be installed via conda because pip fails to build it on ARM:

conda activate tv
conda install -c conda-forge nlopt -y
cd ~/robot-teleop/xr_teleoperate/teleop/robot_control/dex-retargeting
/home/unitree/miniconda3/envs/tv/bin/pip install -e . --no-deps
/home/unitree/miniconda3/envs/tv/bin/pip install anytree pytransform3d trimesh lxml torch==2.3.0
/home/unitree/miniconda3/envs/tv/bin/pip install numpy==1.26.4

Verify:

python -c "from dex_retargeting import RetargetingConfig; print('OK')"

1.7 Install pymodbus (required for Inspire hands)

/home/unitree/miniconda3/envs/tv/bin/pip install pymodbus

1.8 Generate SSL certificates

Quest 3 requires HTTPS for WebXR:

cd ~/robot-teleop/xr_teleoperate/teleop/televuer
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key.pem -out cert.pem
mkdir -p ~/.config/xr_teleoperate/
cp cert.pem key.pem ~/.config/xr_teleoperate/

Part 2: Camera Server

The camera server runs in a separate conda environment to avoid dependency conflicts.

2.1 Create teleimager environment

conda create -n teleimager python=3.10 -y
conda activate teleimager

2.2 Install teleimager

sudo apt install -y libusb-1.0-0-dev libturbojpeg-dev
cd ~/robot-teleop/teleimager

# IMPORTANT: Use the conda env's pip, not system pip
/home/unitree/miniconda3/envs/teleimager/bin/pip install -e ".[server]"
/home/unitree/miniconda3/envs/teleimager/bin/pip install git+https://github.com/unitreerobotics/logging-mp.git

Critical: The system pip at /home/unitree/.local/bin/pip installs to Python 3.8. Always use the full path /home/unitree/miniconda3/envs/teleimager/bin/pip.

2.3 Camera permissions and certificates

cd ~/robot-teleop/teleimager
bash setup_uvc.sh
# Log out and back in for group change to take effect

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout key.pem -out cert.pem
mkdir -p ~/.config/xr_teleoperate/
cp cert.pem key.pem ~/.config/xr_teleoperate/

2.4 Configure camera

A working config is already included at teleimager/cam_config_server.yaml. To discover other cameras, run python -m teleimager.image_server --cf. The G1's Intel RealSense D430i uses /dev/video2 (480x640 RGB).

Default config (~/robot-teleop/teleimager/cam_config_server.yaml):

head_camera:
  enable_zmq: true
  zmq_port: 55555
  enable_webrtc: true
  webrtc_port: 60001
  webrtc_codec: h264
  type: opencv
  image_shape: [480, 640]
  binocular: false
  fps: 30
  video_id: 2
  serial_number: null
  physical_path: null

left_wrist_camera:
  enable_zmq: false
  zmq_port: 55556
  enable_webrtc: false
  webrtc_port: 60002
  webrtc_codec: h264
  type: opencv
  image_shape: [480, 640]
  binocular: false
  fps: 30
  video_id: null
  serial_number: null
  physical_path: null

right_wrist_camera:
  enable_zmq: false
  zmq_port: 55557
  enable_webrtc: false
  webrtc_port: 60003
  webrtc_codec: h264
  type: opencv
  image_shape: [480, 640]
  binocular: false
  fps: 30
  video_id: null
  serial_number: null
  physical_path: null

Wrist cameras must be present in the config (even disabled) — image_client.py expects them.

Inspire RH56DFTP Dexterous Hands

Overview

The Inspire RH56DFTP hands connect to the G1 via Ethernet using Modbus TCP. Each hand has 6 DOFs (little finger, ring, middle, index, thumb bend, thumb rotation) controlled by writing angle values (0–1000) to registers over TCP port 6000.

The original xr_teleoperate code uses inspire_sdkpy (a proprietary DDS-based SDK) to control the FTP hands. Since this SDK is not publicly available, we wrote a drop-in replacement (robot_hand_inspire_modbus.py) that talks directly to the hands via Modbus TCP, bypassing DDS entirely.

Hand Network Configuration

The Inspire hands ship with a default IP of 192.168.11.210 on a different subnet. On our G1, the hands have been reconfigured to the robot's subnet:

Hand	IP Address	Port	Status
Left	192.168.123.210	6000	Working
Right	192.168.123.211	6000	Working

To verify hand connectivity from the Jetson:

nc -zv 192.168.123.210 6000   # Left hand
nc -zv 192.168.123.211 6000   # Right hand

Modbus TCP Register Map

All registers use byte-addressing with device_id=255 (0xFF):

Register	Byte Address	Count	Range	Description
ANGLE_SET	1486	6 × int16	0–1000	Set target angle (0=open, 1000=closed)
ANGLE_ACT	1546	6 × int16	0–1000	Read actual angle
SPEED_SET	1522	6 × int16	0–1000	Movement speed per DOF
FORCE_SET	1498	6 × int16	0–3000	Force limit per DOF (grams)
FORCE_ACT	1582	6 × int16	-4000–4000	Read actual force (grams)
TEMP	1618	6 × uint8	0–100	Temperature (°C)
ERROR	1606	6 × uint8	bitmap	Error codes
HAND_ID	1000	1 byte	1–254	Hand ID

DOF order: [0] little, [1] ring, [2] middle, [3] index, [4] thumb bend, [5] thumb rotation.

The Modbus TCP Driver

The file robot_hand_inspire_modbus.py is a drop-in replacement for Inspire_Controller_FTP. It:

Uses pymodbus to talk directly to hands over Modbus TCP (no DDS, no inspire_sdkpy)
Has the same class name and constructor signature as the original
Auto-reconnects if a hand comes online later (checks every 5 seconds)
Supports environment variables INSPIRE_LEFT_IP and INSPIRE_RIGHT_IP for IP override
Re-establishes Modbus connections in the forked child process (sockets don't survive fork())

The import swap in teleop_hand_and_arm.py (line 188):

# Original (requires proprietary inspire_sdkpy):
# from teleop.robot_control.robot_hand_inspire import Inspire_Controller_FTP

# Our replacement (direct Modbus TCP):
from teleop.robot_control.robot_hand_inspire_modbus import Inspire_Controller_FTP

Standalone Hand Test Script

To test a hand directly without the full teleop stack:

#!/usr/bin/env python3
from pymodbus.client import ModbusTcpClient
import time

HAND_IP = "192.168.123.211"
client = ModbusTcpClient(HAND_IP, port=6000, timeout=3)
assert client.connect()

def read_angles():
    r = client.read_holding_registers(1546, count=6, device_id=255)
    return r.registers if not r.isError() else None

def set_angles(angles):
    client.write_registers(1486, values=angles, device_id=255)

print(f"Current angles: {read_angles()}")

set_angles([0, 0, 0, 0, 0, 0])       # Open all
time.sleep(2)
set_angles([1000, 1000, 1000, 1000, 1000, 1000])  # Close all
time.sleep(2)
set_angles([0, 0, 0, 0, 0, 0])       # Open again
time.sleep(2)

client.close()

Changing a Hand's IP Address

If a hand is still on the factory default IP (192.168.11.210), you need to either add a route or change the hand's IP. To change the IP via Modbus TCP:

# Write new IP octets to registers 1700-1703
# Example: change to 192.168.123.212
client.write_register(1700, 192, device_id=255)   # IP_PART1
client.write_register(1701, 168, device_id=255)   # IP_PART2
client.write_register(1702, 123, device_id=255)   # IP_PART3
client.write_register(1703, 212, device_id=255)   # IP_PART4
client.write_register(1005, 1, device_id=255)     # SAVE to flash
# Power-cycle the hand for the new IP to take effect

Troubleshooting Hands

If a hand is not responding, check:

Verify connectivity: nc -zv <hand_ip> 6000
Check the Ethernet cable inside the forearm — refer to Unitree's G1 Inspire Hand Assembly Guide for the signal board wiring
If the hand is on the factory default IP (192.168.11.210), add a route or change the IP (see above)

Running Teleoperation

1. Put G1 in motion control mode

Using the Unitree remote controller:

L2+B — Damping mode
L2+UP — Locked Standing
R1+X — Main motion control program (Regular mode)

2. Start the camera server (SSH session 1)

conda activate teleimager
cd ~/robot-teleop/teleimager
python -m teleimager.image_server

Expected output:

[OpenCVCamera: head_camera] initialized with 480x640 @ 30 FPS.
[Image Server] Image server has started, waiting for client connections...
[Image Server] head_camera is ready.

3. Launch teleop (SSH session 2)

Arms only (no hands):

conda activate tv
cd ~/robot-teleop/xr_teleoperate/teleop
python teleop_hand_and_arm.py --arm=G1_29 --motion

Arms + Inspire hands:

python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=hand --ee=inspire_ftp --display-mode=immersive

Wait for: 🟢 Press [r] to start syncing

You should see in the logs:

[Inspire_Controller_FTP] Initialize Inspire_Controller_FTP (Modbus TCP)...
[LeftHand] Connected to 192.168.123.210:6000
[RightHand] Connected to 192.168.123.211:6000

4. Connect the Quest 3

Ensure Quest 3 is on the same WiFi network (192.168.123.x subnet)
Open Meta Quest browser
Navigate to: https://192.168.123.164:8012/?ws=wss://192.168.123.164:8012
Accept SSL certificate warning (Advanced → Proceed)
Click Enter VR and allow tracking permissions
Align your arms with the robot's initial pose
Press r in the SSH terminal to start teleoperation
Press q to quit

Controls & Modes

Input modes

Hand tracking (default):

python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=hand

Bare hands tracked by Quest 3 cameras. No locomotion (use Unitree R3 controller for walking).

Controller mode:

python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=controller

Left joystick: walk forward/back + strafe
Right joystick: turn
Right A button: quit
Both thumbsticks pressed: emergency soft stop

Display modes

Flag	Mode
`--display-mode=immersive`	Full VR with robot's camera feed (default)
`--display-mode=pass-through`	See your real room with camera overlay
`--display-mode=ego`	First-person from robot's perspective

End-effectors

Flag	Hand
`--ee=inspire_ftp`	Inspire RH56DFTP hands via Modbus TCP (our custom driver)
`--ee=inspire_dfx`	Inspire RH56DFX hands via DDS (requires dfx_inspire_service)
`--ee=dex3`	Dex3-1 dexterous hand
`--ee=dex1`	Dex1-1 gripper
`--ee=brainco`	BrainCo hand

Example launch commands

# Arm only, hand tracking, immersive VR
python teleop_hand_and_arm.py --arm=G1_29 --motion

# Arms + Inspire hands, hand tracking, immersive VR (camera feed)
python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=hand --ee=inspire_ftp --display-mode=immersive

# Arms + Inspire hands, pass-through
python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=hand --ee=inspire_ftp --display-mode=pass-through

# Full body with controllers and pass-through
python teleop_hand_and_arm.py --arm=G1_29 --motion --input-mode=controller --display-mode=pass-through

# Record episodes for imitation learning
python teleop_hand_and_arm.py --arm=G1_29 --motion --ee=inspire_ftp --record

# Headless mode (no VR display, for testing)
python teleop_hand_and_arm.py --arm=G1_29 --motion --headless

Terminal controls

Key	Action
`r`	Start syncing robot to your movements
`s`	Start/stop recording (with `--record`)
`q`	Quit cleanly (arms return home first)

Replaying recorded episodes

replay_episode.py plays a recorded episode back through the same command interfaces the teleop loop uses (arm_ctrl.ctrl_dual_arm for arms, direct Modbus TCP writes for Inspire FTP hands). This is pure open-loop playback — no camera/state feedback, no training involved.

cd xr_teleoperate/teleop

# Replay an episode (arm + inspire_ftp)
python replay_episode.py \
    --episode-dir ./utils/data/thumbs_up/episode_0002 \
    --arm G1_29 --ee inspire_ftp --motion

The script:

Loads data.json from the episode directory.
Prompts for confirmation (skip with --yes).
Linearly ramps the arms from the current pose to the first recorded frame over --ramp-seconds (default 3s) to avoid a jerk.
Streams each frame at the recorded rate (30 Hz by default), sending actions.left_arm.qpos/right_arm.qpos to the arm controller and the normalized finger angles (0–1, scaled ×1000) to the hands over Modbus TCP.
Calls ctrl_dual_arm_go_home() on exit or Ctrl-C.

Useful flags:

Flag	Purpose
`--frequency 15`	Replay at half speed (easier to observe)
`--ramp-seconds 5`	Gentler initial approach to first frame
`--start-frame N --end-frame M`	Replay only a slice of the episode
`--sim`	Replay against Isaac sim (domain id 1) instead of real robot
`--yes`	Skip the interactive safety prompt

Important caveats:

Initial pose: the robot will ramp from wherever it currently is to the first recorded frame. The ramp prevents a jerk but the first-frame pose should be physically reachable from the arm's starting state.
World must match: open-loop means no feedback. If an object is 2 cm off from where it was during recording, the fingers will close on air. Replay is reliable for choreography (gestures, pose sequences) and only marginally reliable for manipulation of physical objects.
Test in sim first: add --sim to both the recording and replay commands for a dry run before running on real hardware.
Other end-effectors: currently only --ee inspire_ftp is implemented. dex3/dex1/inspire_dfx/brainco share the same structure but need their own low-level send path — extend the send_ee branch in replay_episode.py.
Locomotion: body.qpos (base velocity commands) is recorded but not replayed. Arm/hand only.

Safety

The G1 weighs 35 kg with 120 N·m knee torque and moves at 2+ m/s.

Always have the Unitree remote in hand — emergency stop is L2+B
Never cut power to a standing robot — it will fall
Clear the area — remove obstacles, equipment, and bystanders
Use the gantry/suspension frame for initial tests
Never touch moving joints, especially knees and waist
Monitor battery — stop when last cell indicator flashes (60-90 min runtime)
Short sessions (10-15 min) — visual latency (~120ms) can cause nausea
Never enter debug mode (L2+R2) — requires full reboot
Only Regular mode (R1+X) — Running mode (R2+A) is not supported

Version Pins

Package	Version	Notes
Python	3.10	Required by pinocchio and teleimager
NumPy	1.26.4	Must stay <2.0
Pinocchio	3.1.0	IK solver
CycloneDDS	0.10.2	Installed with SDK
vuer	0.0.60	WebXR streaming
params-proto	2.13.2	Required for vuer import
rerun-sdk	0.21.0	Must be <0.22 for numpy <2
meshcat	0.3.2	3D visualization
sshkeyboard	2.3.1	Terminal keyboard input
matplotlib	3.7.5	Required by weighted_moving_filter
pymodbus	3.12.1	Modbus TCP for Inspire hands
dex-retargeting	0.4.7	Finger pose retargeting for Inspire hands
nlopt	2.10.1	Install via conda, not pip (ARM build)
torch	2.3.0	Required by dex-retargeting

File Paths

What	Path
Repo root	`~/robot-teleop/`
xr_teleoperate	`~/robot-teleop/xr_teleoperate/`
Teleop entry point	`~/robot-teleop/xr_teleoperate/teleop/teleop_hand_and_arm.py`
Inspire Modbus driver	`~/robot-teleop/xr_teleoperate/teleop/robot_control/robot_hand_inspire_modbus.py`
Original Inspire driver	`~/robot-teleop/xr_teleoperate/teleop/robot_control/robot_hand_inspire.py`
dex-retargeting	`~/robot-teleop/xr_teleoperate/teleop/robot_control/dex-retargeting/`
Teleop conda env	`tv`
Unitree SDK	`~/robot-teleop/unitree_sdk2_python/`
Teleimager	`~/robot-teleop/teleimager/`
Teleimager conda env	`teleimager`
Camera config	`~/robot-teleop/teleimager/cam_config_server.yaml`
SSL certificates	`~/.config/xr_teleoperate/cert.pem`, `key.pem`
Hand test scripts	`~/robot-audio-agent/hands/`

Known Issues

Issue	Fix
Camera serial number matching fails	Use `video_id` only; set `serial_number: null`
System pip vs conda pip	Always use `/home/unitree/miniconda3/envs/<env>/bin/pip`
Camera not found after reboot	Unplug and replug the RealSense USB cable; verify with `ls /dev/video*`
numpy gets upgraded by pip	Run `pip install numpy==1.26.4` after every package installation
Wrist cameras required in config	Must be present (even disabled) or `image_client.py` throws KeyError
nlopt fails to build via pip on ARM	Install via conda: `conda install -c conda-forge nlopt -y`
Inspire hand not responding	Check Ethernet cable inside forearm; verify with `nc -zv <ip> 6000`
pymodbus installs to system Python	Always use `/home/unitree/miniconda3/envs/tv/bin/pip install pymodbus`
pymodbus API version differences	v3.12+ uses `device_id=` kwarg, not `slave=` or `unit=`
inspire_sdkpy not available	Use `robot_hand_inspire_modbus.py` instead (direct Modbus TCP)
Inspire hand IP override	Set `INSPIRE_LEFT_IP` / `INSPIRE_RIGHT_IP` env vars (defaults: `192.168.123.210` / `.211`)

Resources

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

This project is built on Unitree Robotics' xr_teleoperate framework (Apache-2.0) and bundles additional open-source components. See the NOTICE file for full third-party attribution, and the sub-project LICENSE files (which are preserved intact) for their respective terms.

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
teleimager		teleimager
unitree_sdk2_python		unitree_sdk2_python
xr_teleoperate		xr_teleoperate
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
NOTICE		NOTICE
README.md		README.md

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

G1 + Quest 3 Teleoperation

Repo Structure

Hardware

Quick Start

Setup

Part 1: Jetson Teleop Environment

1.1 Install Miniconda

1.2 Create the teleop environment

1.3 Clone this repo

1.4 Install packages from the repo

1.5 Install remaining dependencies

1.6 Install dex-retargeting (required for Inspire hands)

1.7 Install pymodbus (required for Inspire hands)

1.8 Generate SSL certificates

Part 2: Camera Server

2.1 Create teleimager environment

2.2 Install teleimager

2.3 Camera permissions and certificates

2.4 Configure camera

Inspire RH56DFTP Dexterous Hands

Overview

Hand Network Configuration

Modbus TCP Register Map

The Modbus TCP Driver

Standalone Hand Test Script

Changing a Hand's IP Address

Troubleshooting Hands

Running Teleoperation

1. Put G1 in motion control mode

2. Start the camera server (SSH session 1)

3. Launch teleop (SSH session 2)

4. Connect the Quest 3

Controls & Modes

Input modes

Display modes

End-effectors

Example launch commands

Terminal controls

Replaying recorded episodes

Safety

Version Pins

File Paths

Known Issues

Resources

License

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages