The Concept
Teleoperating industrial robots remotely is risky. A single wrong move can damage expensive hardware.
This project solves that by implementing a Digital Twin workflow. The user controls a virtual robot in Unity (VR), sets waypoints, and validates the trajectory visually. Only when the path is confirmed is the data sent to the physical Kinova Gen2 robot via ROS.
System Architecture
VR Client (Unity + C#)
The user interacts with the Ghost Robot. When a waypoint is set, Unity serializes the position and rotation (Quaternion) into a JSON packet.
TCP/IP Bridge (Python Middleware)
A custom Python script vr_bridge_server.py listens
on
Port 5005. It parses the JSON and publishes ROS messages to internal topics like /vr/add_point.
Trajectory Manager (ROS ActionLib)
The node trajectory_manager.py receives the
points and stores them in a queue (Queue System). Upon receiving a "Execute" command, it
sends goals sequentially to the Kinova Driver using ActionLib, ensuring smooth
transitions.
The "Safety Queue" Logic
def execute_trajectory(self): # Validate points before moving total = len(self.point_queue) print(f"🚀 Starting trajectory with {total} points...") for i, pose in enumerate(self.point_queue): # Send to Arm Driver goal.pose.pose = pose self.client.send_goal(goal) # BLOCKING: Wait for result before next point if self.client.wait_for_result(timeout=15.0): print(f"✅ Point {i} reached.") else: abort() # Safety Critical Stop
Why a Queue?
Direct teleoperation can be jittery. By buffering inputs into a queue, we ensure the robot executes smooth, planned paths.
Project Status
Fully Functional
- VR Integration: Real-time teleoperation via Meta Quest headset is smooth and responsive.
- Coordinate Mapping: Accurate translation of hand movements to robot end-effector poses.
Critical Challenge
Currently, the mathematical planner validates certain trajectories that create kinematic singularities on the physical robot.
Result: The software says "Move OK", but the real robot locks up safely to prevent damage, causing a desynchronization between the Digital Twin and reality.
Project Team
Project Context
Developed as a research prototype to explore Virtual Reality interfaces for industrial manipulation. Focus on user safety and intuitive control systems.