{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-29-building-a-healthcare-robot-from-simulation-to-deployment/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "## TL;DR\n\nA hands-on guide to collecting data, training policies, and deploying autonomous medical robotics workflows on real hardware\n\n## Table-of-Contents\n\n- [Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac](https://huggingface.co/blog/lerobotxnvidia-healthcare#building-a-healthcare-robot-from-simulation-to-deployment-with-nvidia-isaac)\n - [TL;DR](https://huggingface.co/blog/lerobotxnvidia-healthcare#tldr)\n - [Table-of-Contents](https://huggingface.co/blog/lerobotxnvidia-healthcare#table-of-contents)\n - [Introduction](https://huggingface.co/blog/lerobotxnvidia-healthcare#introduction)\n - [SO-ARM Starter Workflow; Building an Embodied Surgical Assistant](https://huggingface.co/blog/lerobotxnvidia-healthcare#so-arm-starter-workflow-building-an-embodied-surgical-assistant)\n - [Technical Implementation](https://huggingface.co/blog/lerobotxnvidia-healthcare#technical-implementation)\n - [Sim2Real Mixed Training Approach](https://huggingface.co/blog/lerobotxnvidia-healthcare#sim2real-mixed-training-approach)\n - [Hardware Requirements](https://huggingface.co/blog/lerobotxnvidia-healthcare#hardware-requirements)\n - [Data Collection Implementation](https://huggingface.co/blog/lerobotxnvidia-healthcare#data-collection-implementation)\n - [Simulation Teleoperation Controls](https://huggingface.co/blog/lerobotxnvidia-healthcare#simulation-teleoperation-controls)\n - [Model Training Pipeline](https://huggingface.co/blog/lerobotxnvidia-healthcare#model-training-pipeline)\n - [End-to-End Sim Collect–Train–Eval Pipelines](https://huggingface.co/blog/lerobotxnvidia-healthcare#end-to-end-sim-collecttraineval-pipelines)\n - [Generate Synthetic Data in Simulation](https://huggingface.co/blog/lerobotxnvidia-healthcare#generate-synthetic-data-in-simulation)\n - [Train and Evaluate Policies](https://huggingface.co/blog/lerobotxnvidia-healthcare#train-and-evaluate-policies)\n - [Convert Models to TensorRT](https://huggingface.co/blog/lerobotxnvidia-healthcare#convert-models-to-tensorrt)\n - [Getting Started](https://huggingface.co/blog/lerobotxnvidia-healthcare#getting-started)\n - [Resources](https://huggingface.co/blog/lerobotxnvidia-healthcare#resources)\n\n## Introduction\n\nSimulation has been a cornerstone in medical imaging to address the data gap. However, in healthcare robotics until now, it's often been too slow, siloed, or difficult to translate into real-world systems.\n\nNVIDIA Isaac for Healthcare, a developer framework for AI healthcare robotics, enables healthcare robotics developers in solving these challenges via offering integrated data collection, training, and evaluation pipelines that work across both simulation and hardware. Specifically, the Isaac for Healthcare v0.4 release provides healthcare developers with an end-to-end [SO - ARM based starter workflow](https://github.com/isaac-for-healthcare/i4h-workflows/blob/main/workflows/so_arm_starter/README.md) and [the bring your own operating room tutorial](https://github.com/isaac-for-healthcare/i4h-workflows/blob/main/tutorials/assets/bring_your_own_or/README.md). The SO-ARM starter workflow lowers the barrier for MedTech developers to experience the full workflow from simulation to train to deployment and start building and validating autonomous on real hardware right away.\n\nIn this post, we'll walk through the starter workflow and its technical implementation details to help you build a surgical assistant robot in less time than ever imaginable before.\n\n## SO-ARM Starter Workflow; Building an Embodied Surgical Assistant\n\nThe SO-ARM starter workflow introduces a new way to explore surgical assistance tasks, and providing developers with a complete end-to-end pipeline for autonomous surgical assistance:\n\n- Collect real-world and synthetic data with SO-ARM using the LeRobot\n- Fine-tune GR00t N1.5, evaluate in IsaacLab, then deploy to hardware\n\nThis workflow gives developers a safe, repeatable environment to train and refine assistive skills before moving into the Operating Room.\n\n### Technical Implementation\n\nThe workflow implements a three-stage pipeline that integrates simulation and real hardware:\n\n1. Data Collection: Mixed simulation and real-world teleoperation demonstrations using using SO101 and LeRobot\n2. Model Training: Fine-tuning GR00T N1.5 on combined datasets with dual-camera vision\n3. Policy Deployment: Real-time inference on physical hardware with RTI DDS communication\n\nNotably, over 93% of the data used for policy training was generated synthetically in simulation, underscoring the strength of simulation in bridging the robotic data gap.\n\n### Sim2Real Mixed Training Approach\n\nThe workflow combines simulation and real-world data to address the fundamental challenge that training robots in the real world is expensive and limited, while pure simulation often fails to capture real-world complexities. The approach uses approximately 70 simulation episodes for diverse scenarios and environmental variations, combined with 10-20 real-world episodes for authenticity and grounding. This mixed training creates policies that generalize beyond either domain alone.\n\n### Hardware Requirements\n\nThe workflow requires:\n\n- GPU: RT Core-enabled architecture (Ampere or later) with ≥30GB VRAM for GR00TN1.5 inference\n- SO-ARM101 Follower: 6-DOF precision manipulator with dual-camera vision (wrist and room). The SO-ARM101 features WOWROBO vision components, including a wrist-mounted camera with a 3D-printed adapter\n- SO-ARM101 Leader: 6-DOF Teleoperation interface for expert demonstration collection\n\nNotably, developers could run all the simulation, training and deployment (3 computers needed for physical AI) on one [DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/).\n\n### Data Collection Implementation\n\n[![so100-healthcare-real-demo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-real-demo.gif)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-real-demo.gif)\n\nFor real-world data collection with SO-ARM101 hardware or any other version supported in LeRobot:\n\n```bash\npython lerobot-record \\ \n --robot.type=so101_follower \\ \n --robot.port= \\ \n --robot.cameras=\"{wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, room: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}\" \\ \n --robot.id=so101_follower_arm \\ \n --teleop.type=so101_leader \\ \n --teleop.port= \\ \n --teleop.id=so101_leader_arm \\ \n --dataset.repo_id=/surgical_assistance/surgical_assistance \\ \n --dataset.num_episodes=15 \\ \n --dataset.single_task=\"Prepare and hand surgical instruments to surgeon\" \n```\n\nFor simulation-based data collection:\n\n[![so100-healthcare-sim-demo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-sim-demo.gif)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-sim-demo.gif)\n\n```bash\n# With keyboard teleoperation\npython -m simulation.environments.teleoperation_record \\ \n --enable_cameras \\ \n --record \\ \n --dataset_path=/path/to/save/dataset.hdf5 \\ \n --teleop_device=keyboard\n\n# With SO-ARM101 leader arm \npython -m simulation.environments.teleoperation_record \\ \n --port= \\ \n --enable_cameras \\ \n --record \\ \n --dataset_path=/path/to/save/dataset.hdf5 \n```\n\n### Simulation Teleoperation Controls\n\nFor users without physical SO-ARM101 hardware, the workflow provides keyboard-based teleoperation with the following joint controls:\n\n- Joint 1 (shoulder\\_pan): Q (+) / U (-)\n- Joint 2 (shoulder\\_lift): W (+) / I (-)\n- Joint 3 (elbow\\_flex): E (+) / O (-)\n- Joint 4 (wrist\\_flex): A (+) / J (-)\n- Joint 5 (wrist\\_roll): S (+) / K (-)\n- Joint 6 (gripper): D (+) / L (-)\n- R Key: Reset recording environment\n- N Key: Mark episode as successful\n\n### Model Training Pipeline\n\nAfter collecting both simulation and real-world data, convert and combine datasets for training:\n\n```bash\n# Convert simulation data to LeRobot format\npython -m training.hdf5_to_lerobot \\ \n --repo_id=surgical_assistance_dataset \\ \n --hdf5_path=/path/to/your/sim_dataset.hdf5 \\ \n --task_description=\"Autonomous surgical instrument handling and preparation\" \n\n# Fine-tune GR00T N1.5 on mixed dataset \npython -m training.gr00t_n1_5.train \\ \n --dataset_path /path/to/your/surgical_assistance_dataset \\ \n --output_dir /path/to/surgical_checkpoints \\ \n --data_config so100_dualcam \n```\n\nThe trained model processes natural language instructions such as \"Prepare the scalpel for the surgeon\" or \"Hand me the forceps\" and executes the corresponding robotic actions. With LeRobot latest release (0.4.0) you will be able to fine-tune Gr00t N1.5 natively in LeRobot!\n\n## End-to-End Sim Collect–Train–Eval Pipelines\n\nSimulation is most powerful when it's part of a loop: collect → train → evaluate → deploy.\n\nWith v0.3, IsaacLab supports this full pipeline:\n\n### Generate Synthetic Data in Simulation\n\n- Teleoperate robots using keyboard or hardware controllers\n- Capture multi-camera observations, robot states, and actions\n- Create diverse datasets with edge cases impossible to collect safely in real environments\n\n### Train and Evaluate Policies\n\n- Deep integration with Isaac Lab's RL framework for PPO training\n- Parallel environments (thousands of simulations simultaneously)\n- Built-in trajectory analysis and success metrics\n- Statistical validation across varied scenarios\n\n### Convert Models to TensorRT\n\n- Automatic optimization for production deployment\n- Support for dynamic shapes and multi-camera inference\n- Benchmarking tools to verify real-time performance\n\nThis reduces time from experiment to deployment and makes sim2real a practical part of daily development.\n\n## Getting Started\n\nIsaac for Healthcare SO-ARM Starter Workflow is available now. To get started:\n\n1. Clone the repository: `git clone https://github.com/isaac-for-healthcare/i4h-workflows.git`\n2. Choose a workflow: Start with the SO-ARM Starter Workflow for surgical assistance or explore other workflows\n3. Run the setup: Each workflow includes an automated setup script (e.g., `tools/env_setup_so_arm_starter.sh`)\n\n### Resources\n\n- [GitHub Repository](https://github.com/isaac-for-healthcare/i4h-workflows): Complete workflow implementations\n- [Documentation](https://isaac-for-healthcare.github.io/i4h-docs/): Setup and usage guides\n- [GR00T Models](https://huggingface.co/nvidia/GR00T-N1.5-3B): Pre-trained foundation models\n- [Hardware Guides](https://huggingface.co/docs/lerobot/so101): SO-ARM101 setup instructions\n- [LeRobot Repository](https://github.com/huggingface/lerobot): End-to-end robotics learning\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

TL;DR

\n

A hands-on guide to collecting data, training policies, and deploying autonomous medical robotics workflows on real hardware

\n

Table-of-Contents

\n\n

Introduction

\n

Simulation has been a cornerstone in medical imaging to address the data gap. However, in healthcare robotics until now, it's often been too slow, siloed, or difficult to translate into real-world systems.

\n

NVIDIA Isaac for Healthcare, a developer framework for AI healthcare robotics, enables healthcare robotics developers in solving these challenges via offering integrated data collection, training, and evaluation pipelines that work across both simulation and hardware. Specifically, the Isaac for Healthcare v0.4 release provides healthcare developers with an end-to-end SO - ARM based starter workflow and the bring your own operating room tutorial. The SO-ARM starter workflow lowers the barrier for MedTech developers to experience the full workflow from simulation to train to deployment and start building and validating autonomous on real hardware right away.

\n

In this post, we'll walk through the starter workflow and its technical implementation details to help you build a surgical assistant robot in less time than ever imaginable before.

\n

SO-ARM Starter Workflow; Building an Embodied Surgical Assistant

\n

The SO-ARM starter workflow introduces a new way to explore surgical assistance tasks, and providing developers with a complete end-to-end pipeline for autonomous surgical assistance:

\n
    \n
  • Collect real-world and synthetic data with SO-ARM using the LeRobot
    • \n
  • Fine-tune GR00t N1.5, evaluate in IsaacLab, then deploy to hardware
    • \n
\n

This workflow gives developers a safe, repeatable environment to train and refine assistive skills before moving into the Operating Room.

\n

Technical Implementation

\n

The workflow implements a three-stage pipeline that integrates simulation and real hardware:

\n
    \n
  1. Data Collection: Mixed simulation and real-world teleoperation demonstrations using using SO101 and LeRobot
    2. \n
  2. Model Training: Fine-tuning GR00T N1.5 on combined datasets with dual-camera vision
    3. \n
  3. Policy Deployment: Real-time inference on physical hardware with RTI DDS communication
    4. \n
\n

Notably, over 93% of the data used for policy training was generated synthetically in simulation, underscoring the strength of simulation in bridging the robotic data gap.

\n

Sim2Real Mixed Training Approach

\n

The workflow combines simulation and real-world data to address the fundamental challenge that training robots in the real world is expensive and limited, while pure simulation often fails to capture real-world complexities. The approach uses approximately 70 simulation episodes for diverse scenarios and environmental variations, combined with 10-20 real-world episodes for authenticity and grounding. This mixed training creates policies that generalize beyond either domain alone.

\n

Hardware Requirements

\n

The workflow requires:

\n
    \n
  • GPU: RT Core-enabled architecture (Ampere or later) with ≥30GB VRAM for GR00TN1.5 inference
    • \n
  • SO-ARM101 Follower: 6-DOF precision manipulator with dual-camera vision (wrist and room). The SO-ARM101 features WOWROBO vision components, including a wrist-mounted camera with a 3D-printed adapter
    • \n
  • SO-ARM101 Leader: 6-DOF Teleoperation interface for expert demonstration collection
    • \n
\n

Notably, developers could run all the simulation, training and deployment (3 computers needed for physical AI) on one DGX Spark.

\n

Data Collection Implementation

\n

\"so100-healthcare-real-demo\"

\n

For real-world data collection with SO-ARM101 hardware or any other version supported in LeRobot:

\n
python lerobot-record \\ \n  --robot.type=so101_follower \\ \n  --robot.port=<follower_port_id> \\ \n  --robot.cameras="{wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, room: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}" \\ \n  --robot.id=so101_follower_arm \\ \n  --teleop.type=so101_leader \\ \n  --teleop.port=<leader_port_id> \\ \n  --teleop.id=so101_leader_arm \\ \n  --dataset.repo_id=<user>/surgical_assistance/surgical_assistance \\ \n  --dataset.num_episodes=15 \\ \n  --dataset.single_task="Prepare and hand surgical instruments to surgeon" \n
\n

For simulation-based data collection:

\n

\"so100-healthcare-sim-demo\"

\n
# With keyboard teleoperation\npython -m simulation.environments.teleoperation_record \\ \n  --enable_cameras \\ \n  --record \\ \n  --dataset_path=/path/to/save/dataset.hdf5 \\ \n  --teleop_device=keyboard\n\n# With SO-ARM101 leader arm \npython -m simulation.environments.teleoperation_record \\ \n  --port=<your_leader_arm_port_id> \\ \n  --enable_cameras \\ \n  --record \\ \n  --dataset_path=/path/to/save/dataset.hdf5 \n
\n

Simulation Teleoperation Controls

\n

For users without physical SO-ARM101 hardware, the workflow provides keyboard-based teleoperation with the following joint controls:

\n
    \n
  • Joint 1 (shoulder_pan): Q (+) / U (-)
    • \n
  • Joint 2 (shoulder_lift): W (+) / I (-)
    • \n
  • Joint 3 (elbow_flex): E (+) / O (-)
    • \n
  • Joint 4 (wrist_flex): A (+) / J (-)
    • \n
  • Joint 5 (wrist_roll): S (+) / K (-)
    • \n
  • Joint 6 (gripper): D (+) / L (-)
    • \n
  • R Key: Reset recording environment
    • \n
  • N Key: Mark episode as successful
    • \n
\n

Model Training Pipeline

\n

After collecting both simulation and real-world data, convert and combine datasets for training:

\n
# Convert simulation data to LeRobot format\npython -m training.hdf5_to_lerobot \\ \n  --repo_id=surgical_assistance_dataset \\ \n  --hdf5_path=/path/to/your/sim_dataset.hdf5 \\ \n  --task_description="Autonomous surgical instrument handling and preparation" \n\n# Fine-tune GR00T N1.5 on mixed dataset \npython -m training.gr00t_n1_5.train \\ \n  --dataset_path /path/to/your/surgical_assistance_dataset \\ \n  --output_dir /path/to/surgical_checkpoints \\ \n  --data_config so100_dualcam \n
\n

The trained model processes natural language instructions such as "Prepare the scalpel for the surgeon" or "Hand me the forceps" and executes the corresponding robotic actions. With LeRobot latest release (0.4.0) you will be able to fine-tune Gr00t N1.5 natively in LeRobot!

\n

End-to-End Sim Collect–Train–Eval Pipelines

\n

Simulation is most powerful when it's part of a loop: collect → train → evaluate → deploy.

\n

With v0.3, IsaacLab supports this full pipeline:

\n

Generate Synthetic Data in Simulation

\n
    \n
  • Teleoperate robots using keyboard or hardware controllers
    • \n
  • Capture multi-camera observations, robot states, and actions
    • \n
  • Create diverse datasets with edge cases impossible to collect safely in real environments
    • \n
\n

Train and Evaluate Policies

\n
    \n
  • Deep integration with Isaac Lab's RL framework for PPO training
    • \n
  • Parallel environments (thousands of simulations simultaneously)
    • \n
  • Built-in trajectory analysis and success metrics
    • \n
  • Statistical validation across varied scenarios
    • \n
\n

Convert Models to TensorRT

\n
    \n
  • Automatic optimization for production deployment
    • \n
  • Support for dynamic shapes and multi-camera inference
    • \n
  • Benchmarking tools to verify real-time performance
    • \n
\n

This reduces time from experiment to deployment and makes sim2real a practical part of daily development.

\n

Getting Started

\n

Isaac for Healthcare SO-ARM Starter Workflow is available now. To get started:

\n
    \n
  1. Clone the repository: git clone https://github.com/isaac-for-healthcare/i4h-workflows.git
    2. \n
  2. Choose a workflow: Start with the SO-ARM Starter Workflow for surgical assistance or explore other workflows
    3. \n
  3. Run the setup: Each workflow includes an automated setup script (e.g., tools/env_setup_so_arm_starter.sh)
    4. \n
\n

Resources

\n\n", "content_markdown": "## TL;DR\n\nA hands-on guide to collecting data, training policies, and deploying autonomous medical robotics workflows on real hardware\n\n## Table-of-Contents\n\n- [Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac](https://huggingface.co/blog/lerobotxnvidia-healthcare#building-a-healthcare-robot-from-simulation-to-deployment-with-nvidia-isaac)\n - [TL;DR](https://huggingface.co/blog/lerobotxnvidia-healthcare#tldr)\n - [Table-of-Contents](https://huggingface.co/blog/lerobotxnvidia-healthcare#table-of-contents)\n - [Introduction](https://huggingface.co/blog/lerobotxnvidia-healthcare#introduction)\n - [SO-ARM Starter Workflow; Building an Embodied Surgical Assistant](https://huggingface.co/blog/lerobotxnvidia-healthcare#so-arm-starter-workflow-building-an-embodied-surgical-assistant)\n - [Technical Implementation](https://huggingface.co/blog/lerobotxnvidia-healthcare#technical-implementation)\n - [Sim2Real Mixed Training Approach](https://huggingface.co/blog/lerobotxnvidia-healthcare#sim2real-mixed-training-approach)\n - [Hardware Requirements](https://huggingface.co/blog/lerobotxnvidia-healthcare#hardware-requirements)\n - [Data Collection Implementation](https://huggingface.co/blog/lerobotxnvidia-healthcare#data-collection-implementation)\n - [Simulation Teleoperation Controls](https://huggingface.co/blog/lerobotxnvidia-healthcare#simulation-teleoperation-controls)\n - [Model Training Pipeline](https://huggingface.co/blog/lerobotxnvidia-healthcare#model-training-pipeline)\n - [End-to-End Sim Collect–Train–Eval Pipelines](https://huggingface.co/blog/lerobotxnvidia-healthcare#end-to-end-sim-collecttraineval-pipelines)\n - [Generate Synthetic Data in Simulation](https://huggingface.co/blog/lerobotxnvidia-healthcare#generate-synthetic-data-in-simulation)\n - [Train and Evaluate Policies](https://huggingface.co/blog/lerobotxnvidia-healthcare#train-and-evaluate-policies)\n - [Convert Models to TensorRT](https://huggingface.co/blog/lerobotxnvidia-healthcare#convert-models-to-tensorrt)\n - [Getting Started](https://huggingface.co/blog/lerobotxnvidia-healthcare#getting-started)\n - [Resources](https://huggingface.co/blog/lerobotxnvidia-healthcare#resources)\n\n## Introduction\n\nSimulation has been a cornerstone in medical imaging to address the data gap. However, in healthcare robotics until now, it's often been too slow, siloed, or difficult to translate into real-world systems.\n\nNVIDIA Isaac for Healthcare, a developer framework for AI healthcare robotics, enables healthcare robotics developers in solving these challenges via offering integrated data collection, training, and evaluation pipelines that work across both simulation and hardware. Specifically, the Isaac for Healthcare v0.4 release provides healthcare developers with an end-to-end [SO - ARM based starter workflow](https://github.com/isaac-for-healthcare/i4h-workflows/blob/main/workflows/so_arm_starter/README.md) and [the bring your own operating room tutorial](https://github.com/isaac-for-healthcare/i4h-workflows/blob/main/tutorials/assets/bring_your_own_or/README.md). The SO-ARM starter workflow lowers the barrier for MedTech developers to experience the full workflow from simulation to train to deployment and start building and validating autonomous on real hardware right away.\n\nIn this post, we'll walk through the starter workflow and its technical implementation details to help you build a surgical assistant robot in less time than ever imaginable before.\n\n## SO-ARM Starter Workflow; Building an Embodied Surgical Assistant\n\nThe SO-ARM starter workflow introduces a new way to explore surgical assistance tasks, and providing developers with a complete end-to-end pipeline for autonomous surgical assistance:\n\n- Collect real-world and synthetic data with SO-ARM using the LeRobot\n- Fine-tune GR00t N1.5, evaluate in IsaacLab, then deploy to hardware\n\nThis workflow gives developers a safe, repeatable environment to train and refine assistive skills before moving into the Operating Room.\n\n### Technical Implementation\n\nThe workflow implements a three-stage pipeline that integrates simulation and real hardware:\n\n1. Data Collection: Mixed simulation and real-world teleoperation demonstrations using using SO101 and LeRobot\n2. Model Training: Fine-tuning GR00T N1.5 on combined datasets with dual-camera vision\n3. Policy Deployment: Real-time inference on physical hardware with RTI DDS communication\n\nNotably, over 93% of the data used for policy training was generated synthetically in simulation, underscoring the strength of simulation in bridging the robotic data gap.\n\n### Sim2Real Mixed Training Approach\n\nThe workflow combines simulation and real-world data to address the fundamental challenge that training robots in the real world is expensive and limited, while pure simulation often fails to capture real-world complexities. The approach uses approximately 70 simulation episodes for diverse scenarios and environmental variations, combined with 10-20 real-world episodes for authenticity and grounding. This mixed training creates policies that generalize beyond either domain alone.\n\n### Hardware Requirements\n\nThe workflow requires:\n\n- GPU: RT Core-enabled architecture (Ampere or later) with ≥30GB VRAM for GR00TN1.5 inference\n- SO-ARM101 Follower: 6-DOF precision manipulator with dual-camera vision (wrist and room). The SO-ARM101 features WOWROBO vision components, including a wrist-mounted camera with a 3D-printed adapter\n- SO-ARM101 Leader: 6-DOF Teleoperation interface for expert demonstration collection\n\nNotably, developers could run all the simulation, training and deployment (3 computers needed for physical AI) on one [DGX Spark](https://www.nvidia.com/en-us/products/workstations/dgx-spark/).\n\n### Data Collection Implementation\n\n[![so100-healthcare-real-demo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-real-demo.gif)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-real-demo.gif)\n\nFor real-world data collection with SO-ARM101 hardware or any other version supported in LeRobot:\n\n```bash\npython lerobot-record \\ \n --robot.type=so101_follower \\ \n --robot.port= \\ \n --robot.cameras=\"{wrist: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, room: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}}\" \\ \n --robot.id=so101_follower_arm \\ \n --teleop.type=so101_leader \\ \n --teleop.port= \\ \n --teleop.id=so101_leader_arm \\ \n --dataset.repo_id=/surgical_assistance/surgical_assistance \\ \n --dataset.num_episodes=15 \\ \n --dataset.single_task=\"Prepare and hand surgical instruments to surgeon\" \n```\n\nFor simulation-based data collection:\n\n[![so100-healthcare-sim-demo](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-sim-demo.gif)](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/lerobot-blog/nvidia-healthcare/lerobotxnvidia-healthcare-sim-demo.gif)\n\n```bash\n# With keyboard teleoperation\npython -m simulation.environments.teleoperation_record \\ \n --enable_cameras \\ \n --record \\ \n --dataset_path=/path/to/save/dataset.hdf5 \\ \n --teleop_device=keyboard\n\n# With SO-ARM101 leader arm \npython -m simulation.environments.teleoperation_record \\ \n --port= \\ \n --enable_cameras \\ \n --record \\ \n --dataset_path=/path/to/save/dataset.hdf5 \n```\n\n### Simulation Teleoperation Controls\n\nFor users without physical SO-ARM101 hardware, the workflow provides keyboard-based teleoperation with the following joint controls:\n\n- Joint 1 (shoulder\\_pan): Q (+) / U (-)\n- Joint 2 (shoulder\\_lift): W (+) / I (-)\n- Joint 3 (elbow\\_flex): E (+) / O (-)\n- Joint 4 (wrist\\_flex): A (+) / J (-)\n- Joint 5 (wrist\\_roll): S (+) / K (-)\n- Joint 6 (gripper): D (+) / L (-)\n- R Key: Reset recording environment\n- N Key: Mark episode as successful\n\n### Model Training Pipeline\n\nAfter collecting both simulation and real-world data, convert and combine datasets for training:\n\n```bash\n# Convert simulation data to LeRobot format\npython -m training.hdf5_to_lerobot \\ \n --repo_id=surgical_assistance_dataset \\ \n --hdf5_path=/path/to/your/sim_dataset.hdf5 \\ \n --task_description=\"Autonomous surgical instrument handling and preparation\" \n\n# Fine-tune GR00T N1.5 on mixed dataset \npython -m training.gr00t_n1_5.train \\ \n --dataset_path /path/to/your/surgical_assistance_dataset \\ \n --output_dir /path/to/surgical_checkpoints \\ \n --data_config so100_dualcam \n```\n\nThe trained model processes natural language instructions such as \"Prepare the scalpel for the surgeon\" or \"Hand me the forceps\" and executes the corresponding robotic actions. With LeRobot latest release (0.4.0) you will be able to fine-tune Gr00t N1.5 natively in LeRobot!\n\n## End-to-End Sim Collect–Train–Eval Pipelines\n\nSimulation is most powerful when it's part of a loop: collect → train → evaluate → deploy.\n\nWith v0.3, IsaacLab supports this full pipeline:\n\n### Generate Synthetic Data in Simulation\n\n- Teleoperate robots using keyboard or hardware controllers\n- Capture multi-camera observations, robot states, and actions\n- Create diverse datasets with edge cases impossible to collect safely in real environments\n\n### Train and Evaluate Policies\n\n- Deep integration with Isaac Lab's RL framework for PPO training\n- Parallel environments (thousands of simulations simultaneously)\n- Built-in trajectory analysis and success metrics\n- Statistical validation across varied scenarios\n\n### Convert Models to TensorRT\n\n- Automatic optimization for production deployment\n- Support for dynamic shapes and multi-camera inference\n- Benchmarking tools to verify real-time performance\n\nThis reduces time from experiment to deployment and makes sim2real a practical part of daily development.\n\n## Getting Started\n\nIsaac for Healthcare SO-ARM Starter Workflow is available now. To get started:\n\n1. Clone the repository: `git clone https://github.com/isaac-for-healthcare/i4h-workflows.git`\n2. Choose a workflow: Start with the SO-ARM Starter Workflow for surgical assistance or explore other workflows\n3. Run the setup: Each workflow includes an automated setup script (e.g., `tools/env_setup_so_arm_starter.sh`)\n\n### Resources\n\n- [GitHub Repository](https://github.com/isaac-for-healthcare/i4h-workflows): Complete workflow implementations\n- [Documentation](https://isaac-for-healthcare.github.io/i4h-docs/): Setup and usage guides\n- [GR00T Models](https://huggingface.co/nvidia/GR00T-N1.5-3B): Pre-trained foundation models\n- [Hardware Guides](https://huggingface.co/docs/lerobot/so101): SO-ARM101 setup instructions\n- [LeRobot Repository](https://github.com/huggingface/lerobot): End-to-end robotics learning\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-10-29T00:00:00+00:00", "date_modified": "2025-10-29T00:00:00+00:00", "date_published": "2025-10-29T00:00:00+00:00", "description": "TL;DR A hands-on guide to collecting data, training policies, and deploying autonomous medical robotics workflows on real hardware Table-of-Contents Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac TL;DR…", "external_url": "https://huggingface.co/blog/lerobotxnvidia-healthcare", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/10/2025-10-29-building-a-healthcare-robot-from-simulation-to-deployment.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/10/2025-10-29-building-a-healthcare-robot-from-simulation-to-deployment.md" }, "id": "urn:aggr:item:f9a9bb03de836597cfe3d3d40fab9b4105baa83a", "isBasedOn": { "@id": "https://huggingface.co/blog/lerobotxnvidia-healthcare" }, "mainEntity": { "@id": "https://huggingface.co/blog/lerobotxnvidia-healthcare", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-29-building-a-healthcare-robot-from-simulation-to-deployment/#webpage" }, "dateModified": "2025-10-29T00:00:00+00:00", "datePublished": "2025-10-29T00:00:00+00:00", "headline": "Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac", "timeRequired": "PT5M", "url": "https://huggingface.co/blog/lerobotxnvidia-healthcare", "wordCount": 1059 }, "name": "Archived snapshot: Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac", "preview": null, "reading_minutes": 5, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Building a Healthcare Robot from Simulation to Deployment with NVIDIA Isaac", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-10-29-building-a-healthcare-robot-from-simulation-to-deployment/", "word_count": 1059 }