Robotic simulation and learning systems

Robotic simulation enables companies to virtually develop, test, and optimize complex automation solutions such as robots before they are actually used. It enables decision-makers to digitally map material flows and fleet strategies, identify risks at an early stage, and reduce costs. In logistics centers, the use of simulation tools increases planning reliability and efficiency. Fraunhofer IML supports you with in-depth expertise and powerful tools to holistically test and perfect robotic systems and material flows.

With advanced tools such as NVIDIA Isaac Sim and Isaac Lab, as well as our own simulation solutions in combination with material flow simulation, we accelerate the planning and development of logistics robot systems—while ensuring greater safety for both humans and machines.

Request simulation expertise now Sim

The strengths of robotics simulation are impressively demonstrated by the example of “O³dyn”: the omnidirectional pallet transport robot was completely replicated in the virtual world. Engineers used design data to create a precise 3D model, which they validated in the “NVIDIA Isaac Sim” environment. At the same time, virtual motion data from the PACELab was used to record the kinematic and dynamic properties of the real system. As a result, the digital twin is identical to the physical prototype down to the last detail.

Robotic simulation allows complex motion sequences such as sideways driving or skidding behavior to be accurately replicated. GPU-based rendering generates photorealistic sensor data for training computer vision models. This allows developers to change sensor and chassis configurations in seconds without having to build real prototypes. This significantly shortens development time and reduces costs.

With each iteration step, the gap between the digital twin and the real robot narrows. The teams test functions such as autonomous navigation and load carrier pickup entirely in the simulation. Only when they achieve the desired performance indicators do they transfer the software to the physical “O³dyn.” This seamless “sim-to-real transfer” increases reliability and minimizes the risk of costly rework.

We advise you on setting up your simulation environment, support you with modeling, and ensure a smooth transition from virtual testing to real-world operation. This ensures that your logistics and intralogistics robots offer optimal performance and safety right from the start.

How do fleets of mobile robots affect the overall material flow in warehouses or production facilities? This question pushes physically accurate robot simulation to its limits: although mechanics, actuators, and sensors can be precisely modeled, enormous computing resources are required for each robot. Scaling up to dozens or even hundreds of vehicles requires a lot of computing time. However, it is precisely the simultaneous consideration of all robots and adjacent process areas (goods receipt, order picking, goods issue, production stations) that is crucial for creating a robust digital twin of the overall system. Only then can key performance indicators such as throughput, utilization, and profitability be predicted with a high degree of reliability.

While classic conveyor technology (e.g., roller or chain conveyors) is relatively easy to map in material flow simulations, mobile robots require significantly more complex models. However, many tools available today use highly simplified approaches: they neglect differences between robot types, do not take fleet strategies into account, and ignore coordination topologies – with corresponding losses in realism and informative value.

The result is a scalable, realistic simulation of entire robot fleets—with reliable key figures for efficiency, safety, and cost-effectiveness. This gives you reliable planning security for your automation strategy at an early stage.

What sets us apart

• Many years of application experience in robotics simulation within our own robotic developments and customer projects across various simulation tools
• Expertise ranging from the selection of the appropriate simulation tool to modeling and creation of digital twins to the deployment of solutions from simulation into the real world
• Powerful infrastructure such as simulation servers and motion capture systems in the “PACELab” for “sim-to-real” comparison
• Holistic view of robotic systems: from virtual prototyping, development, training, and testing of autonomous functions in simulation to analysis of the resulting material flow of individual systems and entire robot fleets.

»By using simulation, we have been able to significantly reduce the development time for our robots and for our partners, and we are now testing new ideas and concepts in the virtual world in a resource-efficient manner. In the scaling and integration phase of the application, the combination with material flow simulation supports us in planning and optimizing entire fleets.«

Dr.-Ing. Jana Jost, Head of Department Robotics and Cognitive Systeme

Computer vision models based on synthetic training data

Training computer vision models requires a large amount of training data. Capturing real images for training AI models is time-consuming and costly, and data protection regulations also limit data collection.

Thanks to photorealistic renderers, synthetic image and depth data, including object annotations, can be generated in the simulation. This means that thousands of new training images can be generated in a very short time, from any object, using different camera types and with the required annotation format. Targeted randomization, such as varying backgrounds, lighting conditions, or object textures, covers a wide range of scenarios. The resulting data sets are extremely diverse and increase the robustness of the models in real-world use.

Using photorealistic renderers, we generate thousands of new training images of any object in minutes. The IMOCO4.E project developed a pipeline that generates pallet data sets in parallel and trains a model that reliably recognizes real pallets, even though the model had never seen a real pallet before. After deployment, the project partners succeeded in reliably capturing pallets under highly variable conditions.

Robots learn using reinforcement learning

With the increasing complexity of modern robot systems, from mobile manipulators to anthropomorphic and humanoid robots, degrees of freedom, speeds, and dynamic requirements are increasing, as is the variety of interactions with the environment and objects. Classic control and planning methods are reaching their limits in this area. Deep reinforcement learning explores parallel action spaces in simulation, learns through rewards, and uses domain randomization and fine-tuning for robust sim-to-real transfer. 

The successful projects at our institute are diverse: from learned object handling with manipulators to mobile manipulation to autonomous navigation of individual robots and entire fleets. Learned functions also play a central role in humanoids, e.g., in walking or whole-body control. We use available simulation tools such as NVIDIA Isaac Lab as well as our own developments, e.g., MuRoSim, a highly efficient 2D simulation for entire robot fleets.

In projects such as the"ai arena" the "DynaFoRo" junior research group, and the Lamarr Institute for Machine Learning and AI, we have implemented proven, successful solutions ranging from manipulation tasks to multi-robot navigation.