The dance of the RoboMasters or: The language of movement

It is a true spectacle of nature: Each year around 50 million migratory birds migrate south to spend the winter there. These “moving clouds” – flocks of over a million starlings – are especially impressive. The birds know how to move in formation to reach their destination, without an individual starling knowing what the flock is doing as a whole. It only looks to the left and to the right: If its neighbors change direction, it does the same. This prevents collision. The same is true of the RoboMasters, a fleet of small, autonomous vehicles.

Of course, robots do not have “eyes” as birds do. Instead, the RoboMasters have laser scanners. These highly developed sensors allow the robots to precisely measure distances to other objects and robots. This not only allows them to avoid obstacles but also to interpret the behavior of other robots and predict their movements. They know whether another robot is going to pass on the left or right side and can react accordingly. This “swarm behavior” allows the RoboMasters to operate smoothly in different environments without colliding with each other

But how does a RoboMaster learn to act in a vehicle fleet and adapt to unforeseeable obstacles at the same time? The answer lies in machine learning, more precisely in Deep Reinforcement Learning (Deep RL). In the case of the RoboMasters, this means that they are trained in a simulation that was specially developed for them. During the training, the robots receive rewards for good actions and punishments for bad ones. For example, they receive the value 1 when they come closer to their destination and the value –1 when they go away from it. They receive an especially high value when they reach their goal and an especially low one when they drive into a wall, for example. The neural network (the artificial intelligence (AI)) learns from these rewards and optimizes its behavior in the various training scenarios to maximize the reward. “You can think of it like a  video game in which the objective is to score as many points as possible and not lose any lives,” explains Christian Jestel of Fraunhofer IML, who wrote the simulation. 

The simulation has the advantage of being faster and safer than training in the real world. In the simulation, the robots can go through thousands of scenarios without causing physical damage in the real world. This considerably accelerates the learning process. Only after successful training in the simulation is the AI transferred to a real robot equipped with a minicomputer, in the hope that the simulation has described reality accurately enough. This transition from the simulation to reality is decisive; it is one of the greatest challenges in robotics because simulation and reality are never exactly the same. The more accurately the simulation depicts reality, the better the robot will function in the real world.

The term “reality gap” describes the difference between a simulation and reality. The smaller this gap is, the more seamlessly the AI can act in the real world. “Everything depends on how well the simulation depicts the physical characteristics and environments of the real world,” explains Jestel.

A special feature of the RoboMasters is their capacity for decentralized navigation. Unlike conventional autonomous robots that are controlled via a central computer, the RoboMasters make their decisions autonomously and based on their perception of the environment. This makes them ideal for use in dynamic environments such as warehouses or transshipment points. As soon as the RoboMaster has a destination, it is able to find its way independently and safely without the need for prior mapping of the environment or for human interventions or complicated reprogramming. This could considerably increase flexibility and efficiency in industrial logistics in the future.

“The RoboMasters are robots from the Chinese manufacturer DJI, some of which we have slightly adapted to the requirements of the research project. The vehicles are examples for all smart automated guided vehicles and mobile robots that are to be controlled using algorithms,” according to Jestel. 

The research on the RoboMaster shows what opportunities are offered by simulation-based AI in robotics. Decentralized navigation and the ability to react to non-cooperative elements in the environment could shape the future of industrial automation. As the next step, the team led by Christian Jestel plans to integrate “leaders” in the vehicle fleet and to improve evasive behavior not only among the vehicles but also towards people and stationary objects.