Scientists use virtual reality for fish to teach robots how to swarm

Fish are masters of coordinated motion. Schools of fish have no leader, yet individuals manage to stay in formation, avoid collisions, and respond with liquid flexibility to changes in their environment. Reproducing this combination of robustness and flexibility has been a long-standing challenge for human-engineered systems like robots.

Now, using virtual reality for freely-moving fish, a research team based in Konstanz has taken an important step toward that goal.

The research is published in the journal Science Robotics.

“Our work illustrates that solutions evolved by nature over millennia can inspire robust and efficient control laws in engineered systems,” said first author Liang Li from the University of Konstanz. Co-author Máté Nagy from Eötvös University underscores this, saying, “The discovery opens up exciting possibilities for future applications in robotics and autonomous vehicle design.”

Deciphering nature’s hidden algorithm

Using a virtual reality (VR) setup that mimics natural schooling, researchers placed individual juvenile zebrafish into networked arenas where each fish could freely interact with “holographic” virtual conspecifics. Each virtual fish was a projection of a real fish from another arena, meaning that fish could swim and interact together in the same virtual world. The fully immersive 3D environment lets researchers precisely manipulate visual stimuli and record how the fish respond.

This high level of control allowed the scientists to isolate exactly which cues the fish were using to guide their interactions with other fish. In other words, they could reverse-engineer the behavior of schooling in zebrafish to understand how fish solve the complex problem of coordinating their motion.

The solution they discovered was a simple and robust law based only on the perceived position, not the speed, of their neighbors to regulate their following behavior.

“We were surprised by how little information the fish need to effectively coordinate movements within a school,” says Iain Couzin, senior author on the study and Director of MPI-AB and speaker at the Cluster of Excellence: Collective Behavior. “They use local rules that are cognitively minimal, but functionally excellent.”

To see just how realistic the control law was, the team tested it with real fish. They conducted a VR “Turing test,” based on the concept of testing whether people can tell if they are interacting with a real human or with artificial intelligence.

In the aquatic Turing test, a real fish would swim with a virtual fish that switched between being real and being controlled by the algorithm they discovered. The real fish could not tell the difference. They behaved the same whether interacting with a real conspecific or the virtual follower governed by the algorithm.

From fish to machines

To test the broader utility of their discovery, the team embedded it in swarms of robotic cars, drones, and boats. The robots were tasked with following a moving target using either parameters from the zebrafish algorithm or from a state-of-the-art method used in autonomous vehicles called Model Predictive Controller (MPC).

Across all tests, the natural control law that fish have evolved delivered performance that was nearly indistinguishable from MPC in terms of accuracy and energy consumption—but at a fraction of the complexity.

Oliver Deussen, a co-author on the study and Professor of computer science at the University of Konstanz and speaker at the Cluster of Excellence: Collective Behavior:

“This work highlights the reciprocal relationship between robotics and biology—using robotics to explore biological mechanisms, which in turn can inspire new and effective robotic control strategies.”