Realistic avatars for the virtual zoo

Filmmakers and developers of computer games will have a new way of animating animals in the future. A team led by researchers at the Max Planck Institute for Intelligent Systems in Tübingen, Germany, has developed a technique that uses photographs alone to create lifelike 3-D models of almost all quadrupeds. Animations of these avatars can realistically imitate the movements of animals. But the simple method of bringing animals to life on the computer is not only interesting for the entertainment industry. Many people have lost a beloved family pet. Now this technology can bring them back to "life" as a virtual 3-D avatar. It could also benefit biologists in species protection and help to make children in particular aware of the importance of biodiversity.

"Populating the virtual world with animals used to be a lot of manual work," says Michael J. Black, director of the Perceiving Systems department at the MPI for Intelligent Systems. Today, animators laboriously design 3-D animals using graphics software. While there are technologies to create realistic 3-D avatars of humans, no such technology can automatically capture the shape of wild animals. "You can't bring a tiger into the lab to be scanned and you can't take expensive scanners out into nature," says Black.

The team, whose technique creates avatars of animals including their fur or skin with reasonable effort, includes scientists from the CNR Institute for Applied Mathematics and Information Technology (IMATI) in Milan and the University of California in Berkeley as well as Max Planck researchers. "Ultimately, 3-D animal capture is about reconstructing the 3-D structure from 2-D images of moving objects captured by various unknown and moving cameras," explains Silvia Zuffi, the lead author on the project from IMATI.

In their new method, the researchers only need a few photos of an animal in different views to calculate realistic models that can then be animated naturally. Unlike other approaches, the images from which the avatars are generated can also come from different cameras. The computer program delivers such convincing results because it doesn't start from scratch when it creates a model. "The key is to start with a rough 3-D shape model that can explain the shapes of many animals and then refine this based on the observed images," says Michael J. Black.

Helping endangered animals in the wild

In their current work, the researchers demonstrate that their method works for tigers, bears, horses and rhinos. "Our algorithm generates realistic models of animals for which no previous 3-D models exist," says Silvia Zuffi. For example, the researchers were able to reconstruct the 3-D shape of the extinct Tasmanian tiger from low-quality film footage shot in the 1930s. The researchers now want to make the method fully automated so that it can operate on thousands of images of animals in the wild.

Fully automated 3-D animal shape capture would not only facilitate the work of game developers and animators, but also supports animal conservation. Given photographs of animals captured by camera traps, the method could determine whether their shape, i.e. their weight, has changed. Species conservationists would thus receive quick and reliable statements as to whether endangered animals are doing well.

Explore further: Researchers dress virtual avatars with digitally captured clothing

More information: Silvia Zuffi, Angjoo Kanazawa, Michael J. Black, "Lions and Tigers and Bears: Capturing Non-Rigid, 3D, Articulated Shape from Images," Proc. CVPR, 2018.

The work appears in the annual IEEE Conference on Computer Vision and Pattern Recognition (CVPR) held in Salt Lake City. 

For more about the research see: smalr

The 3D models are freely available for download from: sketchfab