Performance capture helps transform human performers into digital marvels. The technology enables artists to bring even the most fantastical visions to life with unmatched detail and humanity.
Specialized cameras and sensors record an actor’s voice, body movements, and facial expressions in real time. This raw information is converted into a digital format, creating a three-dimensional blueprint of the performance. Skilled animators, visual effects (VFX) artists, and game developers take this data and apply it to computer-generated characters, infusing them with uncanny realism and emotional depth.
The results are larger-than-life CGI characters that move, emote, and interact with stunning authenticity. From towering aliens in blockbuster movies to nuanced performances in cutting-edge video games, performance capture technology pushes the boundaries of storytelling and blurs the line between reality and imagination.
Performance capture has reshaped the entertainment industry by bringing digital characters to life with unprecedented realism. Post-production teams are using different performance capture technologies to shape the future of visual storytelling.
In optical performance capture, reflective or LED markers are placed on an actor’s face and body. Multiple cameras record the performance and track the movements of markers. Behind the scenes, sophisticated software translates the marker data into digital skeletons, which are incorporated into rigged 3D character models. The technique helps you create highly accurate and lifelike digital characters, making it widely used in big-name films and AAA video games.
Inertial motion capture gives actors the freedom to perform anywhere, all without cameras. Sensors such as accelerometers, gyroscopes, and magnetometers track body movements, providing real-time data processing for immediate feedback. No elaborate cameras are needed, making IMC ideal for non-location shoots — like those used for sports analysis and virtual reality applications. However, while simpler to implement, using IMC may sacrifice some of the finer details that optical systems capture.
Markerless technology captures a performance with nothing by cameras and powerful algorithms. Advanced computer vision and artificial intelligence (AI) analyze video frames just like a 3D scanner. Actors can perform naturally, unencumbered by markers or suits — Martin Scorsese used this tech in The Irishman (2019) to de-age the actors. While markerless may require complex multi-camera setups for best results, the potential of the technology is clear. People might soon be able to create accurate digital avatars using just their smartphones, especially as AI improves.
The 1990s saw the birth of motion capture technology, laying the groundwork for what would become performance capture. While the term was coined in the mid-1980s, it wasn’t until the 90s that the technology became viable for widespread use in entertainment.
This technology was used throughout the decade, from video games such as Virtual Fighter (1994) to movies like Batman Forever (1995) and Star Wars: Episode I — The Phantom Menace (1990). The character Jar Jar Binks was a fully CGI character animated using motion capture.
An interesting bridge between mocap and performance capture occurred in the early 2000s, with Andy Serkis’ portrayal of Gollum in The Lord of the Rings: The Two Towers (2002) and The Return of the King (2003). The role used a new real-time motion capture system (now often called performance capture), and the producers captured the actor’s performance in several stages.
Since then, performance capture technology has evolved to capture an entire performance in real-time while also being less physically obstructing for the actors.
There continue to be many well-known examples of performance capture as the software and technology become less expensive and more powerful. Serkis has popularized many of them, including Caesar in The Planet of the Apes reboot trilogy (2011–2017) and Supreme Leader Snoke in Star Wars: Episodes VII & VIII (2015–2017). Josh Brolin’s Thanos in Avengers: Infinity War (2018) and Avengers: End Game (2019) were also performance capture, as is Zoe Saldaña’s Neytiri and other characters in the ongoing Avatar (2009) movies.
While often used interchangeably, motion capture (mocap) and performance capture (Pcap) represent distinct stages in the evolution of digital animation technology.
Mocap focuses primarily on recording large-scale body movement. The technology captures major body movements of performers (such as stunt actors and dancers) using specialized mocap suits and software. However, it doesn’t typically capture facial expressions or other subtle gestures.
Pcap takes mocap to new heights, offering a more comprehensive approach. It captures major body movements, subtle gestures, facial expressions, and voice. The method uses advanced software and hardware systems to capture multiple parts of an actor’s performance at the same time.
Performance capture technology adds a luster to CGI productions that’s difficult, if not impossible, to create without it.
Performance capture animation combines the heights of visual artistic style with the minute human nuances of extraordinary natural acting. The fusion results in memorable spectacles that audiences and gamers love by creating iconic characters that become cultural touchstones.
Performance capture software and techniques accurately record the most subtle movements and expressions in an actor’s performance as concise data. This level of detail makes more emotionally engaging CGI character performances possible across games, TV, film, and other media.
Over the last few years, the efficiency and realism of real-time performance capture have improved while total costs have decreased. Such a shift enables more bootstrapped productions to take advantage of performance capture software to humanize CGI characters.
Performance capture lets actors portray characters beyond human limitations of any size, shape, or species. It also allows for seamless interaction between actors and CGI environments, opening up new realism of storytelling.
Contrary to initial concerns, performance capture often enhances an actor’s ability to embody a character by allowing for uninhibited expression. The immediate feedback from real-time visualization enables actors to adjust their performance on the fly, creating a more immersive experience.
Performance capture software can significantly streamline various aspects of production by reducing the need for practical effects and allowing for flexible editing post-production. This often leads to significant time and cost savings, especially for larger productions.
Scale your studio’s rendering and simulation capabilities, while equipping artists with powerful modeling and animation tools
WARNER BROS GAMES AVALANCHE
For the bestselling and acclaimed game Hogwarts Legacy, Avalanche’s virtual production team used real-time cinematics and the latest performance capture technology in Autodesk Maya and Autodesk MotionBuilder to graft actors’ movements onto rigged 3D characters, bringing the well-known fantasy world to life.
Image courtesy of Warner Bros. Games Avalanche
MPC/DISNEY
In Disney’s live-action The Jungle Book remake, the boy Mowgli’s co-stars are a bunch of animals. To achieve director Jon Favreau’s goal of endowing those talking animals with both realism and personality, creative studio MPC (The Mill) used performance capture with Autodesk Maya for rigging and animation.
Image courtesy of MPC ©2016 Disney Enterprises, Inc.
LUMINOUS PRODUCTIONS
Take an inside look at the performance capture, standard character rigging layouts, and animation in Autodesk Maya and MotionBuilder needed to bring Forspoken’s many game cutscenes to life.
Image courtesy of © 2023 Luminous Productions Co, Ltd.
As performance capture technology evolves, the line between reality and digital artistry will continue to blur. The door will open up to incredible new possibilities for storytelling and entertainment:
Advancements in AI and machine learning will likely play a major role in this future of performance capture. These technologies could enhance the accuracy of facial and body tracking, automate complex animation processes, and even generate realistic performances based on minimal input data. With more efficient production pipelines, smaller teams and more artists can create higher-quality digital performances — the only limit being their imagination.
Check out what motion capture can do for your productions, and how the sophisticated performance capture software from Autodesk can map actors’ movements and facial expressions onto 3D modeled characters.
Get up to speed on the overall picture of visual effects (VFX), which includes CGI, motion capture/performance capture technology, green screen filming, and compositing.
Performance capture and motion capture are important techniques within the larger concept of virtual production, which combines game technology with live-action filmmaking to place live actors within digitally created environments.
Read about Autodesk’s initiatives and partnerships that will make cloud-based virtual production more accessible to lower-budget projects as performance capture technology becomes more affordable, and real-time game engine image quality improves.
Learn the technologies associated with recording performance-capture data and then how to import and work with that data in Autodesk 3ds Max to animate a biped character.
Performance capture is the evolution of motion-capture technology, where an actor’s entire performance—body movements, gestures, facial expressions, and voice—are all captured as data simultaneously through the use of sensors, visual markers, and/or advanced camera techniques.
Then, through performance-capture software, animators and game producers can map the actor’s performance onto 3D-modeled characters, resulting in the highest level of realism from CGI characters.
The difference between motion capture (mocap) and performance capture (Pcap) is the extra detail—especially in facial expressions—that Pcap records.
Motion capture records the major body movements of an actor or stunt performer through mocap suits or special camera techniques. That data is transferred to a rigged 3D character model.
In addition to body movements, Pcap also captures an actor’s much more data-dense facial expressions and subtle gestures. This is accomplished through marker tracking on the actor’s face and body or through newer 3D scanning methods that translate every frame of a camera recording into performance data that can be transferred onto 3D character models—resulting in highly realistic and genuine CGI performances.
Performance capture (Pcap) is used because animators and visual effects (VFX) producers have always sought methods for making their work more life-like and believable. Pcap is currently the best way to transfer a human actor’s authentic, full performance onto a 3D-modeled character.
Early motion capture and Pcap transcended the technique of rotoscoping, where animators traced over filmed footage. In recent years, Pcap has evolved to become more effective at a lower overall expense than ever before.
Performance capture technology allows media producers to combine the power and subtlety of actors’ performances with the unique artistic vision of each production’s CGI animation and VFX.
Performance capture demands highly accurate motion sensors, making the process potentially expensive and complex. Acting without traditional sets or props might also the natural expressions of performers.
Advancements in technology are making performance capture more accessible, with improved sensors and real-time rendering creating even more realistic results. This evolution is empowering storytellers to bring more imaginative and immersive worlds to their audiences.
