Alberto Menache, former Digital Effects Supervisor at Sony Pictures’ Imageworks, and author of Understanding Motion Capture for Computer Animation – from our sister publisher Morgan Kaufmann has worked on blockbuster films such as Spiderman, The Polar Express, and Superman Returns. In Understanding Motion Capture for Computer Animation he discusses the many ups and downs experienced implementing motion capture technology in Shrek, Beuwolf, The Polar Express, Avatar, and others. This excerpt from Understanding Motion Capture for Computer Animation examines motion capture in relation to Shrek and Beowolf.
Motion capture, performance capture, Satan’s rotoscope—call it what you may, it is a technology that rose to the top of the hype machine among entertainment industry watchers, and then became a topic of major controversy for various reasons.
The first of many problems was the initial perception of what could be done with motion capture. Years ago, when the technology was first used to generate animated characters in three dimensions, the results amazed most people because nobody had ever seen realistic human motion used for 3D character animation. Many people in the industry thought they had found the replacement for the traditional character animator. However, because the captured motion data available in those days was very noisy and expensive, it quickly became clear that it wasn’t going to replace manual labor any time soon. In addition, off-the-shelf computer animation software with user-friendly interfaces such as Maya started to emerge, which helped more artistic and less technically oriented people enter the business. Character animators started to work in three dimensions.
In the recent past, the use of performance animation in a project was largely driven by cost. Many projects failed or fell into trouble due to lack of understanding, because the decision to use motion capture was taken very lightly and made by the wrong people and for the wrong reasons. Some of these projects entered into production with the client’s informed knowledge that motion capture may or may not work, and a budget that supported this notion. Other clients blindly adopted performance animation as the primary production tool, based on a non-representative test or no test at all, and almost always ran into trouble. In some cases, the problems occurred early in production,
resulting in manageable costs; in other cases, the problems weren’t identified until expenses were so excessive that the projects had to be reworked or cancelled. Projects such as Marvin the Martian in the 3rd Dimension, the first incarnation of Shrek, Casper, and Total Recall were unsuccessful in using performance animation for different reasons.
Others, such as Titanic and Batman and Robin, were successful to a certain degree: They were able to use some of the performance animation, but the key-frame animation workload was larger than expected. A few projects managed to successfully determine early in production whether it was feasible to use motion capture or not. Disney’s Mighty Joe Young, Dinosaur, and Spiderman are good examples of projects in which motion capture was tested well in advance and ended up being discarded. In live-action films, large crowd scenes and background human stunt actions are some of the best applications. Character animation pieces, such as Toy Story or Shrek, will almost never have a use for performance animation, but let’s not forget Happy Feet.
Before the original Shrek animated film started production, DreamWorks Feature Animation had been developing it to be mostly based on performance animation.
The decision to use performance animation on Shrek was based not only on cost (which indeed played a big part in the decision), but also on the desire to achieve a certain look. Before Shrek was even in preproduction, PropellerHead Design (Rob Letterman, Jeffrey Abrams, AndyWaisler, and Loren Soman) produced a test of a similarly shaped character called “Harry” for a potential television show for HBO. The motion capture for the test was done at TSi.
For the Harry test, the propellerheads (as they were known in the Shrek production) prepared an appliance that was worn by their performer. The appliance would add the volume that the performer needed to approach the proportions of Harry. This was a new idea that we had never seen done before, and it seemed like a good one. During our first of two sessions, we placed the markers on the appliance and captured the body motions with our optical system. The tracking of the data was a challenging feat because it was an uncommon setup, but we managed to get it done with our ancient six camera system.
In the second session, we were supposed to capture Harry’s finger motion. We had never tried to capture the motion of fingers with an optical system, and we didn’t guarantee that it would work. It was strictly a developmental session. We placed the small markers that we usually used on face capture on the finger joints. The performer sat down in a chair, and we immobilized his hands as much as possible because the wrist movement had already been captured in the previous session and we knew we couldn’t capture the fingers if the wrists were moving. We captured several passes of the actor moving his fingers with the timing of the dialogue, but in the end it was Impossible to track those motions, and we had to scratch the idea of optical finger capture. Facial capture wasn’t even tested for this project.
The propellerheads finished the Harry test, filling in the additional animation, plus texture and lighting. The end result was stunning. The Harry test laid the foundation for the production of Shrek at DreamWorks. The propellerheads would coproduce the film, which would be animated using techniques similar to those in the Harry test: optical motion capture with body extensions or appliances.
The DreamWorks studio in Glendale was geared up with Silicon Graphics computers running different animation platforms and a brand new 10-camera Motion Analysis optical motion capture system. A production staff was assembled and production of a 30-s test began. Character animators started to work with the optical motion data. Several months later, the test was shown to Jeffrey Katzenberg, DreamWorks’ cofounder and the partner responsible for animation films. Shortly thereafter, the production was halted and the project went back into development. The propellerheads were no longer involved. Exactly what happened is still a mystery, but motion capture was ruled out for this project. Among other things, the production made a mistake by using experienced character animators to work with the captured motion data, even from the stage of retargeting. It is not certain if that actually affected the test, but it would have eventually become a big problem.
Robert Zemeckis’ second directorial effort in the performance capture arena was Beowulf. His initial goal was, as with The Polar Express (see Understanding Motion Capture for Computer Animation for full text), to create stylized environments and characters, but again the project evolved into the almost-photo-real realm. “Zemeckis wants to see realistic textures, realistic aspects to the environment and to the characters,” recalls Jerome Chen. “He doesn’t gravitate toward the stylized designs; he likes it to be realistic. . . . Once again you are starting out trying to be stylized and ending up wanting to see more detail in the characters once you see realistic motion on them.”
Beowulf’s pipeline was almost identical to the one used for The Polar Express. The main difference was in the data acquisition. Imageworks upgraded to Vicon’s latest hardware included cameras with much higher resolution and the ability to accommodate many more of them, over 250 to be more specific. The volume was bigger so all the capture was done in a single stage. Also, the crew had a lot more experience from having done The Polar Express and Monster House. “A lot of the protocols established in the stage capture were now perfected,” says Chen. “We budgeted more time in Beowulf to do facial animation, particularly preserving volume in the face and on the lips.” The facial system in Beowulf was different from the one used on The Polar Express. It was mostly driven by FACS (Facial Action Coding System), where actors would have a number of predetermined expressions and an algorithm would decide the optimal combination of expressions for each frame. They also paid a lot of extra attention to the eyes.
The Wheels process, now called Camera Layout, was still a major piece in the postproduction process as all the shots were designed there. For Zemeckis’ style of filmmaking, that process worked very well because he is a big fan of complex camera work. Camera Layout allowed him to experiment and tweak the camera angles as many times as he wanted.
Alberto Menache is a pioneer in the field of motion capture technology. He founded one of the first motion capture companies – Three Space Imagery, Inc., that developed proprietary software for motion capture animation. The rights to the software were sold to a leading motion capture company and it continues to be an integral part of the motion capture industry today. Mr. Menache is currently the CEO of Menache LLC, a company that develops wireless technologies for motion capture. Mr. Menache was a Digital Effects Supervisor at Sony Pictures’ Imageworks. He was a primary contributor to many of the company’s visual effects productions, including such blockbuster films as “Spiderman,” “The Polar Express,” and “Superman Returns.” In addition to having primary responsibility for digital effects and motion capture efforts, Mr. Menache designed and led implementation of the company’s proprietary muscular deformation and facial animation systems. Alberto holds a bachelor’s degree in mechanical engineering and a master’s degree in mathematics and computer science.