How MARZ became Hollywood’s go-to star for AI-powered visual effects
Monsters Aliens Robots Zombies is No. 7 on Fast Company’s list of the Most Innovative Companies in film and television. Explore the full list of the World’s 50 Most Innovative Companies of 2023.
The team working on last year’s blockbuster Marvel-Sony movie Spider-man: No Way Home faced a serious dilemma as they geared up for production: How do you de-age villains so that they appear exactly as they did in previous installments of the franchise? In the case of No Way Home, both Doc Ock (Alfred Molina) and Green Goblin (Willem Dafoe) needed to appear more spry and youthful in the film as they duked it out with Peter Parker yet again.
Sony and Marvel turned to no less than a dozen digital studios to provide the VFX and AI effects for the film. One of them was MARZ, a company that is emerging as a pioneer in AI technology, leading the way as Hollywood begins to lean deeply into the fast-evolving field of artificial intelligence applications as a way to push the boundaries of storytelling.
A number of projects are in the works that are using AI to transform actors’ appearances in miraculous-seeming ways. In the upcoming Miramax film Here, directed by Robert Zemeckis, Tom Hanks will appear as a man in nearly every stage of his life thanks to AI technology provided by Metaphysic (the company behind the deepfake Tom Cruise videos on TikTok). And in the upcoming Indiana Jones and the Dial of Destiny, Harrison Ford gets an AI face-lift thanks to Lucasfilm, which combed through old Indiana Jones footage to re-create a digital visage for scenes in the movie.
MARZ, which stands for Monsters Aliens Robots Zombies, isn’t interested in being a onetime solution in Hollywood. The company has used its proprietary AI technology on 27 projects so far, including Stranger Things, Jack Ryan, Being the Ricardos, Umbrella Academy, and No Way Home, where it focused its energies on Willem Dafoe. So far MARZ’s AI de-aging technology, called Vanity AI, has been offered as a service, but later this year the company plans to allow businesses and creatives to download the Vanity software for free and pay MARZ a fee based on the number of shots that are processed.
“At that point it’ll be, ‘You guys do what you want with the software,’” says Matt Panousis, MARZ’s cofounder and COO, who describes Vanity as “a super high-end Snapchat filter for Hollywood.”
Vanity works by first “learning” the face of an actor or actress. “You literally upload five images of an actor; you tag it. Those five images teach the system who the actor is,” Panousis says.
“Once you’ve created that mask, Vanity is going to comb through every shot in your movie and find the actor and is actually going to stick the mask on the face automatically,” Panousis says. “So if you’re an artist going in to work on a shot, all you’re actually doing now is confirming that the mask is (fitted) correctly. They can tweak it if it hasn’t done a perfect job. After that, they’re just moving dials.”
This speed and efficiency is in stark contrast to how artists typically de-age a face using VFX effects, which involves working by hand, shot by shot, and correcting facial details in a process known as “tracking.” Panousis says this can take 16 hours to complete five seconds of film. “You have to constantly figure out for each frame, What regions am I working on and what am I doing to make the person look younger? It’s a creative endeavor.”
As well as an incredibly time-consuming and expensive one.
Panousis says that using Vanity on a project as a service cuts costs by about 50%. “You can usually expect to spend at least $1 million and up to $5 million” on a de-aging project. “We’d like to come in and be able to do that same work for $500,000 max. Hopefully, $400,000.”
MARZ was founded in 2018 by Panousis, film producer Jonathan Bronfman, and VFX veteran Lon Molnar as a VFX company specially aimed at providing feature-film-level special effects for TV shows. Panousis says the trio had the realization that as TV production was skyrocketing and more shows were on the air or being streamed than ever before, there weren’t enough VFX artists to meet the demand. They also realized that special effects in TV typically suffered because studios didn’t have the same timeline to work on as they did in film, or the huge budgets.
In 2019 the company began experimenting with AI, seeing it as the vehicle not only to get high-quality results but also to create a faster, cheaper solution to traditional VFX. Hence, the birth of Vanity. (Though MARZ still does VFX projects, including work on Thing, the sidekick hand in Wednesday.) At this point Vanity mainly focuses on “textures,” Panousis says, as opposed to completely replacing an actor’s face.
“The second phase of Vanity is going to include the ability to make structural changes,” he says. “But 90% of the time, all folks really want is 15 years” taken off or added to a face. “Some companies are working on projects where they’re taking someone back by 30 years and you need a face replacement. But that’s not a common scenario. The common scenario is, ‘We need a little de-aging.’ The other common scenario is, ‘We just want this person to look better.’”
Indeed, some of MARZ’s work with Vanity has been more akin to airbrushing—removing acne from a teenager’s face, for example—which opens up interesting questions for Hollywood, not to mention a future in which every single face on a screen may appear surreally flawless. And why hire a younger actor if Brad Pitt can look as though he’s 21 thanks to AI?
MARZ’s next AI product is tackling another dilemma in the industry: How do you dub film and TV projects in a more seamless and effective way than simply laying on new audio? The company’s solution, which is still in development, is called LipDub, and uses AI to re-create the bottom portion of an actor’s face so that it synchs with the dubbed language. This means an end to clunky dub jobs where it’s painfully obvious that an actor is actually speaking Korean or Spanish and not English.
Panousis believes LipDub has even greater implications for Hollywood than Vanity. “When you hear about the media space right now, all these studios are looking for operating leverage. . . . How do we spend less and get more out of our existing content?” he muses.
“To me, it’s like, what makes more sense, spending $20 million to do some (inexpensive) production elsewhere in the world that you call a local production? And maybe it’s not mutually exclusive, but you could also say, ‘You know what? I just spent $100 million on Spider-Man. And it’s only going to cost me another $1 million to localize that properly into all these regions [using AI]. What if Spider-Man becomes a thing that gets me to up viewership as opposed to me trying to do [local productions]?’”
Panousis sees big potential for local productions that lack Hollywood budgets to create the kind of experience that viewers expect. “I think it’s going to work,” he says. “Fixing the lip and getting more leverage out of your IP—we’re going to look back and say, ‘Yeah, that was the better path.’”
Panousis says he’s hoping that by the end of the year LipDub will be used on translation jobs. As for the future of MARZ? “We’d love to look back in five, ten years and be the global leader in AI for VFX by virtue of having a suite of products that all tackle different use cases and do so with a level of automation that’s never been seen before in the industry. We feel like we’re really well positioned to do that now.”
(16)
