AI for lip dubbing could change the film industry; know how

In Hollywood, a lot of money is being lost in translation.

Even if companies invest in quality script translations and better performances from voice actors, dubbed entertainment often still looks as cheesy as old kung fu movies. No matter how good the sound is, it feels wrong. Lips don't lie. 

“The lips are always, always the last piece that no one solves,” says Jonathan Bronfman, co-founder and CEO of visual effects company Monsters Aliens Robots Zombies (MARZ).

How AI for lip dubbing could change the film industry https://t.co/Db0XAqF52G

— Fast Company (@FastCompany) November 19, 2023

LipDub AI

Earlier this year, Bronfman's company introduced a technology called LipDub AI, which digitally manipulates actors' facial expressions to match words spoken in foreign languages. The technology promeachieve an extraordinary level of realism and fluency, learning how to make the actors' lips match the language and the interpreters.

O LipDub AI combines audio analysis with video data to achieve precise lip syncing. The result is a video where lip movements match the words spoken on the audio track, providing greater realism in applications such as film dubbing, video conferencing and character animations.

Adapting this technology into a valuable product for the film and advertising industries presented researchers with complex challenges. The various elements of film production, such as changes in lighting and camera angles, along with scenes with multiple actors or multiple faces, required special care.

A significant obstacle arose when the AI ​​initially failed to differentiate between speakers and non-speakers. This resulted in scenes where each character's lips moved in sync with a single spoken line.

“In the beginning, we had to put black boxes over the faces we didn’t want to talk about,” says Matt Panousis, MARZ co-founder and chief operating officer. “It’s one thing to do this in a simple music video. Another is to upload an entire movie.” 

The future of AI in Hollywood is still being determined. The SAG-AFTRA union and Hollywood studios are discussing the role of technology in productions – and the need for explicit consent from actors will complicate the agreements needed to make it happen. Lipdub AI and other technologies can be used.

Hollywood actors' strike ends with agreement to 'protect members from AI threat'

After 118 days, the longest actor strike in Hollywood history has come to an end with the ratification of a new billion-dollar deal that includes new protections against artificial intelligence.

If the LipDub AI and similar technologies thrive, it could expand the reach of domestic and foreign films, benefiting creators around the world.

Read also

Amazon uses AI to ensure authentic customer reviews; know how

Amazon's artificial intelligence (AI) has the ability to identify irregular patterns in data that may suggest that a specific product review is false. Know more!

* The text of this article was partially generated by artificial intelligence tools, state-of-the-art language models that assist in the preparation, review, translation and summarization of texts. Text entries were created by the Curto News and responses from AI tools were used to improve the final content.
It is important to highlight that AI tools are just tools, and the final responsibility for the published content lies with the Curto News. By using these tools responsibly and ethically, our objective is to expand communication possibilities and democratize access to quality information. 🤖