This AI tool can make actors deliver dialogues in different languages

LOS ANGELES: Bad lip-syncing in dubbing and subtitles can put off audiences and hurt box office takings of foreign films. AI may be about to change all that.

Start-up Flawless AI, co-founded by film director Scott Mann, has a tool that it says can accurately recreate lip sync in dubbing without altering the performance of the actors.

https://twitter.com/Flawlessai/status/1389228583541952515?s=20

The tool studies how actors move their mouths and swaps the movements out according to the dubbed words in different languages, making it seem like Tom Hanks can speak Japanese or Jack Nicholson is fluent in French.

Mann was inspired to come up with the tool when he saw how dubbing affected the narrative cohesion of his 2015 film “Heist”, starring Robert De Niro.

“I hate dubbing as it stands,” he told Reuters. “You have to change so many things to try and catch sync. You’re changing words filmmakers and performers have thought about so deeply. They’re thrown out to find a different word that fits, but it never really does.”

Mann decided to do something about this conspicuous mismatch. After some research, he discovered a white paper by Christian Theobalt of the Max Planck Institute for Informatics, laying out a new approach that uses artificial intelligence to recreate photo-real human visual effects.

The tool worked to translate a De Niro scene in “Heist”.

“It’s able to essentially take an ‘ooh’ sound from De Niro 20 minutes earlier and place that in a different moment in the film,” explained Mann. “It measures at the same time and blends it so that the performance is the same but it’s a different mouth movement.”

Flawless AI is now working with producers and studios to integrate the technology into post-production. The first films using this tool are around a year away.

Mann believes this technology will be a game-changer, helping to eradicate foreign-language categories at award shows and creating a more inclusive film industry with a real global stage.

“From a filmmaking point of view, you’re going to see the rise of a much more diverse range of stars,” he said.

There might be a dissonance and a focus on the AI technology in the beginning when it is introduced, but eventually Mann hopes its use will be seamless.

“When it’s a common thing, we’re not going to think about it anymore and we’ll just enjoy the movies,” he said.