Read My Lips: Researchers Develop New Automated Film Dubbing Technique

Posted April 20th, 2015 at 7:00 pm (UTC-5)
Comments are closed

A series of facial movements that are used when pronouncing phrases listed on the right (Disney Research

A series of facial movements that are used when pronouncing phrases listed on the right (Disney Research)

Sixteen days after its April 3, 2015 release, the new American action film “Furious 7” made a whopping $858.3 million in international markets, compared to a more modest $294.4 million in North America. Movie and television programs companies have taken notice and are aggressively marketing their products to a wide international audience.

But reaching an international audience means the film’s dialog must often be dubbed by actors speaking local languages.

Current methods of dubbing dialogue to match the on screen facial movements of the person talking as closely as possible often come across as terribly disjointed. That makes for an unpleasant movie viewing experience for the audience.

Given such a lucrative international market, filmmakers are taking extraordinary steps to ensure that the translated version’s sound matches the facial movements of onscreen actors as closely to as possible.

Disney Research, Pittsburgh and the United Kingdom’s University of East Anglia have conducted studies that they said will help in the development an automated dialogue re-dubbing system that will make movies more enjoyable for people who speak the languages spoken by international audiences.

This is a movie dubbing studio where dialogue is translated and revoiced into other languages and then dubbed into movies set for international release. (arceus555 via Creative Commons)

This is a movie dubbing studio where dialogue is translated and revoiced into other languages and then dubbed into movies set for international release. (arceus555 via Creative Commons)

The new system, developed by a team led by Sarah Taylor at Disney Research, Pittsburgh, automatically analyzes the on screen actor’s speech. It then allows film producers to reduce or in some cases eliminate even the most subtle differences between words spoken on screen and what the audience hears.

The system is based on something called “dynamic visemes,” which are facial movements that are connected with certain sounds produced in speech.

“The method using dynamic visemes produces many more plausible alternative word sequences that are perceivably better than those produced using a static viseme approaches,” Taylor said in a press release.

The system will provide filmmakers with a wider variety of word sequences that match facial movements.  This will allow producers to write local language dialogue that not only corresponds with the movie’s script, but also ensures that on-screen facial movements are more in synchronization with what the audience hears.

As an example, the researchers found that when an actor says a phrase like “clean swatches,” his facial movements are the same as those for other phrases, such as “likes swats,” “then swine,” or “need no pots.”

Taylor and her research team will present their findings on April 23 at the 40th International Conference on Acoustics, Speech and Signal Processing (ICASSP), that’s being held from April 19-24, 2015, in Brisbane, Australia.

How the new technique will affect international revenues remains unclear. While Furious 7 has performed well at the international box office so far, it still needs to earn at least another $1.17 to beat out James Cameron’s 2009 blockbuster Avatar – the all-time international money-making film that has earned nearly $2.03 in international receipts.


Video demonstrates new dubbing method developed by Disney Research (Disney Research)

Rick Pantaleo
Rick Pantaleo maintains the Science World blog and writes stories for VOA’s web and radio on a variety of science, technology and health topics. He also occasionally appears on various VOA programs to talk about the latest scientific news. Rick joined VOA in 1992 after a 20 year career in commercial broadcasting.

Comments are closed.