In the world of digital media and animation, it is always a challenge to create virtual characters and Avatars Speak realistically to allow. This represents a major hurdle, especially for smaller studios, as affordable face capture solutions are often unable to provide a convincing lip-sync to bring about.
The tools currently available for small studios are not yet sophisticated enough to convincingly record lip and tongue movements.
Challenges and Alternatives
One of the alternatives that must be resorted to are solutions like AccuLips from Reallusion, which work retrospectively based on audio or audio files.
A well-known example of this is AccuLips from Reallusion, which is integrated with iClone and Character Creator 4.
This software can, however, only be used for English language texts unleash their full potential.
In German texts, it often fails to derive the correct lip movements.
Nvidia Omniverse Audio2Face
Here it comes Nvidia Omniverse Audio2Face into the game – a powerful tool for automatic lip-sync of digital characters.
Audio2Face uses advanced AI technologies to synchronize the lip movements of a virtual character with input audio data in real-time.
What makes this tool special is its ability to also with non-English texts to deliver convincing results.
Advantages and Integration
Integrating Audio2Face into the production process can significantly reduce the workload.
Instead of making time-consuming manual adjustments, the software allows virtual characters to speak naturally and fluently, regardless of the language of the audio files.
This opens up new possibilities for smaller studios in particular to implement their creative visions more efficiently.
Our Experience
An example from our own experience shows how Audio2Face has revolutionized our workflow.
We used to have to rely on less precise tools that often required post-production corrections, thus extending production time.
With Audio2Face, we can now ensure that our virtual characters, right from the first pass, Speak realistically, which significantly improves the quality of our projects.
Costs and Availability
Another advantage of Nvidia Omniverse Audio2Face is that the tool is available for free, as long as it is not used in an enterprise environment.
This makes it particularly attractive for smaller studios and independent creatives.
Additionally, there is a special plugin for iClone that makes the integration of Audio2Face even easier and more seamless.
What is Nvidia Omniverse?
Nvidia Omniverse is a platform for collaboration and simulation, specifically designed for creating 3D content.
It allows artists, designers, and developers to share and edit their creative projects in real-time, regardless of the program in which they were created. Omniverse includes importer and exporter modules, so-called „Connectors,“ which cover almost all common 3D tools from the CAD and design sectors. These tools include Unity, Blender, Unreal, Iclone, Substance, 3dsmax, Maya, Arcgris Cityengine, PTC Creo, Houdini, and many more.
This interconnectivity facilitates collaboration through a central, cloud-based environment.
This makes Omniverse a powerful tool for developing complex and realistic digital worlds.
The introduction of Nvidia Omniverse Audio2Face marks a significant step in the development of realistic animations for 3D avatars, even in German, and offers smaller studios a affordable and effective solution for lip-syncing.
Audio2Face is still a beta version, and we're excited to see what features Nvidia will integrate and how they will further transform the way we bring digital characters to life.
Frequently Asked Questions (FAQ)
Because the human eye immediately notices discrepancies between audio and lip movement. Many face-capture solutions capture facial expressions well, but do not hit the mouth positions precisely enough—the result then looks unnatural despite good graphics.
Because they are trained on English phonetics. Tools like Reallusion's AccuLips deliver usable results with English text, but frequently struggle with German sounds and word structures—umlauts and consonant clusters are the typical problem areas.
An AI-powered tool from Nvidia's Omniverse platform that generates facial animation directly from an audio file. Because it works based on sound rather than written text, it is largely language-independent—even with German.
It is available for free for smaller studios and non-enterprise use; license terms apply for use in enterprise environments. It should be noted that the tool was in beta for a long time and that the range of functions and terms may change.
Via the Omniverse connectors, which enable data exchange with common 3D tools such as Unity, Unreal, Blender, and Maya; a dedicated plugin also exists for iClone. The generated animation can then be further edited like any other.
Above all, time. The first version of the speech animation is usually usable right away, instead of having to be corrected frame by frame. Especially with many lines of dialogue—such as in training applications with speaking characters—this makes the difference between feasible and unaffordable.
Exclusive hardware for the trade show booth
e.g. HoloLens 2 Augmented Reality Application
