AI voice lines to lip-sync: TTS, then mouth animation

Run Rhubarb on the line

1. Download Rhubarb Lip Sync from its GitHub releases page.
2. Run: rhubarb -o line01.tsv line01.wav (the audio file goes last).
3. Add -d line01.txt with the line's script so recognition follows your words.

You should see: A TSV file with one timestamp and one mouth-shape letter per row.

Watch out: Without -d, recognition is less reliable, so always pass the script text.

From the official docs: Rhubarb Lip Sync README ↗

Choose the recognizer and output format

1. Keep the default recognizer (-r pocketSphinx) for English lines.
2. Use -r phonetic for other languages.
3. Set -f to tsv, xml, json or dat to match what your engine or animation tool reads.

You should see: The output file is in a format your engine or tool can import.

Watch out: The phonetic recognizer works for any language but is less precise on English, so only switch when you need to.

From the official docs: Rhubarb Lip Sync README ↗

Map mouth shapes to sprites

1. Draw the six basic mouths A to F (A is closed for P, B and M; D is wide open; F is puckered).
2. Optionally add G (F/V sounds), H (long L) and X (rest), or limit them with --extendedShapes.
3. In game, read each row and show that letter's sprite until the next timestamp.

You should see: The mouth changes in time with the audio and rests on X during pauses.

From the official docs: Rhubarb README: mouth shapes ↗

Install uLipSync

1. In Package Manager, choose Add package from git URL and enter https://github.com/hecomi/uLipSync.git#upm.
2. Or add the npm scoped registry (scope com.hecomi), or import the .unitypackage together with Burst and Mathematics.

You should see: uLipSync shows up in Package Manager and under Window > uLipSync.

Watch out: If you use the .unitypackage with VRM models, add USE_VRM0X or USE_VRM10 as a scripting define symbol yourself.

From the official docs: uLipSync README ↗

Add the components

1. Put an AudioSource and a uLipSync component on the character.
2. Add uLipSyncBlendShape for the face's SkinnedMeshRenderer and map phonemes (A, I, U, E, O, N) to blendshapes.
3. Connect uLipSync's On Lip Sync Updated event to uLipSyncBlendShape.OnLipSyncUpdate.

You should see: The face blendshapes move while the clip plays in Play mode.

From the official docs: uLipSync README ↗

Create and calibrate a Profile

1. Create a Profile and register the A, I, U, E, O vowels plus a breath phoneme.
2. Calibrate with a microphone (hold Calib while saying each vowel), or loop vowel clips with the calibration audio player.
3. Assign the Profile to the uLipSync component.

You should see: Each vowel lights up its own phoneme in the inspector.

Watch out: Profiles made in uLipSync v2 don't work in v3, so recalibrate after upgrading.

From the official docs: uLipSync README ↗

Bake to an AnimationClip

1. Open Window > uLipSync > Baked Data Generator and convert your voice clips to baked data.
2. Open Window > uLipSync > Animation Clip Generator to turn the baked data into keyframed clips.
3. Adjust Sample Frame Rate and Threshold to control how many keys are written.

You should see: AnimationClips with blendshape curves that play without real-time analysis.

Watch out: Real-time analysis is slow on WebGL builds, so bake clips there.

From the official docs: uLipSync README ↗

Prepare the project and audio

1. Use Unreal Engine 5.6 or later and enable the MetaHuman Animator plugin.
2. Import the WAV so it becomes a SoundWave asset.

You should see: A SoundWave asset in the Content Browser.

From the official docs: Epic: audio-driven animation ↗

Process the audio in a MetaHuman Performance

1. Create a MetaHuman Performance asset and open it.
2. Set Input Type to Audio, choose the SoundWave, and set the Visualization Mesh to your MetaHuman face.
3. Optionally set head movement, blinks, the process mask (full face or mouth only) and mood, then click Process.

You should see: Facial animation tracks appear in Sequencer.

Watch out: Turning on Realtime Audio removes head motion and hides the other solve options.

From the official docs: Epic: audio-driven animation ↗

Export, or drive it live

1. Use Export Animation for an Animation Sequence, or Export Level Sequence with your MetaHuman Blueprint assigned.
2. For live speech, add a MetaHuman (Audio) Live Link source and set it as your character Blueprint's Live Link subject.

You should see: An Animation Sequence that plays on the face, or lips that follow a microphone live.

Watch out: Processing in a Performance asset is offline even with the realtime solver; use the Live Link source for true real-time.

From the official docs: Epic: using a MetaHuman audio source ↗