Personalized Visual Dubbing through Virtual Dubber and Full Head Reenactment
Bobae Jeon, Eric Paquette, Sudhir Mudur, and Tiberiu Popa
Eurographics 2025 - Short Papers, 2025
Visual dubbing aims to modify facial expressions to ”lip-sync” a new audio track. While person-generic talking head generation methods achieve expressive lip synchronization across arbitrary identities, they usually lack person-specific details and fail to generate high-quality results. Conversely, person-specific methods require extensive training. Our method combines the strengths of both methods by incorporating a virtual dubber, a person-generic talking head, as an intermediate representation. We then employ an autoencoder-based person-specific identity swapping network to transfer the actor identity, enabling fullhead reenactment that includes hair, face, ears, and neck. This eliminates artifacts while ensuring temporal consistency. Our quantitative and qualitative evaluation demonstrate that our method achieves a superior balance between lip-sync accuracy and realistic facial reenactment.