Voicebox by Meta

Voicebox is a generative AI model for speech that can generalize to tasks it was not specifically trained for with state-of-the-art performance. Unlike existing speech synthesizers, it can be trained on diverse, unstructured data without requiring carefully labeled inputs.

Voicebox uses a new approach called Flow Matching, which is a Meta's latest advancement on non-autoregressive generative models that can learn highly non-deterministic mapping between text and speech.

Voicebox can produce high-quality audio clips in a vast variety of styles and can synthesize speech across six languages, as well as perform noise removal, content editing, style conversion, and diverse sample generation.

One of the main advantages of Voicebox is its ability to modify any part of a given sample, not just the end of an audio clip it is given. This makes it highly versatile and suitable for tasks such as in-context text-to-speech synthesis, cross-lingual style transfer, speech denoising and editing, and diverse speech sampling.

Additionally, Voicebox outperforms existing state-of-the-art speech models on word error rate and audio similarity metrics. While Voicebox is not currently available to the public due to potential risks of misuse, Meta has shared audio samples and a research paper detailing its approach and results.

This breakthrough in generative AI for speech is exciting as it has potential applications in helping people communicate and customize voices for virtual assistants.