AudioLDM

AudioLDM is a text-to-audio generation model developed by researchers at the CVSSP centre of the University of Surrey and the Department of EEE at Imperial College London. It turns written descriptions into sound effects, human speech and music, and it can also edit existing audio through text, with zero-shot style transfer, inpainting and super-resolution. The project offers a paper on arXiv, code on GitHub and a Hugging Face Space demo.

AudioLDM builds on contrastive language-audio pretraining (CLAP). Latent diffusion models are trained on audio embeddings in a continuous space learned by a VAE, and at sampling time a text embedding serves as the condition. Because the model learns audio representations without modelling the cross-modal relationship, it can be trained on AudioCaps with a single GPU and without text supervision, which the authors credit for its generation quality and computational efficiency.

AudioLDM's demonstrations show control over the acoustic environment, the material of a space, pitch and temporal order through the prompt, for example a man speaking in a huge room versus a studio. AudioLDM also supports label-to-audio and unconditional generation, accepts prompts written by ChatGPT, and performs zero-shot audio manipulation: changing the style of a clip, filling in missing sections with inpainting, and restoring detail through super-resolution.

AudioLDM is published as a research project rather than a commercial service and has no stated price. The authors share their model evaluation code on GitHub with the aim of making audio generation models easier to compare. They are confirming a data-related copyright issue, and the pretrained models are to be released once that check is complete.

Use cases

• Synthesizing general audio and sound effects from text descriptions
• Generating ambient or other music from a written prompt
• Changing the style of an existing audio clip with a text prompt
• Filling in missing parts of a recording through audio inpainting
• Restoring detail in low-resolution audio
• Benchmarking and comparing audio generation models