AudioCraft (audiocraft.metademolab.com)

AudioCraft is a code base from Meta AI research for generative audio, covering music generation, sound effect generation and audio compression after training on raw audio signals. It brings together three models: MusicGen for text-to-music generation, AudioGen for text-to-sound generation, and the EnCodec neural audio codec. AudioCraft is published as research, with code, papers and model cards for people building or studying generative audio models.

EnCodec maps an audio waveform into one or several parallel streams of discrete tokens. MusicGen and AudioGen each use a single autoregressive language model to model these tokens, with a token interleaving pattern that handles the parallel streams inside one model. The generated tokens are passed to the EnCodec decoder to produce the output waveform, and conditioning models, such as a pretrained text encoder, steer what is generated.

MusicGen generates mono and stereo music from a text description and can also follow a melody extracted from a sample through chromagrams. It is trained on 30-second chunks of audio but can create longer tracks, such as 2 minute-long pieces, by sliding a window and keeping earlier output as context. A Multi-Band Diffusion EnCodec decoder reduces audio artifacts at the cost of extra computation.

AudioGen generates environmental sounds from a text description of an acoustic scene, producing 5 seconds of audio per prompt with realistic recording conditions, such as whistling wind, a passing train or typing on a typewriter. AudioCraft provides sample outputs for both models, along with links to the code, the research papers, technical details, model cards and the Meta AI blog.

Use cases

• Generating music in a chosen style or genre from a text prompt
• Creating environmental sound effects from a description of an acoustic scene
• Making music that follows the melody of an existing sample
• Producing tracks longer than the 30-second training chunks
• Compressing audio into discrete tokens with a neural codec
• Research into simpler designs for generative audio models