What Is the Relationship Between Tokenization and Music Composition in AI Workflows?
Music tokenization explained for producers: MIDI-style REMI tokens, codec tokens, how AI composes by prediction, and why extensions drift in tempo or key.
Music tokenization turns music into a sequence of discrete symbols, and that makes composition in AI workflows a prediction task: the model writes a piece one token at a time, choosing each next note event or audio code from everything before it plus your prompt. Which tokens a system uses decides what you can control.
Two kinds of tokens are in use. Symbolic tokens describe notes, like a MIDI file. Audio tokens come from neural codecs that compress the sound itself. Below is how each works, how language-model-style generators compose with them, and what that means when you extend, inpaint, split or steer a track. Everything here comes from the original papers and code, read on 14 November 2026.
What is music tokenization?
A token is one entry from a fixed vocabulary, stored as a number. A transformer reads and writes nothing else, the same way a text model works with word pieces. To compose with one, music first has to be cut into pieces like that. There are two ways to cut it, and they lead to very different tools.
Symbolic tokens: MIDI-like events and REMI
The first approach writes music down as events. The older "MIDI-like" scheme uses Note-On and Note-Off tokens for each pitch, plus Time-Shift tokens that move the clock forward by 10 to 1,000 milliseconds.
REMI, from Yu-Siang Huang and Yi-Hsuan Yang's Pop Music Transformer paper (2020), adds the structure musicians read by:
- Bar marks the start of each bar.
- Position places an event in one of 16 slots within the bar.
- Tempo covers 30 to 209 BPM, and Chord covers 60 chord types.
- Note-On gives the pitch (MIDI 0 to 127), with Velocity for dynamics and Duration in 32nd-note steps.
So one note becomes a short run of tokens: where it falls in the bar, how hard it's played, which pitch and how long. The authors added Bar and Position because timing built from Time-Shift tokens alone kept slipping. In their words: "When listening to the music generated, the intended bar lines drift over time and the rhythm feels unstable."
For a producer, symbolic output is easy to edit. It drops into a piano roll with an exact tempo and key, but it carries no sound. You still choose the instruments, play or program the parts and produce the record.
Audio tokens: SoundStream, EnCodec and DAC
The second approach tokenizes the recording itself, with a neural audio codec. An encoder squeezes the audio into a stream of frames, a residual vector quantizer (RVQ) turns each frame into several codes from learned codebooks, and a decoder rebuilds the waveform from those codes.
- SoundStream (2021) pairs a convolutional encoder and decoder with a residual vector quantizer, and one model covers speech, music and general audio at 3 to 18 kbps.
- EnCodec (2022) is the tokenizer behind Meta's MusicGen. The MusicGen paper uses it on 32 kHz audio at 50 frames a second, with four codebooks of 2,048 entries each. That's 200 tokens for every second of music.
- DAC (Descript, 2023) compresses 44.1 kHz audio about 90 times, into tokens at 8 kbps.
In RVQ, each codebook encodes what the previous one missed, so the first carries the rough picture and the later ones add detail. MusicLM (Google, 2023) stacks two token types: 25 semantic tokens a second for long-range structure, and 600 SoundStream acoustic tokens a second for the sound.
Audio tokens capture what symbolic ones can't: the timbre of a voice, the room, the mix. They have no field for a note, a bar line, a BPM or a key.

How language-model music generators compose by predicting tokens
Composition in these systems is a loop:
- Encode the prompt. Text becomes an embedding. MusicGen and MusicLM can also take a melody.
- Predict the next token. The transformer scores every token in the vocabulary, one is sampled, it's appended to the sequence, and the loop repeats. MusicGen's code samples from the 250 most likely tokens by default.
- Decode. The codec's decoder turns the finished token sequence back into audio.
MusicGen predicts its four codebooks with a "delay" pattern in a single transformer, which turns 30 seconds of audio into 1,500 prediction steps. MusicLM works in stages instead: semantic tokens first to lay out the structure, then acoustic tokens to fill in the sound.
Sampling explains a workflow habit you already have. The same prompt gives a different song each time because each token is a weighted dice roll, so making several takes and picking one is how these systems are meant to be used.
Commercial tools rarely publish how they work, so treat the points below as how the published token models behave. Our post on AI music models versus AI music platforms explains why the product around a model matters as much as the model.
What music tokenization means in a real workflow
Continuation and extension
Extending a track is the most natural job for a token model. Your audio is encoded, its tokens become the prompt, and the model keeps predicting. Meta's AudioCraft library has a generate_continuation function for exactly this.
The catch is the window. MusicGen was trained on 30-second clips, and its code generates up to 30 seconds per pass. Past that, the code slides forward 18 seconds at a time by default, so each new chunk sees only the last 12 seconds. The comment on that setting in MusicGen's source code states the trade-off: "Larger values will mean less context is preserved, and shorter value will require extra computations."
In practice, extend from a passage that represents the song, such as the end of a chorus rather than a sparse breakdown, and listen at every join. Our guide to extending a song with AI has more on that.
Inpainting a section
A left-to-right model only sees what came before, so it can't make a new bridge lead into the chorus you already have. Inpainting needs context on both sides. VampNet (2023) masks codec tokens and fills them in with a bidirectional transformer, which lets one model handle inpainting, outpainting, continuation and looping with variation.
Start and end your selection on phrase boundaries, leave real music on both sides, and generate a few takes. AI music inpainting explained goes deeper.
Stems
Codec tokens describe the whole mix, not separate parts. The StemGen authors note that most models focus on generating fully mixed music, and propose one that listens and responds to the music already there. So stems from a full-mix generator come from a separation model run afterwards, with some bleed, or from a part generator. Symbolic schemes can give each instrument its own note events, so multi-track output comes naturally.
Length and context limits
Tokens add up. At MusicGen's 50 frames a second, a three-minute song is 9,000 frames, six times the 1,500 steps of a 30-second pass. MusicLM's acoustic stream would need 108,000 tokens for the same three minutes. Transformers can only attend to so much at once, which is why systems work in windows or hierarchies. MusicLM leans on its lighter semantic tokens for long-range coherence, and reports output that stays consistent over several minutes. Build long songs in sections and plan the form yourself.
Why tempo and key can drift
Audio tokens have no BPM or key field. The model infers both from the tokens in its window, so once the opening bars slide out of view, nothing pins the original tempo or key, and small errors compound. REMI's authors hit the symbolic version of the same problem with Time-Shift tokens.
Keep extensions short, check tempo and key after each one, and fix small drift with time-stretching or a fresh take.
Why a MIDI or melody input gives more control
A text prompt describes. A melody constrains. MusicGen's melody model reads a chromagram of your audio, keeping only the strongest pitch class in each time step, alongside the text. MusicLM can transform a whistled or hummed melody into the style of a caption.
That condition lines up with time, so at every frame the model has a target instead of a guess, and the tune, phrasing and much of the tempo come from you. With symbolic tokens you go further and edit the exact notes. Export your topline as MIDI, fix wrong notes first, and let the generator handle the sound.
Where Soundverse fits
Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music. This explainer describes published research models, not how Soundverse's own models work. The workflows map onto its tools:
- Extend Music adds up to 85 seconds of new instrumental per run, after the end or before the start, and works on the Free plan.
- Inpainting, on Creator and above, rewrites a section you select, up to about a minute at a time on the default model, and keeps the rest.
- MIDI to Song and Melody to Song take 5 to 60 seconds of melody, as a .mid file or a hummed recording, plus your lyrics, and build a sung song that follows your tune. Soundverse v9 is available on Pro and Max.
Licensing is set by plan: the Free plan is for personal use, and the Creator plan comes with a royalty-free license.
This walkthrough shows Inpainting from 0:12, Extend Music from 0:36 and Stem Separator from 1:28.
FAQ
What is music tokenization in simple terms? It's how music is cut into numbered pieces a model can read and predict. Symbolic systems use note events such as pitch, position and duration. Audio systems use codes from a neural codec that stand for short slices of sound.
Is REMI the same as MIDI? No. REMI is built from MIDI data, but it adds Bar, Position, Tempo and Chord tokens so a model sees the beat and the harmony directly. Its authors report better rhythmic structure than earlier Transformer models built on MIDI-like events.
Why does an AI-extended song change tempo or key? Audio-token models don't store tempo or key; they infer them from recent context. When a long extension pushes the opening out of the model's window, small slips compound. Shorter extensions and a melody or MIDI input help.
Can an audio-token model give me MIDI? Not directly. Its tokens are compressed sound, not notes. To get MIDI from a generated track, run it through an audio-to-MIDI transcription tool and clean up the result.
Sign up for Soundverse free and try extending, inpainting or turning a melody into a song. More guides on the Soundverse blog.