What Are the Main Technical Hurdles in Generating Long-Form Coherent Musical Compositions Using AI?

Long-form AI music explained for producers: context windows, compute cost, song structure, drift, hierarchical and diffusion models, and extension seams.

Long-form AI music runs into five technical hurdles: a full song is an enormous sequence, memory and compute grow fast with length, structure such as a returning chorus spans minutes, small errors compound into drift in sound or timing, and there is still no reliable automatic score for coherence. Researchers answer with compression, hierarchy and structure conditioning.

Below is each hurdle, the research behind it and what it means for a full track. The arXiv papers cited were read on 19 November 2026; commercial tools rarely publish their methods, so this describes published models.

Context windows: a song is a very long sequence

Raw audio is dense. OpenAI's Jukebox paper (2020) notes that four minutes of CD-quality audio is an input of about 10 million samples, so every system compresses first.

Jukebox squeezed audio into three levels of discrete codes and gave each Transformer a context of 8,192 codes. That still covered only about 24 seconds at the most compressed level, 6 seconds in the middle and 1.5 seconds at the bottom. The YuE team (2025) report that 8,192 tokens hold about 163 seconds of mixed music in their setup, or 81 seconds when vocals and accompaniment are separate tracks, so they extended training to 16,384 tokens.

Anything outside the window doesn't exist for the model. Our explainer on tokenization in AI music covers token rates in other models, and our guide to the technical factors behind AI audio quality covers what short training windows sound like in a finished track.

Memory and compute: why longer costs more than linearly

Self-attention compares every position with every other one. In the Music Transformer paper (Google, 2018), Cheng-Zhi Anna Huang and colleagues note that its time and space cost grows quadratically with sequence length: double the length and that cost roughly quadruples. Their memory-efficient relative attention cut memory per layer from 8.5 GB to 4.2 MB for a 2,048-step sequence, enough for minute-long compositions.

Time hurts too: Jukebox took around an hour to generate one minute of top-level codes, then around eight hours to upsample it.

Stability AI's Stable Audio 2 long-form paper (Zach Evans and colleagues, 2024) shortens the sequence instead: its diffusion transformer works on a latent at 21.5 frames a second, with block-wise attention and gradient checkpointing the authors call crucial. They chose 4 minutes 45 seconds as the target because it fits in modern GPU memory and covers over 90% of their dataset.

Structure: repetition, returning motifs and development

The Music Transformer authors frame the core problem: music builds meaning through repetition on several timescales, from motifs and phrases to whole sections. A model has to bring back a chorus it wrote two minutes earlier, vary it enough to feel like development, and know whether it's in the first verse or the last chorus.

Jukebox shows the failure mode. Its samples held together within the top-level window and kept similar harmonies as the window slid on, but in the authors' words, "because the top-level does not have the context of the entire song, we do not hear long term musical patterns, and we would never hear choruses or melodies that repeat."

Training on whole songs helps. Evans and colleagues saw good structure only once they trained on the full 4 minutes 45 seconds, and their self-similarity plots show generations repeating motifs introduced in the first section. By contrast, the SongGen authors (2025) list as a limitation that their model stops at 30 seconds, too short for a complete song structure.

Drift in timbre, tempo and key over minutes

Drift is the slow loss of what the song started as. The papers point to three causes:

  • Conditioning fades. YuE's authors found that text conditioning at the start of the sequence weakens as audio tokens pile up: degradation began around 3,000 tokens and became complete failure beyond 6,000, even after pre-training on 16,000-token contexts. They put it down to the long-term decay of the RoPE position encoding.
  • Timing slips against the words. Without their fix, models tended to open with long instrumental preludes, so the singing drifted away from the lyrics. YuE also dropped a BPM control because it was tied to lyric length and hurt lyric following.
  • Sliding windows forget. Once the opening leaves the window, nothing pulls the timbre, groove or key back toward it, and small slips compound.

The YuE team sum up the lesson: "Simply scaling up the LM context length does not yield effective song-level lyrics-following capabilities and demands substantial computational resources."

How hierarchical and latent-diffusion models handle long songs

Hierarchical token models plan coarsely, then add detail. Jukebox's top-level model writes the most compressed codes, and upsamplers fill in the levels below. YuE's first model predicts only the first codebook of its tokenizer, which carries melody and vocal content, with separate vocal and accompaniment tokens in each frame; a second model adds the remaining codebooks. The split tracks help in dense genres such as metal, where YuE's authors found lyrics hardest to keep intelligible.

Latent diffusion models generate the whole song at once. Stable Audio 2's diffusion transformer denoises the full latent of up to 4 minutes 45 seconds together, so the ending is shaped with the start in view, and the paper reports 13 seconds of generation on a GPU. DiffRhythm (2025) does the same for songs with vocals: up to 4 minutes 45 seconds of 44.1 kHz stereo in about ten seconds. It trained on about 95 seconds of latent first, then fine-tuned at the full length.

The trade-off is a fixed ceiling. Stable Audio 2 makes shorter pieces by filling the rest of its window with silence, while a token model can keep going and pays for it in drift.

Diagram comparing a sliding-window token model with seams, a hierarchical model that plans coarse codes then detail, and a latent diffusion model that generates the whole song at once

Conditioning on structure and lyrics with timing

Telling a model where it is in the song is one of the most effective fixes:

  • A timing signal. Jukebox fed in the total length, the segment's start time and the fraction elapsed, which let it produce intros and endings, applause included. Stable Audio 2 conditions on timing too.
  • Section labels. YuE's structural progressive conditioning splits each training song into labelled sections such as [verse] and [chorus], most under 30 seconds and 14 per song on average, pairs each label and its lyrics with that section's audio, and generates section by section.
  • Lyric timing. DiffRhythm needs only the start time of each lyric line and places that line's phonemes there. JAM (2025) adds word-level timing and duration control.

Extension and continuation: where the seams come from

When a model can't make a whole song in one pass, it continues. Jukebox's windowed sampling moved ahead by half its context and carried on from the remaining half; a smaller step gave better quality but took longer. At every join, the new audio sees only the overlap.

That causes two problems. A seam can sound like an edit, a jump in energy, mix or tone. And continuation can copy: YuE's authors warn that tight conditioning on reference audio can make a model replicate the reference melody or whole segments, and it weakens style prompts. They also found that using a chorus as the reference made output more musical and stable.

How researchers measure coherence in long-form AI music

There is no single coherence score. YuE's team found the widely used CLAP score diverged from human judgments, and checked lyric alignment by hand because Whisper transcription wasn't robust enough on singing. Researchers rely on:

  • Self-similarity matrices. Evans and colleagues plot which moments of a song resemble which others; repeated sections show up as blocks, compared against real music in the same genre.
  • Listening tests with a structure score. Their listeners rated audio quality, text alignment, musical structure, musicality and stereo correctness, and structure ratings varied widely between listeners.
  • Full-song benchmarks. SongEval (2025) holds 2,399 full-length songs, each rated by four trained annotators on a five-point scale for overall coherence, memorability, naturalness of vocal breathing and phrasing, clarity of song structure and overall musicality. Models trained on it predicted human ratings better than existing objective metrics.

What this means when you make a full-length track

  1. Generate at full length when the model allows it; stitched passes add seams.
  2. Write the structure into the lyrics with section tags. Our prompting playbook for section tags on Soundverse v9 has a cheat sheet.
  3. Check the form you got against the form you asked for.
  4. Fix one weak section rather than regenerating the song.
  5. Extend from a passage that represents the song, such as a chorus, and listen across every join.
  6. Check tempo and key after each extension.

Where Soundverse fits

Soundverse is an AI music platform that takes you from idea to audience, built for independent artists who release music. Soundverse v9, on Pro and Max, makes full songs with no length setting: the model decides how long each song runs, and it can't extend a song.

To go longer, Extend Music adds up to 85 seconds of new instrumental per run, after the end or before the start, and returns the whole track, while Extend Song continues a vocal song from lyrics you write on Soundverse v8, v7 or v6. Each run is a new take, so listen across the join. Inpainting, on Creator and above, rewrites up to about a minute at a time on the default model and keeps the rest. Section Analysis labels intro, verse, chorus, bridge and outro with start and end times, plus tempo and downbeats, so you can see whether a generated song has the bridge you asked for.

Soundverse's Edit walkthrough shows Inpainting from 0:12 and Extend Music from 0:36.

FAQ

Why do AI songs lose the plot after a minute or two? Most models see a limited window, and conditioning placed at the start weakens as the sequence grows; YuE's researchers measured it from about 3,000 tokens. Training on whole songs and labelling sections both help.

Is latent diffusion better than autoregressive generation for long songs? Neither wins outright. Diffusion models see the whole song at once and run fast, but have a fixed maximum length; token models can keep going, but drift and leave seams.

How long can AI-generated songs be? In the research cited here, Stable Audio 2's long-form model and DiffRhythm reach 4 minutes 45 seconds in one pass, and YuE generates up to five minutes. Anything longer needs extension, with joins to check.

Sign up for Soundverse free and build your next full-length track section by section. More guides for producers on the Soundverse blog.

Setting up your studio…Warming up Agent One…Tuning the instruments…Getting your library ready…Almost there…