Apple researchers identified fundamental limits in discrete diffusion models

Apple machine learning researchers recently published a study analyzing how discrete diffusion models create content like words or pixels. These systems generate sequences by predicting multiple tokens at once. The research explores the math behind these predictions.

The team found that these models struggle when there are hidden links between items. If one token depends on another, generating them based only on individual position distributions often fails to match the original training data.

To test this, the authors used a synthetic task called ScanAndAdd. They confirmed that whenever a group of two or more unknown positions is determined, there is an underlying dependency that standard per-position distributions cannot easily resolve.

Even when different models appear identical in their individual per-position statistics, their overall output can differ significantly because of how they handle combinations of values. The researchers measured the generated distribution and found it was 29 times higher than the noise floor.

Think of it like trying to solve a puzzle where the pieces only fit correctly if you account for the entire image, rather than just looking at the color of each square individually.

This study highlights a mathematical gap in current sampling methods. The researchers note that their findings specifically apply to current discrete diffusion approaches, but they did not test every possible architectural variation or future optimization technique for these models.