Multimodal AI models need different data mixtures to learn audio, vision and text

New scaling laws for training AI models provide a roadmap for balancing text, vision, and audio data. The research suggests that not all data types benefit from the same training recipe.

Researchers tested models ranging from 457 million to 8.3 billion parameters using Mixture-of-Experts architectures. They found that compute settings must be tailored to each modality.

Audio is model-heavy. It benefits more when you increase the model size rather than adding more tokens. Conversely, text and vision tasks improve best when both data quantity and model size grow together.

There is also a limit to the value of training data. Repeating vision or audio samples beyond a double-pass provides almost no performance boost for the model.

Interference between different types of data is also a major factor. The study found that vision data strongly interferes with audio learning. However, audio data does not significantly disrupt the learning process for text or vision.

These findings could save developers significant computing costs by optimizing how they mix their training datasets. Do you think this will lead to more efficient AI assistants in the near future?

It is important to note that this study focused specifically on Mixture-of-Experts architectures. The researchers have not yet confirmed if these exact scaling laws apply to other model types, such as dense transformer models.