Hugging Face released Olmo-core 3 to help scale large mixture-of-experts models

Hugging Face released Olmo-core 3, a new infrastructure designed to make training large language models more efficient. The system focuses on mixture-of-experts (MoE) architectures, which use specialized model parts to process information rather than activating the entire model for every request.

Imagine a large library where only the relevant experts are called upon to answer a specific question. This method saves energy and compute time but creates complex coordination challenges. Olmo-core 3 addresses these by keeping expert components resident on individual GPUs, avoiding the need to constantly gather and move model weights between hardware.

The framework introduces several techniques for managing these massive models, including expert parallelism, pipeline parallelism, and distributed optimizers. By splitting the work across hardware effectively, the system allows for training models with over one trillion parameters.

Tests on B300 hardware showed significant performance gains. The new stack achieved 2.7 times the throughput of the previous version. It also incorporates MXFP8, a low-precision format that reduces memory usage while maintaining training speed.

Could this approach eventually make the most powerful models accessible to smaller research labs? While the results are promising, the team noted that some techniques, like overlapping communication and computation, did not always improve end-to-end performance in their tests. Furthermore, some of their large-scale benchmarks used random routing to measure system throughput, which does not necessarily reflect the quality of a fully trained model.