Trillium Labs opened to study high-stakes AI safety through public research

The new nonprofit Trillium Labs officially launched to investigate how modern AI models behave in high-stakes situations. Founders Nathan Lambert and Tom Zick aim to challenge the industry trend of keeping research findings locked inside private labs.

They believe the current practice of restricting access to powerful models limits the scientific community's ability to identify and mitigate risks. Their work will focus on post-training, which involves fine-tuning large models, and recursive self-improvement, where AI systems attempt to develop their own upgrades.

Trillium Labs plans to test how reinforcement learning influences model behavior. This method uses rewards and punishments to guide AI performance, but it can sometimes cause models to become sycophantic or act in unexpected ways.

By publishing the full details of their experiments, the team hopes outside researchers can study and replicate their findings. They want to move away from the closed, secretive environment that currently dominates the development of frontier models.

"The current closed trajectory of frontier AI development is taking us a step backwards," said Lambert.

The lab has secured initial funding from organizations like Schmidt Sciences and Halcyon Futures. The founders intend to raise between $40 million and $100 million, with $30 million dedicated to training costs over the next 18 months. Have you ever wondered if open scientific transparency could actually make these powerful systems safer?

The research does not yet address all types of AI architectures, as the lab is currently in its early stages of operation. The founders have not yet released results from their specific experiments, and the long-term impact of this open-research approach on industry safety standards remains to be seen.