Small language models matched frontier model performance in 4 of 5 home automation tasks
A new evaluation of 9 models reveals that using massive, expensive AI is often unnecessary for managing smart home devices.
Large language models often handle everything in an AI agent, from planning tasks to writing code. A study of the Wactorz home-automation framework tested 9 models, ranging from small 0.8B parameter versions to large, hosted frontier models, to see if this one-size-fits-all approach is truly efficient.
Testing across 2520 calls in two real-world Home Assistant installations showed that bigger is not always better. While frontier models excel at writing complex code, smaller, local models performed just as well at routine tasks like intent routing and device actuation.
Capability is not ordered the same way at every site, and that larger models are not uniformly better.
In fact, one 4B parameter model even outperformed its smaller 2B counterpart at grounding language to specific devices. The results show that smart home agents can effectively delegate simpler tasks to smaller, free-to-run models without sacrificing much accuracy.
- Routing intent
- Classifying actions
- Grounding language
- Planning pipelines
By routing each specific task to the most efficient model rather than relying on a single powerful one, the system maintained high performance while cutting costs significantly. In live user testing, hosting only the two most demanding tasks matched the performance of a fully hosted setup for just 28% of the cost.
However, there are risks to using smaller models for sensitive tasks. The study highlighted a safety failure where some small models incorrectly tried to control devices the system did not actually own, or refused to act entirely.
Do you really need a massive AI to turn off your bedroom lamp?
The research does not suggest that small models can replace frontier models for every use case. Code generation still remains a clear area where larger models hold a statistically significant advantage over their smaller peers.
Source: arXiv cs.AI