CUGA lifted the same open model from 30% to 81% on 90 AppWorld tasks
The CUGA team swapped only the harness around GLM-5.3-flash, and results swung widely; the team admits it ran the test itself.
In brief
Same AI model, scores from 30% to 81%
The CUGA team tested three agent harnesses with one open model on 90 tasks. CUGA passed 20 scenarios on the strict score, Hermes 8, Deep Agents 2. CUGA's own team ran it.
A rule written into an AI agent's instructions is only a request. That is the argument behind CUGA, IBM's open-source harness, and its team backs it with a test. A harness is the software around an AI model. It decides which tools the model can use, how steps are organised, and which mistakes get caught.
Rules the harness enforces
Many teams put business rules, like payment approval, into the model's instructions. The CUGA team argues that this is only a request. CUGA policies move such rules into the harness, so they hold whatever the model decides.
In a demo, an agent told to "Pay everything I owe" paid $90 without policies, including a denied claim in collections. With policies, it held the payment for approval and then paid only the $10 that was due.
CUGA is open source under Apache 2.0. It runs locally, on Kubernetes, or in air-gapped clusters. Which would you trust more: a rule written in a prompt, or one the system enforces?
The gap
The CUGA team kept the model fixed and changed only the harness. Task completion ran from 30% to 81%, depending on the harness. The team also counted scenarios, where each one has three variants and all three must succeed. On that count CUGA passed 20 scenarios, Hermes 8 and Deep Agents 2.
The source also cites a community analysis in which another model, GLM-5.2, rose from 23% to 52% on SWE-bench Pro with only the harness changed.
The test
The three harnesses were CUGA, Hermes from Nous Research, and Deep Agents from LangChain. All used the open-weight model GLM-5.3-flash on 90 tasks from AppWorld. That benchmark spans nine apps and 457 APIs, with things like email, payments and file storage. Each task got one attempt.
Limits
The sample is 90 tasks on one benchmark, so results may not carry over. The setups were not identical, and CUGA used an AppWorld configuration the others did not. The CUGA team ran the test itself. The runs used no policies, so they do not measure policy enforcement. Hermes and Deep Agents were not built for this kind of multi-app work. The team invites better configurations and promises to rerun them.
Source: Hugging Face Blog