Microsoft introduced a new skill to automate the testing and patching of AI agents

Microsoft released a new tool called run-assert-eval designed to help developers secure AI agents without manual oversight. The skill automates the process of finding weaknesses, creating defensive policies, and proving those protections work.

Building secure agents has been a manual, error-prone task for developers. Teams often had to identify failure modes, write specific rules, and re-test everything by hand. Microsoft's new approach simplifies this by linking several existing tools together into a single workflow.

How the process works

• Discovery: The tool scans an agent to find potential risks using Clarity.
• Evaluation: It uses ASSERT to measure how frequently the agent fails in specific scenarios.
• Patching: The tool automatically generates an Agent Control Specification policy to block identified failures.
• Verification: Finally, it reruns the evaluation to ensure the agent is safer without losing its helpfulness.

In one test case, a billing support agent leaked customer data in 30% of baseline conversations. After applying the automated controls, the error rate dropped significantly to under 6%.

The tool relies on existing research and industry standards to define what constitutes a failure. By using a consistent set of test cases for both the baseline and the patched version, developers get clear evidence of whether their changes actually fixed the problem.

This workflow requires the developer to choose which risks to prioritize after the tool identifies them. It does not replace human judgment regarding the final business requirements. The current implementation is limited to the specific agent frameworks and policies defined by the system, and results can vary depending on the complexity of the task.