Claude Opus generated tasks that a 9B model solved 81% of

A research team investigated why training AI agents to use computer terminals often fails. They published their findings on arXiv, detailing a specific pipeline for generating tasks.

The process uses a large model, specifically Claude Opus, to create terminal tasks and tests. This acts as a "meta-agent" that designs the homework for smaller models to solve.

The researchers found that just having a working Docker image is not enough. They identified three main problems: invalid benchmarks, fragile testing tools, and misaligned rewards.

They redesigned the prompts and added more context to the system. This change increased the baseline solvability by 5.6 times.

A smaller model with 9 billion parameters was then tested on these new tasks. It reached a mean pass rate of 81.3% within 20 steps.

However, the results changed when the team added harder tasks. The pass rate dropped to 20.6% without changing the training configuration.

This drop shows that the difficulty level is specific to the model being trained. What is easy for one model may be impossible for another.

The study suggests that reliability requires careful calibration of task difficulty. Verifier audits and infrastructure error accounting must be primary evaluation criteria.

Imagine a teacher creating a test. If the test is too easy, the student learns nothing. If it is too hard, the student gives up. The difficulty must match the student's current ability.

The paper does not detail the specific infrastructure errors encountered. It also does not provide a universal formula for calibrating task difficulty across all model sizes.