A 66 MB mini model caught a Transformers bug that hid in main for two months

Hugging Face says a tiny test model caught a bug that slipped through its normal checks for almost two months. The company described the case in a blog post on its Transformers library, and the numbers below come from its own replay of past commits.

The bug nobody saw

On July 30, 2026, Transformers turned on PyTorch's scaled dot-product attention (SDPA) for T5. A side effect hit UMT5. Its decoder could look at future words during training, so it could peek at the answer it was meant to predict.

Nothing crashed. The bug stayed in main until a fix on September 28.

The idea: shrink the model

Transformers supports over 750 model types. It runs 88,000+ CPU tests on main and 200,000+ GPU tests daily. Running real checkpoints on GPUs for every pull request is not practical, so a regression can land before a scheduled run notices.

Hugging Face's answer is a fixture: a small model built from the real configuration, with fewer and narrower layers but every kind of layer kept. Its tokenizer files are copied unchanged. The UMT5 fixture used 66 MB of weights, against 1,227 MB for the source checkpoint.

What a contract contains

The proposed standard is called transformers-mic, for Model Integration Contract. Each contract has three parts:

• a pinned fixture, published once
• a runnable usage example, such as generating text or a training step
• baselines: recorded outputs, reviewed by a human

The contract holds if the example on the fixture still gives the baselines. The SmolLM contract, for instance, checks token IDs, training loss, finite gradients and that an optimizer step changes the weights.

The results

Replaying five months of commits, the small UMT5 fixture flagged the regression on every commit from July 30 to September 28. Label log-probabilities were off by 1.22 and training loss by about 0.20. The contract had to use SDPA, the default path. Eager attention alone missed the bug.

On a proof branch with the fix reverted, the check failed again. The T5 runner step took 13.4 seconds on CPU, excluding setup.

Building the fixtures

A CLI called tmic drafts the contract and builds the fixture. An LLM writes the draft, but humans approve any changed expectations. For Llama 4 Scout, 108.6 billion parameters shrank to 44.7 million.

Random weights made a Whisper fixture barely react to audio. Training it on a few tones paired with words fixed that. Random Whisper fixtures caught 18 of 24 injected faults; the overfit ones caught all 24.

Imagine a smoke alarm that only needs a match, not a burning house. That is the trade here.

Limits

As of November 1, 2026, the MIC job in Transformers CI reports failures but does not block pull requests. Hugging Face has not measured how many real bugs it catches on PRs, how many false alarms it causes, or how much baseline review costs. The Whisper faults were synthetic, and both Whisper versions missed an assisted-decoding bug that the real checkpoint exposed. Coverage across models is still partial, and fixtures will not replace full integration tests.