EdgeLLM-Nano found KV-cache compression hurt a 33.6 MB model
The testbed used an 8.39M-parameter model and a new LCDF check to compare cache methods token by token.
EdgeLLM-Nano tested KV-cache compression on a tiny on-device language model and found that several popular tricks did not help as much as expected.
The setup was a decoder-only transformer with 8.39M parameters, or 33.6 MB, built to make end-to-end checks practical on edge hardware.
LCDF separated the methods by using paired statistics across seeds and effect sizes.
The paper says older evaluation methods can miss the KV-cache entirely, which makes comparisons look cleaner than they really are.
One result was harsh for a polar-guided key-cache quantizer. The authors say a round-to-nearest control caused about 10 times less damage.
The same pattern showed up in pruning. A consistency-anchored progressive weight pruner did not beat simple one-shot magnitude-based pruning, and sometimes matched it.
To measure quality more directly, the paper introduced Long-Context Decode Fidelity, or LCDF. It checks teacher-forced log-loss token by token while using the cache under test.
That matters because perplexity gave every method the same 5.461 score. LCDF separated the methods by using paired statistics across seeds and effect sizes.
There was also an energy check on a Raspberry Pi 5. Every compression backend tested used more energy per token than plain FP16 inference.
KIVI came closest to lossless behavior under LCDF, but the broader message was that KV-cache compression still has room to improve. What would you rather trust for edge AI: one smooth perplexity score or a token-by-token test that actually looks at the cache?
The source does not show a public product release, and it only reports the methods and devices tested in this measurement study.
Source: arXiv cs.CL