A 1.1-billion-parameter TinyLlama matched a 70-billion-parameter teacher while running up to 20 times as fast

A tiny language model now matches a giant one on travel tasks, and it answers 10 to 20 times faster. Amazon Science reports the result, from work with Booking.com. The work was presented at the Association for Computational Linguistics (ACL) meeting, where it won the Industry Track best-paper award.

The problem: big models are slow and costly

Large language models give good travel advice, but they have tens of billions of parameters. That makes them expensive, and the bill grows with every customer question. They can also be too slow for web and mobile apps, where even a 100-millisecond delay can cost customers.

Two tricks, used together

The first is knowledge distillation. A big "teacher" model labels data, and a small "student" model learns to copy its behaviour.

The second is Medusa. Normally a model writes one word or word fragment at a time. Medusa adds extra prediction parts, called heads, so the model guesses several pieces at once. Graphics chips handle that kind of parallel work well.

The team chose TinyLlama, with 1.1 billion parameters, as the student. The teacher was Llama-3.1 with 70 billion. They attached Medusa heads to the student while it trained on the teacher's predictions plus human-labeled data.

What each step added

• Fine-tuning TinyLlama on each task already beat few-shot GPT-4o, but it still trailed the teacher.
• Distillation closed more than 90% of that gap and made inference about 6 times as fast.
• Medusa heads added another 2 to 3.6 times.

They tested two Medusa variants. Both gave similar speedups with no meaningful quality loss. Medusa-2 trains everything in one stage, so it is simpler for teams running many models.

Three Booking.com features

• Smart Filters, which turns a typed search into structured filters, responded 15 times as fast for 99% of queries.
• AI Trip Planner showed a 2.9% rise in user engagement compared with GPT-3.5.
• AI-written accommodation descriptions got 1.4% more "helpful" ratings, at less than one-third the cost of GPT-4.

Imagine typing "quiet beach hotel with a pool" and getting filters before you blink. That is the kind of speed this targets.

Limits

Each feature got its own fine-tuned model trained on domain data, so this is not one model for everything. The source gives the gains as percentages and speed multiples, not raw timings. It does not say how the quality was scored beyond matching the teacher, and it does not report tests outside these three travel features.