A garage door became a grumpy pirate ship that talks to trick-or-treaters on one workstation
The MonsterHouse project uses NVIDIA's Nemotron Voice Agent and runs on two GPUs, with a 4GB language model writing the lines.
In brief
A garage door talks back in 7 moods
MonsterHouse listens, thinks and speaks as a grumpy pirate ship. It runs on one workstation with two GPUs. Real trick-or-treat crowds are still untested.
A garage door can now listen to children, answer in the voice of a cranky old pirate ship and soften when they say please. It is called MonsterHouse. An NVIDIA employee built it for Halloween and described it in a Hugging Face blog post.
How the house works
The whole thing runs on one workstation. NVIDIA Nemotron Voice Agent ties the pieces together.
- Listening: Nemotron ASR turns a visitor's speech into text.
- Thinking: Nemotron 3.5 Bolt E4B 4GB, a small open language model, writes each reply. A LoRA adapter, a small add-on that tunes a model's personality, makes it grumpy.
- Speaking: Magpie TTS Zeroshot makes the voice from a short sample of the character.
- Animating: Audio2Face-3D moves the wooden mouth in Unreal Engine 5.
- Seeing: Nemotron 3 Nano Omni describes costumes. YOLO and ByteTrack count and track people.
Teaching it manners
The backstory is that the house was built from the timbers of a sunken pirate ship. At first it brushes visitors off. A "please" wins it over quickly, and "thank you" earns a warm goodbye.
The builder tested more than a dozen models before choosing Bolt E4B. He says it has a smaller footprint than Qwen3 30B-A3B and Qwen 3.6 35B-A3B. He also notes that he works at NVIDIA and tested Bolt before its public release.
The LoRA was trained on a dataset of over 5000 multi-turn conversations. A local agent and a very large LLM created and judged it.
Speed and splitting the work
To avoid awkward pauses, the system starts speaking before a full answer is written. Prepared greetings and candy jokes are stored as text and recorded audio, which saves compute. The voice has seven moods, each with its own 3-10 second reference sample.
Two GPUs share the load. An NVIDIA RTX PRO 6000 Blackwell Max-Q runs language and voice. An RTX PRO 4000 Blackwell runs graphics and animation, so the mouth keeps a steady frame rate.
Imagine a control desk for a puppet. The builder has one, with a soundboard, custom lines and a microphone mode that lets him speak through the house directly.
Limits
The MonsterHouse code is not packaged for release, though the NVIDIA models it uses are available. The sheep costume test was synthetic, and the author says it is not proof of a successful Halloween night. Real groups, costumes and night lighting still need testing. Costume recognition and witty replies may need more training. Automatic candy handing and a fully hands-off night have not been reached.
Source: Hugging Face Blog