Researchers released a new method to train smaller AI models for multiple tasks
The JIVEAdapter technique separates shared information from task-specific data to improve efficiency in AI training.
Researchers recently introduced JIVEAdapter, a way to make AI training more efficient. The method allows a single base model to handle several different jobs without needing to be fully retrained for each one.
Most current AI training requires significant computing power. JIVEAdapter addresses this by breaking down weight updates into two parts: a shared 'joint' structure used for everything, and a 'specific' structure for individual tasks.
JIVEAdapter decomposes every weight update into a Joint structure shared across all tasks plus a per-task Individual structure.
Think of it like learning to play instruments. You learn the basic theory once, which acts as your shared knowledge. Then, you practice specific techniques for the guitar or the piano separately. This way, you do not have to relearn how music works every time you pick up a new instrument.
This system freezes the shared part once it is learned. When a new task comes along, the AI only needs to train the smaller, specific part. This saves a lot of time and resources compared to training the whole model from scratch.
Tests on DeBERTaV3-base showed that this approach keeps performance high even with less data. It helps the model keep shared and task-specific signals separated, which makes the results easier to interpret.
Does this approach mean we might soon see more specialized, efficient AI running on smaller hardware?
However, this research focuses on specific language benchmarks like GLUE and SuperGLUE. The paper does not show how this method scales for larger models like those used for image generation or complex reasoning beyond text tasks.
Source: arXiv cs.AI