NVIDIA NeMo-Aligner Enhances Supervised Fine-Tuning with Data-Efficient Knowledge Distillation

NVIDIA’s NeMo-Aligner has unveiled a new methodology for enhancing supervised fine-tuning (SFT) through data-efficient knowledge distillation. This innovative approach allows for the transfer of knowledge from a larger teacher model to a more compact student model, achieving comparable accuracy with reduced data requirements, according to NVIDIA.

Advancements in Knowledge Distillation

Knowledge distillation is a technique that has been widely used in pretraining scenarios but is less explored in the context of supervised fine-tuning. NeMo-Aligner aims to bridge this gap by leveraging knowledge distillation during SFT to enhance model accuracy and efficiency. The method achieves higher accuracy than standard SFT by utilizing only 70% of the training steps, as demonstrated in their experiments.

Implementation and Benefits

The NeMo-Aligner uses a KD-logit approach, where the student model is trained to match the teacher’s output logits. This technique, known as “dark knowledge,” provides a more informative gradient signal by understanding the similarities and dissimilarities across classes. The process involves preprocessing where the teacher model’s predictions are cached, and the student model is trained to align with these predictions, resulting in memory savings and faster training times.

The approach significantly reduces the need for simultaneous loading of both teacher and student models, thus saving GPU memory. Instead, only the top-K logits of the teacher are stored, optimizing memory usage while maintaining detailed information transfer.

Empirical Results

Experiments conducted with the Nemotron-4 15B student model and a fine-tuned Nemotron-4 340B teacher model reveal that the KD-finetuned models outperform the vanilla SFT models in multiple benchmarks, including HumanEval, MBPP, and MATH. Notably, the KD-finetuned model requires fewer training tokens while achieving superior performance across six of seven evaluation metrics.

The KD approach also excels in the MMLU benchmark, which assesses a wide range of language understanding tasks, outperforming the baseline in both zero-shot and five-shot settings.

Conclusion

NVIDIA’s implementation of knowledge distillation in NeMo-Aligner demonstrates that this technique not only enhances model performance in data-scarce environments but also synergizes effectively with synthetic data generation (SDG) techniques. As a result, it offers a powerful tool for developers aiming to maximize model efficiency and accuracy through supervised fine-tuning.

Image source: Shutterstock