NVIDIA Enhances TensorRT Model Optimizer v0.15 with Improved Inference Performance

NVIDIA has introduced the latest v0.15 release of the NVIDIA TensorRT Model Optimizer, a cutting-edge quantization toolkit designed to enhance model optimization techniques such as quantization, sparsity, and pruning. This update aims to reduce model complexity and optimize the inference speed of generative AI models, according to NVIDIA Technical Blog.

Cache Diffusion

The new version includes support for cache diffusion, building on the previously established 8-bit post-training quantization (PTQ) technique. This feature accelerates diffusion models at inference time by reusing cached outputs from previous denoising steps. Methods like DeepCache and block caching optimize inference speed without additional training. This mechanism leverages the temporal consistency of high-level features between consecutive denoising steps, making it compatible with models like DiT and UNet.

Developers can enable cache diffusion by using a single ‘cachify’ instance in the Model Optimizer with the diffusion pipeline. For instance, enabling cache diffusion in a Stable Diffusion XL (SDXL) model on an NVIDIA H100 Tensor Core GPU delivers a 1.67x speedup in images per second. This speedup further increases when FP8 is also enabled.

Quantization-Aware Training with NVIDIA NeMo

Quantization-aware training (QAT) simulates the effects of quantization during neural network training to recover model accuracy post-quantization. This process involves computing scaling factors and incorporating simulated quantization loss into the fine-tuning process. The Model Optimizer uses custom CUDA kernels for simulated quantization, achieving lower precision model weights and activations for efficient hardware deployment.

Model Optimizer v0.15 expands QAT integration support to include NVIDIA NeMo, an enterprise-grade platform for developing custom generative AI models. This first-class support for NeMo models allows users to fine-tune models directly with the original training pipeline. For more details, see the QAT example in the NeMo GitHub repository.

QLoRA Workflow

Quantized Low-Rank Adaptation (QLoRA) is a fine-tuning technique that reduces memory usage and computational complexity during model training. It combines quantization with Low-Rank Adaptation (LoRA), making large language model (LLM) fine-tuning more accessible. Model Optimizer now supports the QLoRA workflow with NVIDIA NeMo using the NF4 data type. For a Llama 13B model on the Alpaca dataset, QLoRA can reduce peak memory usage by 29-51% while maintaining model accuracy.

Expanded Support for AI Models

The latest release also expands support for a wider suite of AI models, including Stability.ai’s Stable Diffusion 3, Google’s RecurrentGemma, Microsoft’s Phi-3, Snowflake’s Arctic 2, and Databricks’ DBRX. For more details, refer to the example scripts and support matrix available in the Model Optimizer GitHub repository.

Get Started

NVIDIA TensorRT Model Optimizer provides seamless integration with NVIDIA TensorRT-LLM and TensorRT for deployment. It is available for installation on PyPI as nvidia-modelopt. Visit the NVIDIA TensorRT Model Optimizer GitHub page for example scripts and recipes for inference optimization. Comprehensive documentation is also available.

Image source: Shutterstock