Nvidia is banding together with a list of tech partners on a game-changing piece of software that’s set to double the performance of its flagship H100 Tensor Core GPUs.
The open source TensorRT-LLM update, which is set for release in the coming weeks, sees an up-to-date system outperform the A100 by eightfold, whereas H100s would previously outperform the A100 by just fourfold. This was tested on the GPT-J 6B, a model that’s used to summarise articles from CNN and Daily Mail.
When tested on Meta’s Llama2 LLM, TensorRT-LLM-powered H100s outperformed A100s by 4.6 times – versus 2.6 times before the update.
Nvidia H100s faster than ever
The versatility and dynamism of large language models (LLMs) can make it difficult to batch requests and execute them in parallel, which means some requests finish much earlier than others.
To solve this, Nvidia and its partners embedded TensorRT-LLM with a more powerful scheduling technique called in-flight batching. This takes advantage of the fact text generation can be broken down into multiple subtasks.
Put simply, instead of waiting for an entire batch of tasks from one request to finish before moving on to the next request, the system can continue processing new batches from different requests in parallel.
TensorRT-LLM comprises a TensorRT deep learning compiler and includes optimized kernels, pre-processing and post-processing steps, as well as multi-GPU and multi-node communication primitives.
The result? Groundbreaking performance on Nvidia’s GPUs paving the way for new large language model experimentation, quick customization, and peak performance.
This software uses tensor parallelism, in which individual weight matrices are split across devices, in turn, allowing efficient inference at scale; each model runs in parallel across multiple GPUs and across multiple servers.
TensorRT-LLM also includes fully optimized and read-to-run versions of popular LLMs including Llama 2, GPT-2 and GPT-3, as well as Falcon, Mosaic MPT, BLOOM, and dozens of others. These can be accessed through a Python API.
The update is available in early access, and will soon be integrated into the Nvidia NeMo framework, which is part of Nvidia AI Enterprise. Researchers can access this through the NeMo framework, the NGC portal, or through the source repository on GitHub.
