A Comprehensive Guide to Installing CUDA, cuDNN, and TensorRT on Linux and Integrate it with PyTorch, TensorFlow, and Paddleocr versions. | by Masoud Masoumi Moghadam | Aug, 2024

As a computer vision engineer, I work closely with libraries like Tensorflow, PyTorch, and PaddleOCR in day-to-day tasks. These libraries depend heavily on Nvidia cuda, cudnn, and TensorRT frameworks to utilize GPUs and optimize deep neural networks.

As the project gets bigger, one of the tasks that I find challenging is keeping cuda, cudnn, tensorrt, and the deep learning libraries depending on them like tensorflow, and pytorch updated. Finding the matching version of tensorflow, pytorch, (and paddlepaddle) with cuda (let alone the OS requirements like gcc, clang, etc) require time to research and test.

Recently I upgraded Cuda from version 11.8 to 12.3 to be able to install tensorflow integrated with Python 3.12, but it was not that simple and ended up having issues with other libraries working peacefully along them. After spending a couple of days struggling among different operating systems, and libraries versions, finally found a way to install the best working environment for my project.

In this article, I share the procedure that I got through to install Cuda, CudNN, TensorRT, Tensorflow, PyTorch, and PaddleOCR on Ubuntu 23.10 and run codes to verify their installations.

Before I start downloading and installation, these are the list of the items that I want them to work cohesively for my project:

• OS ubuntu x64–86 23.10
• CUDA
• CUDNN
• Tensorflow
• Pytorch
• Paddlepaddle
• Paddleocr

Most people prefer the most recent versions, but the first thing I noticed is that Cuda and cudnn development is faster than tensorflow, and pytorch development. for example, at the time of speaking, the most recent version of Cuda is 12.6, but tensorflow docs state that we need to install Cuda 12.3. In the same way, paddlepaddle docs suggest integration with cuda 12.0, PyTorch suggested cuda 2.4, and so on. So we shouldn’t always install the most recent version of Cuda and cudnn unless you want to build tensorflow, or pytorch from source.

The other important thing is the GPU hardware you are using. Based on the computing capability of GPU hardware, you can choose the CuDNN version. for example, I use Nvidia Geforce RTX 3090, and it has compute capability 8.6 which is supported by cuDNN 8.9.7.

Here are links to read to know how to pick the best matching version of items we want to install:

Here in this article, I chose these items and matching versions to be installed:

• Cuda 12.3 (The version for tensorflow 2.17)
• CudNN 8.9 (it is needed for tensorflow version 2.17)
• TensorRT 10.2 (the version that matches Python 3.12)
• Python 3.12
• Tensorflow 2.17
• PyTorch 2.4

Let’s begin with OS upgrades:

Install gcc and g++ for building capability.
(Check CUDA requirements for gcc versions)

if you probably have installed different versions of gcc and wondered how to switch between them, do it this way:

sudo ln -s -f /usr/bin/gcc-12 /usr/bin/gcc

It’s also recommended to install clang-17 for building tensorflow, but I installed tensorflow without needing a clang to be installed.

- sudo apt update

sudo apt install clang

or to install specific versions (like 16 or 17):

sudo apt install clang-17

Before installing the Nvidia driver, you have to make sure to have all the previous installations of Nvidia drivers, or Cuda are removed. so use this to remove them:

And remove these lines from ~/.bashrc if they exist:

Now let’s install Nvidia drivers using ubuntu-drivers which I found more peaceful than installation through .run files:

Now reboot the system, and after Linux is loaded again, run this command:

nvidia-smi

This must be the results:

Cuda installation through network repo is very straightforward, so as the Nvidia official docs for cuda installation suggest, we start it with Linux headers update:

You might encounter this error:

If so, just update /etc/apt/sources.listfor Ubuntu 23.04 (Lunar Lobster), according to this solution here.

Now to make CUDA available for Linux and its software, you have to update PATHand LD_LIBRARY_PATH in the ~/.bashrc file:

Now check Cuda availability by running nvcc --version in terminal:

From CuDNN archives, I downloaded cudnn-linux-x86_64-8.9.7.29_cuda12 version .tar file and installed it using these commands:

Download Anaconda from here, install it, and create a virual environment.

Then open a terminal make sure you have the `conda` command in the terminal and create an environment for dependency installation:

Now it’s time to install Python libraries and check if they are working fine with GPU.

Before downloading tensorRT, you should know which Python version you want to use. Since I wanted to use Python version 3.12, I installed tensorrt 10.2.0 which is the newest version at the time of speaking. You can download it from here.

Now after opening the zip file, I copy its subdirectory (TensorRT-10.2.0.19) to /home/my-user/, but you can choose another directory for this.

Now, Add these lines to ~/.bashrc the file:

Now let’s validate if tensorrt is installed correctly. Save this code as check_trt.py and run it using python check_trt.py :

install the available cuda-enabled version of tensorflow:

pip install tensorflow[and-cuda]

You can use TensorFlow’s device query functionality to get information about the available GPUs and their drivers using this code:

Here is the result:

Compute Capability is a version number that defines the features supported by a GPU’s hardware architecture in NVIDIA’s CUDA. Each version of the compute capability corresponds to a particular NVIDIA GPU architecture and dictates what CUDA features, instructions and optimizations are available on that GPU.

Compute Capability: 8.6 corresponds to the NVIDIA Ampere architecture, which is the architecture used in the NVIDIA GeForce RTX 30 series, including the RTX 3090.

Device 0, usually refers to the first GPU detected by the system or TensorFlow in a multi-GPU setup.

Name indicates NVIDIA GeForce RTX 3090 which is the model name of the GPU in question.

Finally, PCI Bus ID: 0000:06:00.0 is the identifier for the location of the GPU on the system’s PCI bus, which helps the operating system and software locate the device.

Finally, at the end of this procedure, we run these commands:

pip install paddlepaddle-gpu

pip install paddleocr

and to ensure that it is integrated with Cuda, run this code:

This article is designed not only for installing specific versions that are introduced here but also a comprehensive guide to finding the best working versions of Nvidia GPU acceleration frameworks (Cuda, cuDNN, TensorRT) and deep learning frameworks (Tensorflow, Pytorch, and Paddleocr). Although the versions in this article get old as time passes, I hope the instructions here will be helpful for future versions.