Mico's Blog

Bypassing NVIDIA NVENC transcoding limits

Published in Linux, Servers.

Since my server came with a GeForce GT 710, I wanted to harvest it’s powers to transcode my media.

Sadly, Nvidia puts a limit on how much it can transcode at once. It can only transcode 2 things at once. This is the case on all consumer Nvidia GPUs and I don’t really understand why it’s there.

Intel and AMD does not have any limits as far as I’m aware.

Thankfully, a script has been made by the community to bypass these limits and it’s stupid simple to install.

The following tutorial is mainly for Debian-based distributions.

First, clone the git repository to your machine.

git clone https://github.com/keylase/nvidia-patch

Then go in to the nvidia-patch directory.

If you have compatible drivers you can run the script and you are good to go. If not, you can still run the script and it will tell you that you do not have compatible drivers.

To install compatible drivers, first remove the old drivers. On Ubuntu/Debian you can run,

sudo apt remove nvidia*

to uninstall the old nvidia drivers. If you are on a different distribution, please look it up.

Switch to the root user and make a folder called nvidia under /opt and go to that folder.

mkdir /opt/nvidia && cd /opt/nvidia

Now you can download the drivers, the latest driver right now is 510.60.02. The command below downloads that version.

wget https://international.download.nvidia.com/XFree86/Linux-x86_64/510.60.02/NVIDIA-Linux-x86_64-510.60.02.run

Then make it executable

chmod +x NVIDIA-Linux-x86_64-510.60.02.run

Next, just run the nvidia installer


Then just follow the setup guide.

Now you should have the latest Nvidia drivers installed, to verify that it’s has been installed successfully, run nvidia-smi

The output should look something like this:

| NVIDIA-SMI 430.50       Driver Version: 430.50       CUDA Version: 10.1     |
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|   0  GeForce GT 710      Off  | 00000000:09:00.0 N/A |                  N/A |
| 50%   43C    P0    N/A /  N/A |      0MiB /   980MiB |     N/A      Default |

| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|    0                    Not Supported                                       |

If this doesn’t show up, your drivers probably failed installation or something else happened, try restarting your server and running the command again, if it still doesn’t work try reinstalling the drivers.

Then after we have verified that the drivers are installed properly, we can go back to the nvidia-patch folder.

Then we execute the patch:


And that should be it!

You have successfully patched your drivers to support more than 2 transcodes at once.

To verify that the patch is working you can try running ffmpeg, here is a example command to see if its working

ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda \
-f lavfi -i testsrc -t 50 \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 4M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 1M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 8M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 6M -f null -  

If you get a message like:

[h264_nvenc @ 0x55fc31535fc0] OpenEncodeSessionEx failed: out of memory (10): (no details)
Error initializing output stream 3:0 -- Error while opening encoder for output stream #3:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!

The patch has failed, try running it again or if that doesn’t work, install older drivers.

The patch is working if you get a output like:

frame=  560 fps=329 q=8.0 Lq=8.0 q=8.0 q=8.0 size=N/A time=00:00:22.40 bitrate=N/A speed=13.2x

Thanks to Bryde on the ScooterHacking discord for making me aware of this patch.