Bypassing NVIDIA NVENC transcoding limits
Since my server came with a GeForce GT 710, I wanted to harvest it’s powers to transcode my media.
Sadly, Nvidia puts a limit on how much it can transcode at once. It can only transcode 2 things at once. This is the case on all consumer Nvidia GPUs and I don’t really understand why it’s there.
Intel and AMD does not have any limits as far as I’m aware.
Thankfully, a script has been made by the community to bypass these limits and it’s stupid simple to install.
The following tutorial is mainly for Debian-based distributions.
First, clone the git repository to your machine.
git clone https://github.com/keylase/nvidia-patch
Then go in to the nvidia-patch directory.
If you have compatible drivers you can run the script and you are good to go. If not, you can still run the script and it will tell you that you do not have compatible drivers.
To install compatible drivers, first remove the old drivers. On Ubuntu/Debian you can run,
sudo apt remove nvidia*
to uninstall the old nvidia drivers. If you are on a different distribution, please look it up.
Switch to the root user and make a folder called nvidia under /opt and go to that folder.
mkdir /opt/nvidia && cd /opt/nvidia
Now you can download the drivers, the latest driver right now is 510.60.02. The command below downloads that version.
wget https://international.download.nvidia.com/XFree86/Linux-x86_64/510.60.02/NVIDIA-Linux-x86_64-510.60.02.run
Then make it executable
chmod +x NVIDIA-Linux-x86_64-510.60.02.run
Next, just run the nvidia installer
./NVIDIA-Linux-x86_64-510.60.02.run
Then just follow the setup guide.
Now you should have the latest Nvidia drivers installed, to verify that it’s has been installed successfully, run nvidia-smi
The output should look something like this:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.50 Driver Version: 430.50 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GT 710 Off | 00000000:09:00.0 N/A | N/A |
| 50% 43C P0 N/A / N/A | 0MiB / 980MiB | N/A Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 Not Supported |
+-----------------------------------------------------------------------------+
If this doesn’t show up, your drivers probably failed installation or something else happened, try restarting your server and running the command again, if it still doesn’t work try reinstalling the drivers.
Then after we have verified that the drivers are installed properly, we can go back to the nvidia-patch folder.
Then we execute the patch:
./patch.sh
And that should be it!
You have successfully patched your drivers to support more than 2 transcodes at once.
To verify that the patch is working you can try running ffmpeg, here is a example command to see if its working
ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda \
-f lavfi -i testsrc -t 50 \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 4M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 1M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 8M -f null - \
-vf hwupload -c:a copy -c:v h264_nvenc -b:v 6M -f null -
If you get a message like:
[h264_nvenc @ 0x55fc31535fc0] OpenEncodeSessionEx failed: out of memory (10): (no details)
Error initializing output stream 3:0 -- Error while opening encoder for output stream #3:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
The patch has failed, try running it again or if that doesn’t work, install older drivers.
The patch is working if you get a output like:
frame= 560 fps=329 q=8.0 Lq=8.0 q=8.0 q=8.0 size=N/A time=00:00:22.40 bitrate=N/A speed=13.2x
Thanks to Bryde on the ScooterHacking discord for making me aware of this patch.