New Ubuntu Update Broke Both OBS NVENC and Wine Gaming – Fixed with iommu=pt

kibasnowpaw

Well-Known Member
Joined
Jan 2, 2022
Messages
548
Reaction score
493
Credits
7,712
After installing a large update through KDE Discover, both OBS and gaming through Wine/Proton stopped working correctly at the same time.

Everything had worked normally the day before.

My system:

  • Ubuntu 26.04
  • KDE Plasma using X11
  • Kernel 7.0.0-28-generic
  • NVIDIA GeForce RTX 2070 Super
  • NVIDIA driver 595.84
  • OBS Studio 32.1.0
  • Lutris with UMU and GE-Proton11-3

The update itself did not install a new NVIDIA driver. It mainly updated glibc/libc6 packages from:

Code:
2.43-2ubuntu2

to:

Code:
2.43-2ubuntu2.3

This included both the 64-bit and 32-bit versions of libc6.

OBS and NVENC stopped working

Immediately after the update, OBS could no longer start NVENC.

OBS reported:

Code:
NV_ENC_ERR_OUT_OF_MEMORY

and:

Code:
NVENC Error: Too many concurrent sessions

However, there were no active NVENC sessions:

Code:
nvidia-smi encodersessions

The GPU also had plenty of available memory. Only around 538 MiB of 8 GiB VRAM was being used.

I tested NVENC directly through FFmpeg:

Code:
ffmpeg -hide_banner \
-f lavfi -i testsrc2=size=1920x1080:rate=60 \
-t 5 -c:v h264_nvenc -f null -

FFmpeg failed with:

Code:
CreateBitstreamBuffer failed: out of memory (10)
Segmentation fault

This confirmed that OBS itself was not the problem. NVENC was broken at the NVIDIA driver or kernel level.

Reinstalling the NVIDIA packages temporarily restored NVENC:

Code:
sudo apt update

sudo apt install --reinstall \
nvidia-driver-595 \
nvidia-utils-595 \
libnvidia-encode-595:amd64 \
libnvidia-compute-595:amd64

sudo update-initramfs -u
sudo reboot

After that, OBS could use NVENC again.

Gaming was still broken

Far Cry 2 would start normally once, but after closing it, the game would fail to start again. It also crashed when OBS was running.

Lutris showed that all Wine and game processes had exited, and this command returned nothing:

Code:
pgrep -a -f 'farcry2|wineserver|wine64|wine-preloader|umu|proton'

So this was not caused by a stuck Wine process.

The important error was in the kernel log:

Code:
sudo dmesg -T | grep -iE \
'swiotlb|DMAR|IOMMU|DMA mapping|NVRM|Xid' | tail -n 200

The log showed:

Code:
iommu: Default domain type: Translated
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

It was followed by many NVIDIA errors:

Code:
NVRM: 0000:03:00.0: Failed to create a DMA mapping!

The system also reported firmware-related IOMMU warnings:

Code:
DMAR: [Firmware Bug]: RMRR entry for device 03:00.2 is broken - applying workaround
DMAR: [Firmware Bug]: RMRR entry for device 08:00.0 is broken - applying workaround

This explained why OBS and gaming failed together.

OBS NVENC and Wine/DXVK both needed the NVIDIA driver to create DMA mappings. Once those mappings started failing, NVENC reported a false out-of-memory error, and Wine games stopped launching correctly.

The fix

The fix was to use IOMMU passthrough mode.

I opened the GRUB configuration:

Code:
sudo nano /etc/default/grub

I added this kernel parameter:

Code:
iommu=pt

For example:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt"

If the line already contains other options, keep them:

Code:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt"

I then updated GRUB and rebooted:

Code:
sudo update-grub
sudo reboot

After rebooting, I confirmed that the option was active:

Code:
cat /proc/cmdline

The output included:

Code:
iommu=pt

Result

After adding
Code:
iommu=pt
:

  • OBS NVENC worked again.
  • FFmpeg NVENC encoding worked.
  • Far Cry 2 could be started multiple times.
  • Far Cry 2 worked while OBS was running.
  • The NVIDIA DMA-mapping failures stopped breaking both applications.

The strange part is that the KDE Discover update did not update the NVIDIA driver. It mainly updated glibc/libc6.

I therefore cannot say for certain that glibc directly caused the DMA problem. It may have exposed an existing problem between kernel 7.0, NVIDIA 595.84, my motherboard firmware and translated IOMMU mode.

But the timeline was clear:

  1. OBS and Wine gaming worked normally.
  2. KDE Discover installed the update.
  3. OBS NVENC and Wine gaming both stopped working.
  4. The kernel reported repeated NVIDIA DMA-mapping failures.
  5. Adding
    Code:
    iommu=pt
    fixed both problems.

Has anyone else experienced this after the recent Ubuntu 26.04 libc6 update, especially with NVIDIA driver 595.84 or kernel 7.0?

I would also be interested to know whether this is mainly an NVIDIA 595 issue, a kernel 7.0 IOMMU regression, or a motherboard firmware problem exposed by the update.
 


Follow Linux.org

Members online


Top