Hardware video decoding with OpenGL

[leon-costa]

Hello,

In the plplay demo it doesn't seem possible to use hardware decoding with OpenGL. All the hardware configs returned by libavcodec seem to have an incompatible pixel format. According to #270, this behavior is expected. I guess it's outside the scope of libplacebo?

Does anyone have any pointers to resources on how to make it work, if it's even possible? @Hlongyu mentioned some "hardware map" used by MPV. Any hint on where I can find this in the MPV source code?

[haasn]

It works for me out of the box:

λ plplay -H -w glfw-gl ~/test.mp4
Initialized libplacebo v7.349.0 (API v349)
Opening file: '/home/nand/test.mp4'
Format: mov,mp4,m4a,3gp,3g2,mj2
Found video track (stream 0)
Resolution: 1920 x 1080
FPS: 25.000000
TBR: 25.000000
TBN: 0.000011
Bitrate: 2466 kbps
Format: yuv420p
Attempting to initialize API: GLFW (opengl)
Creating 1920x1080 window...
Detected OpenGL version strings:
    GL_VERSION:  4.6 (Core Profile) Mesa 24.1.3
    GL_VENDOR:   AMD
    GL_RENDERER: AMD Radeon RX 7900 XT (radeonsi, navi31, LLVM 18.1.8, DRM 3.57, 6.8.8-1-default)
    GL_SHADING_LANGUAGE_VERSION: 4.60
GPU information:
    GLSL version: 460
      max_shmem_size:            65536
      max_group_threads:         1024
      max_group_size[0]:         1024
      max_group_size[1]:         1024
      max_group_size[2]:         1024
      subgroup_size:             0
      min_gather_offset:         -32
      max_gather_offset:         31
    Limits:
      thread_safe:               1
      callbacks:                 1
      max_buf_size:              18446744073709551615
      max_ubo_size:              2147483647
      max_ssbo_size:             2147483647
      max_vbo_size:              18446744073709551615
      max_mapped_size:           18446744073709551615
      max_buffer_texels:         0
      align_host_ptr:            4096
      host_cached:               1
      max_tex_1d_dim:            16384
      max_tex_2d_dim:            16384
      max_tex_3d_dim:            8192
      blittable_1d_3d:           0
      buf_transfer:              1
      align_tex_xfer_pitch:      4
      align_tex_xfer_offset:     32
      max_variable_comps:        16384
      max_constants:             0
      max_pushc_size:            0
      align_vertex_stride:       1
      max_dispatch[0]:           2147483646
      max_dispatch[1]:           65535
      max_dispatch[2]:           65535
      fragment_queues:           1
      compute_queues:            1
    External API interop:
      UUID: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
      PCI: 0000:00:00:0
      buf export caps: 0x0
      buf import caps: 0x10
      tex export caps: 0x8
      tex import caps: 0x8
      sync export caps: 0x0
      sync import caps: 0x0
GLFW err 65544: Wayland: Window position retrieval not supported
Codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
Using hardware frame format: vaapi
Spent 151.257 ms loading cache (slow!)
Spent 26.007 ms compiling shader
Dithering to 8 bit depth
Estimated source FPS: 25.000, display FPS: 178.022
...
Exiting...
Received frame after EOF signaled... discarding frame!
Dropped frame with PTS 3.520
Dropped frame with PTS 3.440
Dropped frame with PTS 3.480
Spent 30.894 ms saving cache

OpenGL should have the ability to import DMABUFs, and vaapi hwdec images get auto-derived to dmabuf by the wrapping code: https://github.com/haasn/libplacebo/blob/master/src/include/libplacebo/utils/libav_internal.h#L1041

[leon-costa]

I'm using the nvidia proprietary driver which doesn't seem to support VAAPI.

I usually use hwdec=cuda with MPV. plplay tries the cuda hardware config but stops because the number of components in the pixel format descriptor is 0 here. The comment there seems to say it's expected to have 0 components with hardware acceleration.

It also tries the vdpau hardware config and also stops because nb_components is 0.

plplay -H -w glfw-gl ~/tmp/test.mp4

Initialized libplacebo v7.349.0 (v7.349.0-5-g82bf46ae) (API v349)
Opening file: '/home/leon/tmp/test.mp4'
Format: mov,mp4,m4a,3gp,3g2,mj2
Found video track (stream 0)
Resolution: 3600 x 1200
FPS: 30.000000
TBR: 30.000000
TBN: 0.000065
Bitrate: 4415 kbps
Format: yuv420p
Attempting to initialize API: GLFW (opengl)
Creating 3600x1200 window...
Detected OpenGL version strings:
    GL_VERSION:  4.6.0 NVIDIA 555.58.02
    GL_VENDOR:   NVIDIA Corporation
    GL_RENDERER: NVIDIA GeForce GTX 1080/PCIe/SSE2
    GL_SHADING_LANGUAGE_VERSION: 4.60 NVIDIA
GPU information:
    GLSL version: 460
      max_shmem_size:            49152
      max_group_threads:         1536
      max_group_size[0]:         1536
      max_group_size[1]:         1024
      max_group_size[2]:         64
      subgroup_size:             0
      min_gather_offset:         -32
      max_gather_offset:         31
    Limits:
      thread_safe:               1
      callbacks:                 1
      max_buf_size:              18446744073709551615
      max_ubo_size:              65536
      max_ssbo_size:             2147483647
      max_vbo_size:              18446744073709551615
      max_mapped_size:           18446744073709551615
      max_buffer_texels:         0
      align_host_ptr:            0
      host_cached:               1
      max_tex_1d_dim:            32768
      max_tex_2d_dim:            32768
      max_tex_3d_dim:            16384
      blittable_1d_3d:           0
      buf_transfer:              1
      align_tex_xfer_pitch:      4
      align_tex_xfer_offset:     32
      max_variable_comps:        4096
      max_constants:             0
      max_pushc_size:            0
      align_vertex_stride:       1
      max_dispatch[0]:           2147483647
      max_dispatch[1]:           65535
      max_dispatch[2]:           65535
      fragment_queues:           1
      compute_queues:            1
    External API interop:
      UUID: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
      PCI: 0000:00:00:0
      buf export caps: 0x0
      buf import caps: 0x0
      tex export caps: 0x8
      tex import caps: 0x8
      sync export caps: 0x0
      sync import caps: 0x0
Codec: h264 (H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10)
[AVHWDeviceContext @ 0x559208b8eb80] Failed to initialise VAAPI connection: -1 (unknown libva error).
libavcodec: Failed opening HW device context, skipping
Using software decoding
GL: Program/shader state performance warning: Vertex shader in program 1 is being recompiled based on GL state.
GL: Program/shader state performance warning: Vertex shader in program 2 is being recompiled based on GL state.
GL: Program/shader state performance warning: Vertex shader in program 3 is being recompiled based on GL state.
GL: Program/shader state performance warning: Vertex shader in program 4 is being recompiled based on GL state.
GL: Program/shader state performance warning: Vertex shader in program 5 is being recompiled based on GL state.
Dithering to 8 bit depth
Estimated source FPS: 30.000, display FPS: 51.586
Exiting...
Received frame after EOF signaled... discarding frame!
Dropped frame with PTS 4.733
Dropped frame with PTS 4.667
Dropped frame with PTS 4.700

[haasn]

We have no support for vdpau or cuda unfortunately. It's a bit out-of-scope because it would require "third party" API calls (unlike dmabufs which are supported by OpenGL directly).

Here's how mpv does it:

• https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/hwdec_cuda.c
• https://github.com/mpv-player/mpv/blob/master/video/out/hwdec/hwdec_cuda_gl.c

[Hlongyu]

happy to know libplacebo support vaapi, how about d3d11?