drm: vc4: txp: Do not allow 24bpp formats when transposing

The hardware doesn't support transposing to 24bpp (RGB888/BGR888) formats. There's no way to advertise this through DRM, so block it from atomic_check instead. Signed-off-by: Dave Stevenson <[email protected]>
raspberrypi · Dec 11, 2024 · c0b2307 · c0b2307
1 parent 574ea34
commit c0b2307
Showing 1 changed file with 7 additions and 0 deletions.
diff --git a/drivers/gpu/drm/vc4/vc4_txp.c b/drivers/gpu/drm/vc4/vc4_txp.c
@@ -272,6 +272,13 @@ static int vc4_txp_connector_atomic_check(struct drm_connector *conn,
 		return -EINVAL;
 	}
 
+	if (conn_state->rotation == (DRM_MODE_ROTATE_0 | DRM_MODE_TRANSPOSE) &&
+	    (fb->format->format == DRM_FORMAT_RGB888 ||
+	     fb->format->format == DRM_FORMAT_BGR888)) {
+		DRM_DEBUG_KMS("24bpp formats not supported when tranposing\n");
+		return -EINVAL;
+	}
+
 	for (i = 0; i < ARRAY_SIZE(drm_fmts); i++) {
 		if (fb->format->format == drm_fmts[i])
 			break;