Deferred Renderer (bevyengine#9258)

# Objective - Add a [Deferred Renderer](https://en.wikipedia.org/wiki/Deferred_shading) to Bevy. - This allows subsequent passes to access per pixel material information before/during shading. - Accessing this per pixel material information is needed for some features, like GI. It also makes other features (ex. Decals) simpler to implement and/or improves their capability. There are multiple approaches to accomplishing this. The deferred shading approach works well given the limitations of WebGPU and WebGL2. Motivation: [I'm working on a GI solution for Bevy](https://youtu.be/eH1AkL-mwhI) # Solution - The deferred renderer is implemented with a prepass and a deferred lighting pass. - The prepass renders opaque objects into the Gbuffer attachment (`Rgba32Uint`). The PBR shader generates a `PbrInput` in mostly the same way as the forward implementation and then [packs it into the Gbuffer](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/render/pbr.wgsl#L168). - The deferred lighting pass unpacks the `PbrInput` and [feeds it into the pbr() function](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/deferred_lighting.wgsl#L65), then outputs the shaded color data. - There is now a resource [DefaultOpaqueRendererMethod](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L599) that can be used to set the default render method for opaque materials. If materials return `None` from [opaque_render_method()](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L131) the `DefaultOpaqueRendererMethod` will be used. Otherwise, custom materials can also explicitly choose to only support Deferred or Forward by returning the respective [OpaqueRendererMethod](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L603) - Deferred materials can be used seamlessly along with both opaque and transparent forward rendered materials in the same scene. The [deferred rendering example](https://github.com/DGriffin91/bevy/blob/deferred/examples/3d/deferred_rendering.rs) does this. - The deferred renderer does not support MSAA. If any deferred materials are used, MSAA must be disabled. Both TAA and FXAA are supported. - Deferred rendering supports WebGL2/WebGPU. ## Custom deferred materials - Custom materials can support both deferred and forward at the same time. The [StandardMaterial](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/render/pbr.wgsl#L166) does this. So does [this example](https://github.com/DGriffin91/bevy_glowy_orb_tutorial/blob/deferred/assets/shaders/glowy.wgsl#L56). - Custom deferred materials that require PBR lighting can create a `PbrInput`, write it to the deferred GBuffer and let it be rendered by the `PBRDeferredLightingPlugin`. - Custom deferred materials that require custom lighting have two options: 1. Use the base_color channel of the `PbrInput` combined with the `STANDARD_MATERIAL_FLAGS_UNLIT_BIT` flag. [Example.](https://github.com/DGriffin91/bevy_glowy_orb_tutorial/blob/deferred/assets/shaders/glowy.wgsl#L56) (If the unlit bit is set, the base_color is stored as RGB9E5 for extra precision) 2. A Custom Deferred Lighting pass can be created, either overriding the default, or running in addition. The a depth buffer is used to limit rendering to only the required fragments for each deferred lighting pass. Materials can set their respective depth id via the [deferred_lighting_pass_id](https://github.com/DGriffin91/bevy/blob/b79182d2a32cac28c4213c2457a53ac2cc885332/crates/bevy_pbr/src/prepass/prepass_io.wgsl#L95) attachment. The custom deferred lighting pass plugin can then set [its corresponding depth](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/deferred_lighting.wgsl#L37). Then with the lighting pass using [CompareFunction::Equal](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/mod.rs#L335), only the fragments with a depth that equal the corresponding depth written in the material will be rendered. Custom deferred lighting plugins can also be created to render the StandardMaterial. The default deferred lighting plugin can be bypassed with `DefaultPlugins.set(PBRDeferredLightingPlugin { bypass: true })` --------- Co-authored-by: nickrart <[email protected]>
robtfm · Oct 12, 2023 · a15d152 · a15d152
1 parent c8fd390
commit a15d152
Show file tree

Hide file tree

Showing 40 changed files with 2,959 additions and 485 deletions.
diff --git a/Cargo.toml b/Cargo.toml
@@ -611,6 +611,16 @@ description = "Illustrates bloom configuration using HDR and emissive materials"
 category = "3D Rendering"
 wasm = true
 
+[[example]]
+name = "deferred_rendering"
+path = "examples/3d/deferred_rendering.rs"
+
+[package.metadata.example.deferred_rendering]
+name = "Deferred Rendering"
+description = "Renders meshes with both forward and deferred pipelines"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "load_gltf"
 path = "examples/3d/load_gltf.rs"

diff --git a/assets/shaders/array_texture.wgsl b/assets/shaders/array_texture.wgsl
@@ -1,6 +1,6 @@
 #import bevy_pbr::mesh_vertex_output    MeshVertexOutput
 #import bevy_pbr::mesh_view_bindings    view
-#import bevy_pbr::pbr_types             STANDARD_MATERIAL_FLAGS_DOUBLE_SIDED_BIT
+#import bevy_pbr::pbr_types             STANDARD_MATERIAL_FLAGS_DOUBLE_SIDED_BIT, PbrInput, pbr_input_new
 #import bevy_core_pipeline::tonemapping tone_mapping
 #import bevy_pbr::pbr_functions as fns
 
@@ -16,7 +16,7 @@ fn fragment(
 
     // Prepare a 'processed' StandardMaterial by sampling all textures to resolve
     // the material members
-    var pbr_input: fns::PbrInput = fns::pbr_input_new();
+    var pbr_input: PbrInput = pbr_input_new();
 
     pbr_input.material.base_color = textureSample(my_array_texture, my_array_texture_sampler, mesh.uv, layer);
 #ifdef VERTEX_COLORS

diff --git a/crates/bevy_core_pipeline/src/core_3d/main_opaque_pass_3d_node.rs b/crates/bevy_core_pipeline/src/core_3d/main_opaque_pass_3d_node.rs
@@ -1,7 +1,7 @@
 use crate::{
     clear_color::{ClearColor, ClearColorConfig},
     core_3d::{Camera3d, Opaque3d},
-    prepass::{DepthPrepass, MotionVectorPrepass, NormalPrepass},
+    prepass::{DeferredPrepass, DepthPrepass, MotionVectorPrepass, NormalPrepass},
     skybox::{SkyboxBindGroup, SkyboxPipelineId},
 };
 use bevy_ecs::{prelude::*, query::QueryItem};
@@ -34,6 +34,7 @@ impl ViewNode for MainOpaquePass3dNode {
         Option<&'static DepthPrepass>,
         Option<&'static NormalPrepass>,
         Option<&'static MotionVectorPrepass>,
+        Option<&'static DeferredPrepass>,
         Option<&'static SkyboxPipelineId>,
         Option<&'static SkyboxBindGroup>,
         &'static ViewUniformOffset,
@@ -53,12 +54,24 @@ impl ViewNode for MainOpaquePass3dNode {
             depth_prepass,
             normal_prepass,
             motion_vector_prepass,
+            deferred_prepass,
             skybox_pipeline,
             skybox_bind_group,
             view_uniform_offset,
         ): QueryItem<Self::ViewQuery>,
         world: &World,
     ) -> Result<(), NodeRunError> {
+        let load = if deferred_prepass.is_none() {
+            match camera_3d.clear_color {
+                ClearColorConfig::Default => LoadOp::Clear(world.resource::<ClearColor>().0.into()),
+                ClearColorConfig::Custom(color) => LoadOp::Clear(color.into()),
+                ClearColorConfig::None => LoadOp::Load,
+            }
+        } else {
+            // If the deferred lighting pass has run, don't clear again in this pass.
+            LoadOp::Load
+        };
+
         // Run the opaque pass, sorted front-to-back
         // NOTE: Scoped to drop the mutable borrow of render_context
         #[cfg(feature = "trace")]
@@ -69,23 +82,17 @@ impl ViewNode for MainOpaquePass3dNode {
             label: Some("main_opaque_pass_3d"),
             // NOTE: The opaque pass loads the color
             // buffer as well as writing to it.
-            color_attachments: &[Some(target.get_color_attachment(Operations {
-                load: match camera_3d.clear_color {
-                    ClearColorConfig::Default => {
-                        LoadOp::Clear(world.resource::<ClearColor>().0.into())
-                    }
-                    ClearColorConfig::Custom(color) => LoadOp::Clear(color.into()),
-                    ClearColorConfig::None => LoadOp::Load,
-                },
-                store: true,
-            }))],
+            color_attachments: &[Some(
+                target.get_color_attachment(Operations { load, store: true }),
+            )],
             depth_stencil_attachment: Some(RenderPassDepthStencilAttachment {
                 view: &depth.view,
                 // NOTE: The opaque main pass loads the depth buffer and possibly overwrites it
                 depth_ops: Some(Operations {
                     load: if depth_prepass.is_some()
                         || normal_prepass.is_some()
                         || motion_vector_prepass.is_some()
+                        || deferred_prepass.is_some()
                     {
                         // if any prepass runs, it will generate a depth buffer so we should use it,
                         // even if only the normal_prepass is used.

diff --git a/crates/bevy_core_pipeline/src/core_3d/mod.rs b/crates/bevy_core_pipeline/src/core_3d/mod.rs
@@ -10,6 +10,9 @@ pub mod graph {
     pub mod node {
         pub const MSAA_WRITEBACK: &str = "msaa_writeback";
         pub const PREPASS: &str = "prepass";
+        pub const DEFERRED_PREPASS: &str = "deferred_prepass";
+        pub const COPY_DEFERRED_LIGHTING_ID: &str = "copy_deferred_lighting_id";
+        pub const END_PREPASSES: &str = "end_prepasses";
         pub const START_MAIN_PASS: &str = "start_main_pass";
         pub const MAIN_OPAQUE_PASS: &str = "main_opaque_pass";
         pub const MAIN_TRANSPARENT_PASS: &str = "main_transparent_pass";
@@ -24,13 +27,16 @@ pub mod graph {
 }
 pub const CORE_3D: &str = graph::NAME;
 
+// PERF: vulkan docs recommend using 24 bit depth for better performance
+pub const CORE_3D_DEPTH_FORMAT: TextureFormat = TextureFormat::Depth32Float;
+
 use std::{cmp::Reverse, ops::Range};
 
 pub use camera_3d::*;
 pub use main_opaque_pass_3d_node::*;
 pub use main_transparent_pass_3d_node::*;
 
-use bevy_app::{App, Plugin};
+use bevy_app::{App, Plugin, PostUpdate};
 use bevy_ecs::prelude::*;
 use bevy_render::{
     camera::{Camera, ExtractedCamera},
@@ -50,12 +56,17 @@ use bevy_render::{
     view::ViewDepthTexture,
     Extract, ExtractSchedule, Render, RenderApp, RenderSet,
 };
-use bevy_utils::{nonmax::NonMaxU32, FloatOrd, HashMap};
+use bevy_utils::{nonmax::NonMaxU32, tracing::warn, FloatOrd, HashMap};
 
 use crate::{
+    deferred::{
+        copy_lighting_id::CopyDeferredLightingIdNode, node::DeferredGBufferPrepassNode,
+        AlphaMask3dDeferred, Opaque3dDeferred, DEFERRED_LIGHTING_PASS_ID_FORMAT,
+        DEFERRED_PREPASS_FORMAT,
+    },
     prepass::{
-        node::PrepassNode, AlphaMask3dPrepass, DepthPrepass, MotionVectorPrepass, NormalPrepass,
-        Opaque3dPrepass, ViewPrepassTextures, DEPTH_PREPASS_FORMAT, MOTION_VECTOR_PREPASS_FORMAT,
+        node::PrepassNode, AlphaMask3dPrepass, DeferredPrepass, DepthPrepass, MotionVectorPrepass,
+        NormalPrepass, Opaque3dPrepass, ViewPrepassTextures, MOTION_VECTOR_PREPASS_FORMAT,
         NORMAL_PREPASS_FORMAT,
     },
     skybox::SkyboxPlugin,
@@ -69,7 +80,8 @@ impl Plugin for Core3dPlugin {
     fn build(&self, app: &mut App) {
         app.register_type::<Camera3d>()
             .register_type::<Camera3dDepthLoadOp>()
-            .add_plugins((SkyboxPlugin, ExtractComponentPlugin::<Camera3d>::default()));
+            .add_plugins((SkyboxPlugin, ExtractComponentPlugin::<Camera3d>::default()))
+            .add_systems(PostUpdate, check_msaa);
 
         let render_app = match app.get_sub_app_mut(RenderApp) {
             Ok(render_app) => render_app,
@@ -82,6 +94,8 @@ impl Plugin for Core3dPlugin {
             .init_resource::<DrawFunctions<Transparent3d>>()
             .init_resource::<DrawFunctions<Opaque3dPrepass>>()
             .init_resource::<DrawFunctions<AlphaMask3dPrepass>>()
+            .init_resource::<DrawFunctions<Opaque3dDeferred>>()
+            .init_resource::<DrawFunctions<AlphaMask3dDeferred>>()
             .add_systems(ExtractSchedule, extract_core_3d_camera_phases)
             .add_systems(ExtractSchedule, extract_camera_prepass_phase)
             .add_systems(
@@ -92,6 +106,8 @@ impl Plugin for Core3dPlugin {
                     sort_phase_system::<Transparent3d>.in_set(RenderSet::PhaseSort),
                     sort_phase_system::<Opaque3dPrepass>.in_set(RenderSet::PhaseSort),
                     sort_phase_system::<AlphaMask3dPrepass>.in_set(RenderSet::PhaseSort),
+                    sort_phase_system::<Opaque3dDeferred>.in_set(RenderSet::PhaseSort),
+                    sort_phase_system::<AlphaMask3dDeferred>.in_set(RenderSet::PhaseSort),
                     prepare_core_3d_depth_textures.in_set(RenderSet::PrepareResources),
                     prepare_prepass_textures.in_set(RenderSet::PrepareResources),
                 ),
@@ -101,6 +117,15 @@ impl Plugin for Core3dPlugin {
         render_app
             .add_render_sub_graph(CORE_3D)
             .add_render_graph_node::<ViewNodeRunner<PrepassNode>>(CORE_3D, PREPASS)
+            .add_render_graph_node::<ViewNodeRunner<DeferredGBufferPrepassNode>>(
+                CORE_3D,
+                DEFERRED_PREPASS,
+            )
+            .add_render_graph_node::<ViewNodeRunner<CopyDeferredLightingIdNode>>(
+                CORE_3D,
+                COPY_DEFERRED_LIGHTING_ID,
+            )
+            .add_render_graph_node::<EmptyNode>(CORE_3D, END_PREPASSES)
             .add_render_graph_node::<EmptyNode>(CORE_3D, START_MAIN_PASS)
             .add_render_graph_node::<ViewNodeRunner<MainOpaquePass3dNode>>(
                 CORE_3D,
@@ -118,6 +143,9 @@ impl Plugin for Core3dPlugin {
                 CORE_3D,
                 &[
                     PREPASS,
+                    DEFERRED_PREPASS,
+                    COPY_DEFERRED_LIGHTING_ID,
+                    END_PREPASSES,
                     START_MAIN_PASS,
                     MAIN_OPAQUE_PASS,
                     MAIN_TRANSPARENT_PASS,
@@ -341,12 +369,14 @@ pub fn extract_camera_prepass_phase(
                 Option<&DepthPrepass>,
                 Option<&NormalPrepass>,
                 Option<&MotionVectorPrepass>,
+                Option<&DeferredPrepass>,
             ),
             With<Camera3d>,
         >,
     >,
 ) {
-    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass) in cameras_3d.iter()
+    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass, deferred_prepass) in
+        cameras_3d.iter()
     {
         if camera.is_active {
             let mut entity = commands.get_or_spawn(entity);
@@ -361,6 +391,13 @@ pub fn extract_camera_prepass_phase(
                 ));
             }
 
+            if deferred_prepass.is_some() {
+                entity.insert((
+                    RenderPhase::<Opaque3dDeferred>::default(),
+                    RenderPhase::<AlphaMask3dDeferred>::default(),
+                ));
+            }
+
             if depth_prepass.is_some() {
                 entity.insert(DepthPrepass);
             }
@@ -370,6 +407,9 @@ pub fn extract_camera_prepass_phase(
             if motion_vector_prepass.is_some() {
                 entity.insert(MotionVectorPrepass);
             }
+            if deferred_prepass.is_some() {
+                entity.insert(DeferredPrepass);
+            }
         }
     }
 }
@@ -428,8 +468,7 @@ pub fn prepare_core_3d_depth_textures(
                     mip_level_count: 1,
                     sample_count: msaa.samples(),
                     dimension: TextureDimension::D2,
-                    // PERF: vulkan docs recommend using 24 bit depth for better performance
-                    format: TextureFormat::Depth32Float,
+                    format: CORE_3D_DEPTH_FORMAT,
                     usage,
                     view_formats: &[],
                 };
@@ -445,6 +484,22 @@ pub fn prepare_core_3d_depth_textures(
     }
 }
 
+// Disable MSAA and warn if using deferred rendering
+pub fn check_msaa(
+    mut msaa: ResMut<Msaa>,
+    deferred_views: Query<Entity, (With<Camera>, With<DeferredPrepass>)>,
+) {
+    if !deferred_views.is_empty() {
+        match *msaa {
+            Msaa::Off => (),
+            _ => {
+                warn!("MSAA is incompatible with deferred rendering and has been disabled.");
+                *msaa = Msaa::Off;
+            }
+        };
+    }
+}
+
 // Prepares the textures used by the prepass
 pub fn prepare_prepass_textures(
     mut commands: Commands,
@@ -458,6 +513,7 @@ pub fn prepare_prepass_textures(
             Option<&DepthPrepass>,
             Option<&NormalPrepass>,
             Option<&MotionVectorPrepass>,
+            Option<&DeferredPrepass>,
         ),
         (
             With<RenderPhase<Opaque3dPrepass>>,
@@ -467,8 +523,12 @@ pub fn prepare_prepass_textures(
 ) {
     let mut depth_textures = HashMap::default();
     let mut normal_textures = HashMap::default();
+    let mut deferred_textures = HashMap::default();
+    let mut deferred_lighting_id_textures = HashMap::default();
     let mut motion_vectors_textures = HashMap::default();
-    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass) in &views_3d {
+    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass, deferred_prepass) in
+        &views_3d
+    {
         let Some(physical_target_size) = camera.physical_target_size else {
             continue;
         };
@@ -489,7 +549,7 @@ pub fn prepare_prepass_textures(
                         mip_level_count: 1,
                         sample_count: msaa.samples(),
                         dimension: TextureDimension::D2,
-                        format: DEPTH_PREPASS_FORMAT,
+                        format: CORE_3D_DEPTH_FORMAT,
                         usage: TextureUsages::COPY_DST
                             | TextureUsages::RENDER_ATTACHMENT
                             | TextureUsages::TEXTURE_BINDING,
@@ -544,10 +604,56 @@ pub fn prepare_prepass_textures(
                 .clone()
         });
 
+        let cached_deferred_texture = deferred_prepass.is_some().then(|| {
+            deferred_textures
+                .entry(camera.target.clone())
+                .or_insert_with(|| {
+                    texture_cache.get(
+                        &render_device,
+                        TextureDescriptor {
+                            label: Some("prepass_deferred_texture"),
+                            size,
+                            mip_level_count: 1,
+                            sample_count: 1,
+                            dimension: TextureDimension::D2,
+                            format: DEFERRED_PREPASS_FORMAT,
+                            usage: TextureUsages::RENDER_ATTACHMENT
+                                | TextureUsages::TEXTURE_BINDING,
+                            view_formats: &[],
+                        },
+                    )
+                })
+                .clone()
+        });
+
+        let deferred_lighting_pass_id_texture = deferred_prepass.is_some().then(|| {
+            deferred_lighting_id_textures
+                .entry(camera.target.clone())
+                .or_insert_with(|| {
+                    texture_cache.get(
+                        &render_device,
+                        TextureDescriptor {
+                            label: Some("deferred_lighting_pass_id_texture"),
+                            size,
+                            mip_level_count: 1,
+                            sample_count: 1,
+                            dimension: TextureDimension::D2,
+                            format: DEFERRED_LIGHTING_PASS_ID_FORMAT,
+                            usage: TextureUsages::RENDER_ATTACHMENT
+                                | TextureUsages::TEXTURE_BINDING,
+                            view_formats: &[],
+                        },
+                    )
+                })
+                .clone()
+        });
+
         commands.entity(entity).insert(ViewPrepassTextures {
             depth: cached_depth_texture,
             normal: cached_normals_texture,
             motion_vectors: cached_motion_vectors_texture,
+            deferred: cached_deferred_texture,
+            deferred_lighting_pass_id: deferred_lighting_pass_id_texture,
             size,
         });
     }

diff --git a/crates/bevy_core_pipeline/src/deferred/copy_deferred_lighting_id.wgsl b/crates/bevy_core_pipeline/src/deferred/copy_deferred_lighting_id.wgsl
@@ -0,0 +1,19 @@
+#import bevy_pbr::utils
+#import bevy_core_pipeline::fullscreen_vertex_shader FullscreenVertexOutput
+
+@group(0) @binding(0)
+var material_id_texture: texture_2d<u32>;
+
+struct FragmentOutput {
+    @builtin(frag_depth) frag_depth: f32,
+
+}
+
+@fragment
+fn fragment(in: FullscreenVertexOutput) -> FragmentOutput {
+    var out: FragmentOutput;
+    // Depth is stored as unorm, so we are dividing the u8 by 255.0 here.
+    out.frag_depth = f32(textureLoad(material_id_texture, vec2<i32>(in.position.xy), 0).x) / 255.0;
+    return out;
+}
+