Unexpected render item behavior for 'casts shadows'

[csyshing]

Describe the bug

Hi,

We received a feature request from production to see if we can toggle "casts shadows" for USD geometries in the same way as native Maya geometries, since proxy render delegate internally creates MRenderItem, we were experimenting such possibility by setting MRenderItem::castsShadows() , it seems straightforward if we make the changes in two places in lib/mayaUsd/render/vp2RenderDelegate/mayaPrimCommon.cpp (see attached diff files for the full changes):

1. MayaUsdRPrim::_InitRenderItemCommon(): set MRenderItem::castsShadows() when initialize the render item;
2. MayaUsdRPrim::_SyncSharedData(): keep tracking of primvar dirtiness to toggle the "casts shadows" value for corresponding render item.

The "casts shadows" attribute is exposed as an additional primvar named "primvars:visibility:castsShadows" (see attached USD file CORRECT_casts_shadows_off_two_groups.usda for the full content), we created a directional light, turned on "all lights" and "shadow" on the viewport, everything seems working as expected:

However, when we started to test production scene, something looks fishy, for example the above example has two groups of planes, the left & right groups, if we have them separated in different USD files, toggling the primvar does not change anything, it remains as initial state ("castsShadows" == false):

The left group (see attached USD file INCORRECT_casts_shadows_off_left_group_only.usda) - looks incorrect:

The right group (see attached USD file INCORRECT_casts_shadows_off_right_group_only.usda) - looks incorrect:

By having further tests, it looks like if foreground plane is smaller than background plane (see attached USD file CORRECT_casts_shadows_off_fg_smaller_than_bg.usda) , it works as expected:

But if foreground plane is larger than background plane (see attached USD file INCORRECT_casts_shadows_off_fg_larger_than_bg.usda) , it DOES NOT work as expected:

When debugging, we accidently found something even confusing, if we leave MayaUsdRPrim::_InitRenderItemCommon() untouched (which expected to see inconsistent result at init) , keeps the changes in MayaUsdRPrim::_SyncSharedData() (which expected to see inconsistent result at the first time when toggling the primvar), and the further toggling works as expected. For example the previous INCORRECT_casts_shadows_off_fg_larger_than_bg USD file now behaves as expected:

We can't reproduce the issue if they were in native Maya geometries, so it looks like related to MayaUSD only.
What did we miss? I feel like there is a bug somewhere but not sure where it is.
I attached our changes but I could submit a draft PR if that's easier to review - let me know!

Steps to reproduce
Steps to reproduce the behavior:

1. Apply the attached file casts_shadows_changes.diff.txt to get the cast shadows reacts on toggling
2. Unzip attached example USD files
3. Load one of the example USD files in Maya
4. Turn on "Usa all lights" and "Shadows" on the viewport
5. Observe the expected / unexpected shadows behaviors

Expected behavior
MayaUSD could properly supports "casts shadows".

Attachments

• GIFs - see above inline images
• Example USD files:
  casts_shadows_off_example_files.zip
• Code changes in diff format:
  casts_shadows_changes.diff.txt

Specs (if applicable):

• OS & version : CentOS 7.8
• Compiler & version : gcc 9.3
• Maya version: tested against Maya-2023.2.x - 2023.3.3
• Maya USD commit SHA: release/v0.30.0 branch b3d4a06 + custom changes
• Pixar USD commit SHA: official v24.08 / v22.05

[wallworm]

Thanks for reporting. We will discuss this one internally and get back to you.

[pierrebai-adsk]

Note: I'm not the expert of VP2 delegate, but one possible issue I see with your change: in the _Init, you reuse the EnqueueCommit, but other values set in _Init are set directly without Enqueue. Maybe try initializing the render item directly instead of post-poning it with Enqueue?

[pierrebai-adsk]

I tried my suggested change and it worked for me. In _InitRenderItemCommon, I wrote:

bool isCastsShadows = true;
if (getCastsShadows(_delegate, SdfPath(_rprimId.asChar()), isCastsShadows)) {
        renderItem->castsShadows(isCastsShadows);
        renderItem->receivesShadows(isCastsShadows);
}

[csyshing]

Thanks @pierrebai-adsk for the suggestion.

I tried to remove the Enqueue from the _InitRenderItemCommon(), tested the two examples (INCORRECT_casts_shadows_off_left_group_only.usda, INCORRECT_casts_shadows_off_fg_larger_than_bg.usda) but still cannot get the shadow working as expected when toggling the primvar.

I did another test to remove the Enqueue from _SyncSharedData() and Maya crashes due to race condition (multi threading), I put renderItem->castsShadows() in Enqueue at _InitRenderItemCommon() was for the same reason, MayaUsdRPrim might be run in secondary thread from caller HdRenderIndex::SyncAll(), thought that could avoid the multi threading issue.

But regardless of the crash, removing the Enqueue still does not work as expected, may I ask which example scene you have used for your testing? Wanted to see if I miss something.

[pierrebai-adsk]

You should definitely not remove Enqueue from Sync, only from Init. Sync is meant to run in multiple parallel threads. For testing, I used the scenes you provided that are attached to this issue. I do not understand why my suggested changes work for me but not for you. I used the exact code diff you provided and only changed what I quoted.

Here is my branch that I used to test: #3997

I'll retest again right now. In case it is relevant, I'm on Windows 10, nvidia graphics, Maya 2025.3.

[pierrebai-adsk]

I restested and it does work for me. (I forgot to turn on the cast shadow in Maya 2 minutes ago and thought it did not work.

See video:

Test.Cast.Shadow.mp4

[csyshing]

Appreciate @pierrebai-adsk ! Let me test again to see if there is any diff compared with our internal (production) code base.

[csyshing]

Hi @pierrebai-adsk , I rebuilt maya-usd v0.30.0 and applied exactly the same changes in this issue, ran a vanilla Maya 2023.3.3-32.0 environment without any production setup, I still cannot get any INCORRECT_* USD scene working.

However, if I import multiple incorrect USD files into Maya, casting shadows somehow started to work .......

@pierrebai-adsk , instead of importing multiple USD file in one Maya session, have you tried to import just ONE file (one of the INCORRECT_* files), does that working as expected on your end?

Here is the repro steps on my end (for the following video):

1. Launch Maya
2. Import the INCORRECT_casts_shadows_off_fg_larger_than_bg.usda file, it DOES NOT work as expected
3. Keep the scene opened, import the INCORRECT_casts_shadows_off_left_group_only.usda file, this one starts to work as expected
4. Keep the scene opened, import the INCORRECT_casts_shadows_off_right_group_only.usda file, this one starts to work as expected as well
5. Go back to click the large foreground plane -- it also starts to working as expected.

This is very confusing. I tried to revert the changes to my originally version (with Enqueue at init), actually it behaves exactly the same as without Enqueue :-(

[pierrebai-adsk]

When you say import, you mean opening the file as a stage, correct? When generally call this "staging a file" internally.

It does work for a single file. Maybe the difference is that I'm working in Maya 2025.3.

[csyshing]

I see. I won't be able to test in Maya 2024/2025 any time soon but will keep that in mind and keep you posted. Thanks again for your time!

[pierrebai-adsk]

Sorry to have misled you, indeed if I start off with the file you mentioned, it does not cast shadow right away! So the problem is also in 2025. Except in my case, the large-than-bg starts to work as soon as I load a 2nd stage,

[pierrebai-adsk]

I can also make it work without loading a 2nd file by calling the "ogs -reset" MEL command. There seems to be other manipulations that also make it work without loading any other stage. Must be anything that causes a recompute of the scene delegate. For example, after turning using light on and shadow, merely selecting the directional light makes shadows appear.

That may be why I initially thought it worked right away. Some seemingly unrelated manipulations make the shadows work right away.

[csyshing]

I tested "ogs -reset" and selecting/deselecting the light in Maya 2023, sometimes works sometimes not, haven't figured out the pattern yet.

I am not a VP2 expert, do you see any obvious problem for my changes that might not trigger the viewport update correctly (when toggling the casts shadow value)? Or this could be considered a bug somewhere else?

[pierrebai-adsk]

I'm no VP2 expert, unfortunately. VP2 and OGS (the internal Autodesk graphics framework used by VP2) are very complex.