USD threading problem with arnold batch render

[Sudarshanhavale]

Hello Guys,

I have a Maya scene file with _Arnold standins which are sourcing the animated USD characters from the disk. Just FYI, these USD caches are massive in size and there are multiple instances of the same aiStandIn nodes for crowd purposes.

Now, My problem is the Interface render works fine, but the batch render on the farm keeps throwing the below error in the render log on random frames and with random USD files.

2023-09-08 16:14:12 - Warning (secondary thread): in _ReportErrors at line 2841 of S:\tmp\arnold-kook\usd\build\usd-21.08_static_windows_vc-14.2_cxx14\pxr\usd\usd\stage.cpp -- Could not load sublayer @orca_02_cache_ani_anim_primary_def_hi_v006__stage.usd@ of layer @j:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/orca_02_cache_primary_def_hi/v006/usd/orca_02_cache_ani_anim_primary_def_hi_v006.usd@; skipping. (instantiating stage on stage @j:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/orca_02_cache_primary_def_hi/v006/usd/orca_02_cache_ani_anim_primary_def_hi_v006.usd@ <000002B29B9C2F70>)
2023-09-08 16:14:12 - Warning (secondary thread): in _ReportErrors at line 2841 of S:\tmp\arnold-kook\usd\build\usd-21.08_static_windows_vc-14.2_cxx14\pxr\usd\usd\stage.cpp -- Could not load sublayer @orca_02_cache_ani_anim_primary_def_hi_v006__geom.usd@ of layer @j:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/orca_02_cache_primary_def_hi/v006/usd/orca_02_cache_ani_anim_primary_def_hi_v006.usd@; skipping. (instantiating stage on stage @j:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/orca_02_cache_primary_def_hi/v006/usd/orca_02_cache_ani_anim_primary_def_hi_v006.usd@ <000002B29B9C2F70>)
2023-09-08 16:14:12 - Runtime Error (secondary thread): in _MmapAsset at line 2159 of S:\tmp\arnold-kook\usd\build\usd-21.08_static_windows_vc-14.2_cxx14\pxr\usd\usd\crateFile.cpp -- Couldn't map asset 'J:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/humpbackWhale_01_cache_primary_def_hi/v004/usd/humpbackWhale_01_cache_ani_anim_primary_def_hi_v004.usd'
2023-09-08 16:14:12 - 00:01:26  1851MB ERROR   |  [usd] Failed to open file (J:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/humpbackWhale_01_cache_primary_def_hi/v004/usd/humpbackWhale_01_cache_ani_anim_primary_def_hi_v004.usd)
2023-09-08 16:14:12 - Runtime Error (secondary thread): in _MmapAsset at line 2159 of S:\tmp\arnold-kook\usd\build\usd-21.08_static_windows_vc-14.2_cxx14\pxr\usd\usd\crateFile.cpp -- Couldn't map asset 'J:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/tigerShark_01_cache_primary_def_hi/v006/usd/tigerShark_01_cache_ani_anim_primary_def_hi_v006.usd'
2023-09-08 16:14:12 - 00:01:26  1852MB ERROR   |  [usd] Failed to open file (J:/<showName>/sequences/aq003/aq003_0010/publish/cache/ani/anim/tigerShark_01_cache_primary_def_hi/v006/usd/tigerShark_01_cache_ani_anim_primary_def_hi_v006.usd)

We are stuck with show deliveries because of this problem so can someone please help to fix this?

Note: The reason for massive USD files is the really long frame range per shot.

Thanks,
Sudarshan

[Sudarshanhavale]

For now, I am turning off the parallel_node_init using arnold render settings to render out the frames but a permanent solution might help to increase render efficiency.

[BigRoy]

Just FYI, these USD caches are massive in size and there are multiple instances of the same aiStandIn nodes for crowd purposes.

You might benefit from using USD value clips here (I believe also sometimes referred to as clip stitching?) to ensure only related data is loaded at a given point in time.

See:

• https://openusd.org/release/glossary.html#USDGlossary-ValueClips
• https://openusd.org/dev/api/_usd__page__value_clips.html