Error: Invalid Tensor Format "fp32" Causing Training Process to Stop on Apple Silicon

[Ryohei96]

Hello.

I am encountering an error that causes the training process to stop prematurely. Here are the details of the error message:

warning: loc("mps_resize"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/91a344b1-f985-11ee-b563-fe8bc7981bff/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":180:0)): 'anec.resize' op Invalid configuration for the following reasons: Error: Invalid tensor format: "fp32"

Here are the details of my environment:

SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-14.5-arm64-arm-64bit
The processor is Apple Silicon.

Could you please provide guidance on how to resolve this issue?

Thank you for your assistance.

[FangyuanLinGoBears2024]

I'm literally have the same issue. I think the a lot of things on the website are outdated.

[talmo]

We're working on updating the website -- are there specific pages with outdated information you've noticed? (Guessing this comment is referring to the installation page.)

[talmo]

Hi folks,

Apologies for the inconvenience! This is an issue with tensorflow-metal and Mac OS (Sonoma?) compatibility.

We don't have much control over those dependencies and it seems like we're not the only ones affected by this issue.

Unfortunately we don't have a workaround at the moment, but will be looking into a short term fix while we work towards our migration to Pytorch, which will hopefully circumvent all of the problems we've been having with TensorFlow.

Hopefully more soon!

Talmo

[Ryohei96]

Hi Talmo,

Thank you for addressing this issue.

I found a workaround by using the command sleap-label --reset. However, sometimes it encounters errors and results in a forced shutdown.

I look forward to your updates.

Best regards,
Ryohei

[talmo]

Hmm, sleap-label --reset should only affect the GUI and nothing in the backend/tensorflow, so this might indicate that it's an intermittent issue (ugh).

In any case, thanks for letting us know! Keep us posted on whether you run into any other errors and we'll let you know if we have a fix for the short term.