Does the model support bf16 training？

[atomrun39]

--precision
floating-point precision to use during training
Default: 16
Can this parameter in training only be filled in with 16 or 32, representing fp16 and fp32?

[atomrun39]

When I use DDP training and pass in -- precision bf16, it can be trained normally.

However, when using deepspeed, an error will be generated stating 'InstanceError:' weight_norm_fwd_first_im_kernel 'not implemented for' BFloat16 '

How can I solve this?

[BingliangLi]

Hi did you solved it? Does deepspeed faster than DDP?