RuntimeError

[Jayku88]

Traceback (most recent call last):
File "train.py", line 907, in
main()
File "train.py", line 97, in main
main_worker(args.train_gpu, args.ngpus_per_node, args)
File "train.py", line 415, in main_worker
loss_train, mIoU_train, mAcc_train, allAcc_train = train(train_loader, model, criterion, optimizer, epoch, scaler, scheduler, gpu)
File "train.py", line 519, in train
scaler.scale(loss).backward()
File "/home/vrlabhlbs/anaconda3/envs/spheretest/lib/python3.7/site-packages/torch/tensor.py", line 245, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/vrlabhlbs/anaconda3/envs/spheretest/lib/python3.7/site-packages/torch/autograd/init.py", line 147, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
RuntimeError: transform: failed to synchronize: cudaErrorAssert: device-side assert triggered

[Jaywxy]

I have got the same error