Memorey issue on inferencing on multi gpu with 4 X NVIDIA A10G with 23gb VRAM using xdit

[dibyajitquilt]

I am having issue in using multi gpu for inference , having 4*NVIDIA A10G with 23gb VRAM , still its getting the error in loading the model with below logs
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

sample_video.py FAILED

Failures:
[1]:
time : 2024-12-11_10:10:29
host : ip-172-31-2-204.us-east-2.compute.internal
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 26222)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-12-11_10:10:29
host : ip-172-31-2-204.us-east-2.compute.internal
rank : 3 (local_rank: 3)
exitcode : 1 (pid: 26223)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
time : 2024-12-11_10:10:29
host : ip-172-31-2-204.us-east-2.compute.internal
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 26221)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

The command i am using as follows
torchrun --nproc_per_node=4 sample_video.py
--video-size 1280 720
--video-length 129
--infer-steps 50
--prompt "A cat walks on the grass, realistic style."
--flow-reverse
--seed 42
--use-cpu-offload
--ulysses-degree 4
--ring-degree 1
--save-path ./results

tried all the combinations of ulysses-degreexring-degree 

[os-sos]

Im interested in doing something similar. Are you using NVlink between these cards? Do you know if shared memory can work?

[os-sos]

Im interested in doing something similar. Are you using NVlink between these cards? Do you know if shared memory can work?

[ftaibi]

same issue