CUDA out of memory

[DJC666666]

When I tried to create scene graphs, it showed 'CUDA out of memory'. Is there any good solution?

~/HOV-SG$ python application/create_graph.py main.dataset=hm3dsem main.dataset_path=data/hm3dsem_walks/val/00824-Dd4bFSTQ8gi/ main.save_path=data/scene_graphs/00824-Dd4bFSTQ8gi
[2024-07-23 11:58:29,144][root][INFO] - Loaded ViT-H-14 model config.
[2024-07-23 11:58:37,214][root][INFO] - Loading pretrained ViT-H-14 weights (checkpoints/laion2b_s32b_b79k.bin).
Creating RGB-D point cloud: 100%|█████████████| 226/226 [00:18<00:00, 12.31it/s]
Extracting features:   0%|                              | 0/226 [00:00<?, ?it/s]
Error executing job with overrides: ['main.dataset=hm3dsem', 'main.dataset_path=data/hm3dsem_walks/val/00824-Dd4bFSTQ8gi/', 'main.save_path=data/scene_graphs/00824-Dd4bFSTQ8gi']
Traceback (most recent call last):
  File "/home/guojb/HOV-SG/application/create_graph.py", line 19, in main
    hovsg.create_feature_map() # create feature map
  File "/home/guojb/HOV-SG/hovsg/graph/graph.py", line 177, in create_feature_map
    F_2D, F_masks, masks, F_g = extract_feats_per_pixel(
  File "/home/guojb/HOV-SG/hovsg/models/sam_clip_feats_extractor.py", line 102, in extract_feats_per_pixel
    masks = mask_generator.generate(image)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/automatic_mask_generator.py", line 163, in generate
    mask_data = self._generate_masks(image)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/automatic_mask_generator.py", line 206, in _generate_masks
    crop_data = self._process_crop(image, crop_box, layer_idx, orig_size)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/automatic_mask_generator.py", line 236, in _process_crop
    self.predictor.set_image(cropped_im)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/predictor.py", line 60, in set_image
    self.set_torch_image(input_image_torch, image.shape[:2])
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/predictor.py", line 89, in set_torch_image
    self.features = self.model.image_encoder(input_image)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/modeling/image_encoder.py", line 112, in forward
    x = blk(x)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/modeling/image_encoder.py", line 174, in forward
    x = self.attn(x)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/modeling/image_encoder.py", line 234, in forward
    attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (H, W), (H, W))
  File "/home/guojb/anaconda3/envs/hovsg/lib/python3.9/site-packages/segment_anything/modeling/image_encoder.py", line 358, in add_decomposed_rel_pos
    attn.view(B, q_h, q_w, k_h, k_w) + rel_h[:, :, :, :, None] + rel_w[:, :, :, None, :]
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

[abwerby]

I will look into this issue, for now, you can change the merge_type to hierarchical, here:

HOV-SG/config/create_graph_config.yaml

Line 35 in 6e75600

merge_type: sequential # hierarchical, sequential

this may solve the problem.

[DJC666666]

Thank you for the solution, but it still does not work.

[2024-07-24 13:54:24,706][root][INFO] - Loaded ViT-H-14 model config.
[2024-07-24 13:54:30,576][root][INFO] - Loading pretrained ViT-H-14 weights (checkpoints/laion2b_s32b_b79k.bin).
Creating RGB-D point cloud: 100%|█████████████| 226/226 [00:15<00:00, 14.63it/s]
Extracting features:   0%|                              | 0/226 [00:00<?, ?it/s]
CUDA out of memory: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU