cudaErrorInvalidDevice: invalid device ordinal

[rshahrabani]

We have the following environment:

spaCy version 3.7.4
Location C:\Program Files\Python\Lib\site-packages\spacy
Platform Windows-2022Server-10.0.20348-SP0
Python version 3.12.3
Pipelines en_core_web_lg (3.7.1), en_core_web_md (3.7.1), en_core_web_sm (3.7.1), en_core_web_trf (3.7.3)

We've installed CUDA v12.4 and the GPU is set up as follows:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 551.78 Driver Version: 551.78 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 TCC | 00000001:00:00.0 Off | 0 |
| N/A 35C P8 13W / 70W | 9MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

We get the error "cudaErrorInvalidDevice: invalid device ordinal' back from spaCy when executing the following line of code:

doc = nlp(text)

We have set spaCy to use the gpu:

spacy.prefer_gpu()

Any ideas as to why this error is occurring and how we can resolve it?

Apologies if this was posted in the wrong forum.

Thanks in advance.

[GiovanniNova]

We started to get this same error after a restart today

Spacy 3.7.4
Windows 10 Pro 22H2
Python 3.9.6

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 552.12                 Driver Version: 552.12         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060      WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   31C    P8             10W /  170W |    4339MiB /  12288MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

  File "c:\users\workstation\appdata\local\programs\python\python39\lib\site-packages\spacy_transformers\layers\trfs2arrays.py", line 58, in forward
    dst, get_d_src = apply_alignment(model.ops, trf_data.align, src)
  File "spacy_transformers\\align.pyx", line 55, in spacy_transformers.align.apply_alignment
  File "cupy\_core\core.pyx", line 1521, in cupy._core.core._ndarray_base.__getitem__
  File "cupy\_core\_routines_indexing.pyx", line 42, in cupy._core._routines_indexing._ndarray_getitem
  File "cupy\_core\core.pyx", line 829, in cupy._core.core._ndarray_base.take
  File "cupy\_core\_routines_indexing.pyx", line 131, in cupy._core._routines_indexing._ndarray_take
  File "cupy\_core\_routines_indexing.pyx", line 825, in cupy._core._routines_indexing._take
  File "cupy\_core\_kernel.pyx", line 869, in cupy._core._kernel.ElementwiseKernel.__call__
  File "cupy\_core\_kernel.pyx", line 159, in cupy._core._kernel._preprocess_args
  File "cupy\_core\_kernel.pyx", line 130, in cupy._core._kernel._preprocess_arg
  File "cupy\_core\_kernel.pyx", line 117, in cupy._core._kernel._check_peer_access
  File "cupy\cuda\device.pyx", line 97, in cupy.cuda.device._enable_peer_access
  File "cupy_backends\cuda\api\runtime.pyx", line 385, in cupy_backends.cuda.api.runtime.deviceCanAccessPeer
  File "cupy_backends\cuda\api\runtime.pyx", line 144, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorInvalidDevice: invalid device ordinal

Cuda itself seems to be working just fine, the following code can be used to test:

import tensorflow as tf

tf.debugging.set_log_device_placement(True)

with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

The console output:

2024-04-25 16:38:54.157066: I tensorflow/core/common_runtime/placer.cc:114] input: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.157114: I tensorflow/core/common_runtime/placer.cc:114] _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0
output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.157151: I tensorflow/core/common_runtime/placer.cc:114] output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.417969: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0
2024-04-25 16:38:54.418976: I tensorflow/core/common_runtime/placer.cc:114] input: (_Arg): /job:localhost/replica:0/task:0/device:CPU:0
_EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.419019: I tensorflow/core/common_runtime/placer.cc:114] _EagerConst: (_EagerConst): /job:localhost/replica:0/task:0/device:GPU:0
output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.419057: I tensorflow/core/common_runtime/placer.cc:114] output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.419600: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
tensor: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.420173: I tensorflow/core/common_runtime/placer.cc:114] tensor: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0
2024-04-25 16:38:54.420449: I tensorflow/core/common_runtime/placer.cc:114] shape: (_DeviceArg): /job:localhost/replica:0/task:0/device:CPU:0
Reshape: (Reshape): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.420779: I tensorflow/core/common_runtime/placer.cc:114] Reshape: (Reshape): /job:localhost/replica:0/task:0/device:GPU:0
output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.420909: I tensorflow/core/common_runtime/placer.cc:114] output_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.421605: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.422248: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.422475: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op _EagerConst in device /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.422671: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op Reshape in device /job:localhost/replica:0/task:0/device:GPU:0
a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.423350: I tensorflow/core/common_runtime/placer.cc:114] a: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.423634: I tensorflow/core/common_runtime/placer.cc:114] b: (_Arg): /job:localhost/replica:0/task:0/device:GPU:0
MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.423940: I tensorflow/core/common_runtime/placer.cc:114] MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.424225: I tensorflow/core/common_runtime/placer.cc:114] product_RetVal: (_Retval): /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.424769: I tensorflow/core/common_runtime/eager/execute.cc:1423] Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0
2024-04-25 16:38:54.440949: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.```

[GiovanniNova]

I have figured out why I myself was getting this error, I was running this behind a Flask application, by default Flask uses threading and it seems that even https://spacy.io/api/language#pipe is not thread-safe, so nlp = spacy.load in a app creation and then nlp(text,... or nlp.pipe([text],... on a thread is not going to work.