Fix test_python matrix (#1579)

This fixes the matrix for `test_python` to allow for CUDA 11.4. Fixes a testing regression from #1542.

Authors:
  - Kyle Edwards (https://github.com/KyleFromNVIDIA)

Approvers:
  - Bradley Dice (https://github.com/bdice)
  - Jake Awe (https://github.com/AyodeAwe)

URL: #1579

dependencies.yaml

@@ -294,21 +294,15 @@ dependencies:
         packages:
           - pytest
           - pytest-cov
-    # Needed for numba in tests
     specific:
       - output_types: conda
         matrices:
-          - matrix:
-              arch: x86_64
-              cuda: "11.8"
-            packages:
-              - nvcc_linux-64=11.8
-          - matrix:
-              arch: aarch64
-              cuda: "11.8"
-            packages:
-              - nvcc_linux-aarch64=11.8
+          # Needed for numba in tests on CUDA 12+ since Conda CI environments
+          # do not have a CUDA compiler preinstalled. Wheel tests are run on
+          # images that supply a CUDA compiler.
           - matrix:
               cuda: "12.*"
             packages:
               - cuda-nvcc
+          - matrix:
+            packages: