Throwing an error from inside a kernel

[jagoosw]

Hi all,

I was wondering if you had any advice about how to throw an error from inside a kernel that would work on any architecture? And would it be something that I could implement in KA?

Since the normal error throwing doesn't work in GPU kernels it would probably need to be like KernelAbstractions.@print and call @error on CPU and perhaps @assert false on GPU since I can't see a different way to intentionally crash CUDA kernels?

[vchuravy]

Since the normal error throwing doesn't work in GPU kernels

Normal kernel error throwing works on GPUs?

[vchuravy]

julia> using AMDGPU

julia> function f(cond)
          if cond
             error("It failed!")
          end
       end
f (generic function with 1 method)

julia> @roc f(false)
AMDGPU.Runtime.HIPKernel{typeof(f), Tuple{Bool}}(f, AMDGPU.HIP.HIPFunction(Ptr{Nothing} @0x000000002e91ac90, AMDGPU.HIP.HIPModule(Ptr{Nothing} @0x000000002a249fa0), Symbol[]))

julia> @roc f(true)
AMDGPU.Runtime.HIPKernel{typeof(f), Tuple{Bool}}(f, AMDGPU.HIP.HIPFunction(Ptr{Nothing} @0x000000002e91ac90, AMDGPU.HIP.HIPModule(Ptr{Nothing} @0x000000002a249fa0), Symbol[]))

julia> AMDGPU.synchronize()
ERROR: GPU Kernel Exception
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] throw_if_exception(dev::HIPDevice)
   @ AMDGPU ~/.julia/packages/AMDGPU/jLWP2/src/exception_handler.jl:123
 [3] synchronize(stm::HIPStream; blocking::Bool, stop_hostcalls::Bool)
   @ AMDGPU ~/.julia/packages/AMDGPU/jLWP2/src/highlevel.jl:40
 [4] synchronize(stm::HIPStream)
   @ AMDGPU ~/.julia/packages/AMDGPU/jLWP2/src/highlevel.jl:36
 [5] top-level scope
   @ REPL[8]:1

CUDA will actually tell you the error in most cases. So AMDGPU "ought" to be fixed to tell you more than "just error occured".

[jagoosw]

In the case above I get:

julia> @cuda f(true)
CUDA.HostKernel for f(Bool)

ERROR: a exception was thrown during kernel execution on thread (1, 1, 1) in block (1, 1, 1).
Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable).

and when I use level 2 debugging I just get a bit more info:

CUDA.HostKernel for f(Bool)

ERROR: a exception was thrown during kernel execution on thread (1, 1, 1) in block (1, 1, 1).
Stacktrace:
 [1] error at ./error.jl:35
 [2] f at ./REPL[4]:1

[vchuravy]

There is some special handling for bounds errors, but I think it's generally hard to get the information from the GPU to the CPU.

JuliaGPU/CUDA.jl#2485

But if it is hard for CUDA.jl it is hard for KernelAbstractions.jl

[jagoosw]

Wouldn't @print be sufficient here? for example:

julia> f(cond) = if cond @print "An error is about to occur"; throw("an error") end

julia> f(true)
An error is about to occur
ERROR: "an error"
Stacktrace:
 [1] f(cond::Bool)
   @ Main ./REPL[4]:1
 [2] top-level scope
   @ REPL[5]:1

julia> @cuda f(true)
CUDA.HostKernel for f(Bool)

julia> An error is about to occur
ERROR: a exception was thrown during kernel execution on thread (1, 1, 1) in block (1, 1, 1).
Stacktrace not available, run Julia on debug level 2 for more details (by passing -g2 to the executable).

so it still crashes the GPU but at least we can tell the user some information before.

And my thinking was we could wrap this up like:

function kernel_error(msg)
    @print msg
    throw(msg)
end

and then it would hopefully exfiltrate some information on whatever platforms KA already supports for @print.

[vchuravy]

Sure! You could add a macro:

macro device_error(args....)
     esc(quote
            @print $(args....)
            throw(DeviceError())
     end)
end

Or something similar.