Is automatic copying to host in AOT outputs introduced in Halide 14?

[dragly]

In Halide 13, if I modify lesson 10 in the the python bindings tutorial to use GPU tiling (see patch below) then the output does not seem to be properly copied back to the host. To get the correct output data, I had to modify the generated .py.cpp file to include a call to halide_copy_to_host.

However, when applying the same patch to Halide 14, there is no need for such a call to halide_copy_to_host.

Was there a change between Halide 13 and Halide 14 that automatically ensures the buffer is copied to host after calling an AOT function? There is not call to halide_copy_to_host in the generated .py.cpp, so it must be a change elsewhere.

As a side-note: For performance reasons we have usually been happy that the result is not copied back to host automatically when calling AOT-compiled functions in C++. That allows us to keep the data on the GPU only for as long as possible. However, when working exclusively in Python the new behavior in Halide 14 almost seems necessary, since I could not find any other way to trigger a copy to host.

EDIT: Added patch.

diff --git a/python_bindings/tutorial/lesson_10_aot_compilation_generate.py b/python_bindings/tutorial/lesson_10_aot_compilation_generate.py
index 8aae5d6dd..e75fcc5c3 100644
--- a/python_bindings/tutorial/lesson_10_aot_compilation_generate.py
+++ b/python_bindings/tutorial/lesson_10_aot_compilation_generate.py
@@ -32,6 +32,8 @@ def main():
     # We'll define a simple one-stage pipeline:
     brighter = hl.Func("brighter")
     x, y = hl.Var("x"), hl.Var("y")
+    x_inner, y_inner = hl.Var("x_inner"), hl.Var("y_inner")
+    x_outer, y_outer = hl.Var("x_outer"), hl.Var("y_outer")

     # The pipeline will depend on one scalar parameter.
     offset = hl.Param(hl.UInt(8), name="offset")
@@ -54,7 +56,8 @@ def main():
     brighter[x, y] = input[x, y] + offset

     # Schedule it.
-    brighter.vectorize(x, 16).parallel(y)
+    # brighter.vectorize(x, 16).parallel(y)
+    brighter.gpu_tile(x, y, x_outer, y_outer, x_inner, y_inner, 16, 16)

     # This time, instead of calling brighter.realize(...), which
     # would compile and run the pipeline immediately, we'll call a
@@ -64,9 +67,12 @@ def main():
     # arguments to the routine. This routine takes two. Arguments are
     # usually Params or ImageParams.
     fname = "lesson_10_halide"
-    brighter.compile_to({hl.Output.object: "lesson_10_halide.o",
-                         hl.Output.python_extension: "lesson_10_halide.py.cpp"},
-                        [input, offset], "lesson_10_halide")
+    brighter.compile_to(
+        {hl.Output.object: "lesson_10_halide.o", hl.Output.python_extension: "lesson_10_halide.py.cpp"},
+        [input, offset],
+        "lesson_10_halide",
+        target=hl.get_host_target().with_feature(hl.TargetFeature.OpenCL),
+    )

     print("Halide pipeline compiled, but not yet run.")

[abadams]

There was no such change (at least, not intentionally), so this may be a bug. Maybe @steven-johnson knows if something has changed that might have caused this.

[steven-johnson]

I'm not aware of any such (intentional) change to the Python Buffer code between Halide 13 and Halide 14.

Generally, the intended behavior is to never to a copy to/from host/gpu unless needed -- i.e., it should never pre-emptively copy in either direction until that data is going to be immediately used -- so this sounds like a regression.

(I don't see a "patch below", can you include it?)

[dragly]

Ooops, sorry! I forgot to add the patch. Added it to the original post now.

Thanks for the clarification. Sounds like it could definitely be a regression then. Even though it is nice that the copy happens in the case of the generated Python extension, since a NumPy buffer is of little use with the data still on the GPU. But I guess that should be done in the generated Python code and not elsewhere.

[steven-johnson]

OK, this is interesting: The lesson-10 tutorial just uses a numpy array for the input and output buffers. This is fine and actually super-handy, but (AFAIK) a numpy array can't have any off-host storage, so for correctness, we really should always be doing a copy-to-host when the outputs are going into a numpy array (rather than an explicit hl.Buffer, which can handle the host-vs-device distinction).

[steven-johnson]

For performance reasons we have usually been happy that the result is not copied back to host automatically when calling AOT-compiled functions in C++

So I'm assuming that you used the tutorial here as a convenient example, but the issue is really in "real" code you have. In that "real" code, are you using hl.Buffer explicitly, or are you using numpy arrays (or something similar)?

EDIT: hl.Buffer is a compile-time-only thing -- oops -- and our Python bindings don't currently have anything analogous to Halide::Runtime::Buffer for managing this more intelligently. Opening an issue on it.

[steven-johnson]

#6868

[dragly]

Great, thanks for looking into this! I also concluded that calling copy_to_host in the generated Python seemed like the right thing to do (as long as the output is a NumPy array).