[GSoC] Julia wrapper

[johnnychen94]

Hi,

My name is Jiuning Chen(Johnny Chen), I'm a second-year Ph.D. student in ECNU(East China Normal University), a GSoC 19 student in Julia, and also a core maintainer of JuliaImages since that. My previous main research interest is low-level image processing (e.g., denoise) using filter techs.

Both Julia and halide have shown their great advantages in image processing and I want to see if I can try to get the best of the two worlds by providing a Julia interface to Halide.

I've learned a lot from the halide paper when I try to optimize my Julia codes but never tried halide; it was quite a long time ago since my last usage of C++. To apply for the Halide Code Generation For More Languages GSoC project, I definitely need some refresh of my C++ knowledge, I'm wondering if you could provide some easy-to-start issues that I can work on to get a better understanding of halide (and of course C++) before I start to write my proposal.

Also, is this issue an appropriate place to update progress, or should I contact you via other channels/apps?

Best regards

cc: @zvookin @rootjalex @BachiLi

[rootjalex]

If you haven't already tried these out, the tutorials are a good way to get more familiar with Halide.

I'm not sure about specific issues that are relevant to you at the moment, but Tzu-Mao recommended trying to understand our PyTorch codegen module as a good starting point.

The gitter is a good communication method that we use.

[rootjalex]

A bit more on this, specific to Julia: after a quick glance over Cxx.jl and CxxWrap.jl, CxxWrap seems to be the more plausible option for embedding (if there are others who are more familiar with Julia, please correct me). This issue links to some examples that may be relevant

[johnnychen94]

I plan to follow a similar route OpenCV-Julia binding project did since it's written in C++, too. So yeah I'll use CxxWrap. But before that, I'll need to try and learn the halide internals to understand it. I'll do a halide-Julia benchmark first in the next few days. Hopefully, I can put my hands on the wrapper project next weekend.

[alexreinking]

only MinGW is available for Windows targets. According to the Halide README.md, it seems that building halide on Windows only supports visual studio 2019, does this mean that building windows artifacts using binarybuilder is not possible?

Correct, we dropped support for MinGW a while back.

[alexreinking]

Now the good news I get so far is that with this specific build setup, it passes all tests except one cross-compilation test, will this be an issue?

Any failing test is an issue.

[alexreinking]

CxxWrap_example is my first attempt to test if I'm making things correctly, and it works.

Btw - before contributing any CMake code to Halide, you should read our README_cmake.md. In particular note our minimum version requirement of 3.16 and absolute prohibition of directory commands like include_directories.

[alexreinking]

I worry about having the project repo as separate from the main repo, mostly due to the need to update it's Halide sub-repository, integrate it's tests with some build system, etc. etc. It's not a huge issue, but I'm curious what the other mentors think.

For anything GSOC, the aim is to introduce first-party support. That means upstreaming and setting up CI on our systems. So a separate repo is fine for rapid development, but it must be integrated into our repository in the end. From a workflow perspective, I would suggest forking Halide and working on a branch in the fork.

[alexreinking]

I deliberately turned off every backend except for X86 but it still says that other backends are enabled. Is this because I'm using a shared LLVM library?

Yes. When linking to shared LLVM, we enable all the backends that are available.

Also, the Halide specific backends OpenCL, Meta and Direct3D 12 Compute are disabled, otherwise, it will throw undefined symbol error during linking.

Can you open an issue about this? Also, can you open an issue about the failing test with the Julia LLVM? Please provide full reproduction steps.

[johnnychen94]

Sorry if I'm asking the wrong question. Does Halide provide a C API? If so it's very possible that we could get rid of CxxWrap dep and thus make things easier.

[alexreinking]

No it does not.

[johnnychen94]

I see both python_bindings and CodeGen_Pytorch in this repo, my understanding for them is: CodeGen_Pytorch is doable because PyTorch has a C++ API that halide can directly communicate with. And Python in general needs pybind to wrap and pass the C+ codes, which is why python_bindings exists.

It seems that if I take the CxxWrap way, I'm doing a CxxWrap-version of python_bindings and thus make it available to call libhalide inside Julia.

[johnnychen94]

it must be integrated into our repository in the end.

Actually, c45be97 is what I've got so far with the suggested folder structure and now I get one simple method passed:

$ julia --project=julia_bindings/Halide

julia> using Halide

julia> f = Halide.Func("Blur")
Halide.FuncAllocated(Ptr{Nothing} @0x00000000028ee750)

julia> Halide.name(f)[] # [] is to dereference the C++ string reference
"Blur"

To make a real working Halide demo (e.g., minimal JIT compiled example), there're more types that needed to be wrapped. My plan is to finish the concept validation with a working demo before April 1, and then come up with a detailed proposal and send it to the mentioned mail [email protected]

The folder structure of it is:

Halide/julia_bindings/
├── CMakeLists.txt
├── Halide
│   ├── Manifest.toml
│   ├── Project.toml
│   └── src
│       └── Halide.jl
└── src
    ├── CMakeLists.txt
    ├── JlFunc.cpp
    ├── JlFunc.h
    ├── JlHalide.cpp
    └── JlHalide.h

Building Halide with Julia's shared LLVM library is a little bit puzzling so I chose to build it with a self-build version of LLVM 11.0.1 and keep using it until I actually hit symbolic issues mentioned in the TVM issue.

[rootjalex]

it must be integrated into our repository in the end.

Actually, c45be97 is what I've got so far with the suggested folder structure and now I get one simple method passed:

$ julia --project=julia_bindings/Halide

julia> using Halide

julia> f = Halide.Func("Blur")
Halide.FuncAllocated(Ptr{Nothing} @0x00000000028ee750)

julia> Halide.name(f)[] # [] is to dereference the C++ string reference
"Blur"

Looks great!

To make a real working Halide demo (e.g., minimal JIT compiled example), there're more types that needed to be wrapped. My plan is to finish the concept validation with a working demo before April 1, and then come up with a detailed proposal and send it to the mentioned mail [email protected]

👍

The folder structure of it is:

Halide/julia_bindings/
├── CMakeLists.txt
├── Halide
│   ├── Manifest.toml
│   ├── Project.toml
│   └── src
│       └── Halide.jl
└── src
    ├── CMakeLists.txt
    ├── JlFunc.cpp
    ├── JlFunc.h
    ├── JlHalide.cpp
    └── JlHalide.h

Also looks good - generally, I would advise having as-similar-as-possible structure to our python bindings, but I won't pretend to know all of the intricacies of Julia's build system.

Building Halide with Julia's shared LLVM library is a little bit puzzling so I chose to build it with a self-build version of LLVM 11.0.1 and keep using it until I actually hit symbolic issues mentioned in the TVM issue.

Seems reasonable for now!

[rootjalex]

Oh, I missed the higher-level comment. Personally, I agree, I think Julia should be more python_bindings style (not sure if others agree/disagree), as opposed to CodeGen_Pytorch.

[alexreinking]

@rootjalex those are actually different use-cases and they're both important. python_bindings exists to allow writing Halide programs from Python. CodeGen_PyTorch exists to allow compiled Halide pipelines to be used as PyTorch steps, without requiring a Halide installation for the end user.

When we were drafting the proposal for this GSOC project, the intent was that the latter was actually higher priority, but having both would be great.

[steven-johnson]

Personally, I agree, I think Julia should be more python_bindings style (not sure if others agree/disagree), as opposed to CodeGen_Pytorch

Agree 100%

[johnnychen94]

Julia itself uses its own flavor of LLVM and it seems to be problematic for Halide (See the TVM issue).

@vchuravy If it's not too complicated, may I ask will this be an issue for Halide? Or I have to build Julia from scratch just like https://github.com/vchuravy/MLIR.jl? I'm not familiar with LLVM at all so have no idea what would happen if I use my own LLVM instead of Julia's shared LLVM.

[alexreinking]

On Linux and macOS, where we have linker scripts that hide LLVM's symbols, it should be okay to link a static, custom LLVM to shared Halide. There's more work to be done on Windows, where ODR violations are still a legitimate concern.

[vchuravy]

From the Julia side we do strongly encourage users to stick with our patch-set otherwise you will eventually run into code-gen bugs.
LLVM12 hasn't been released yet, but we started the upgrading process (that takes time and will only be available on 1.7-dev)

From a deployment perspective you should likely build Halide in Yggdrasil and then have the wrapper use the Julia LLVM.

[johnnychen94]

Is it possible to pass the Halide Buffer as a Julia Array? I did some experiment by passing the pointer to internal halide_buffer_t::host data (get_buffer_host), and then use Julia's unsafe_wrap to reinterpret it:

julia> using Halide.CppBindings: _Buffer

julia> data = _Buffer.Buffer{Float32}((4, 4))
Halide.CppBindings._Buffer.BufferAllocated{Float32}(Ptr{Nothing} @0x0000000003005570)

julia> make_array(data) = unsafe_wrap(Array{Float32, 2}, Ptr{Float32}(_Buffer.get_buffer_host(data).cpp_object), (4, 4); own=true)

julia> A = make_array(data)
4×4 Matrix{Float32}:
 1.68832f25   1.66783f19  7.14788f22  1.13189f21
 1.06028f-38  1.76852f19  1.2867f22   7.00624f22
 1.10953f27   7.17676f31  1.15879f24  1.28391f22
 6.73472f22   1.81781f31  3.02213f32  1.15879f24

I'm really not sure if this is legal from Halide side because the ownership is unclear to me.

[johnnychen94]

An update on this:

I think it's a good start to share the Halide buffer object with Julia. (Haven't yet considered how to share a Julia array with Halide but it's on my todolist.)

This is the Julia side of the Buffer wrapper, I use ---> to show the C++/Julia function mapping relationship

# Halide           ---> Julia
# dimensions  ---> dimensions
# dim               ---> dim # a simplified tuple with (min, extent, stride)
# fill                 ---> buffer_fill!
# [](const Buffer &b) -> uint8_t* {return b.get()->raw_buffer()->host;}   ---> get_buffer_host

Base.eltype(data::Buffer{T}) where T = T
Base.ndims(data::Buffer) = convert(Int, dimensions(data))
Base.size(data::Buffer) = ntuple(i->convert(Int, dim(data, i-1)[2]), ndims(data))
Base.fill!(data::Buffer, x) = buffer_fill!(data, convert(eltype(data), x))

function unsafe_wrap_from_buffer(data::Buffer{T}; own=false) where T<:Real
    AType = Array{T, Base.ndims(data)}
    data_ptr = Ptr{T}(get_buffer_host(data).cpp_object)
    return unsafe_wrap(AType, data_ptr, size(data); own=own)
end

Then, by filling values on the Halide side, and view it on the Julia side, the result says I'm interpreting
the data correctly.

julia> data = _Buffer.Buffer{Float32}((2, 5))
Halide.CppBindings._Buffer.BufferAllocated{Float32}(Ptr{Nothing} @0x000000000406e570)

julia> data_view = _Buffer.unsafe_wrap_from_buffer(data; own=false)
2×5 Matrix{Float32}:
 3.9f-44  NaN  7.27143f-38  0.0  5.70253f-38
 0.0      NaN  0.0          0.0  0.0

julia> fill!(data, 0) # it calls fill in the Halide side

julia> data_view # the Julia side array gets updated 🎉 
2×5 Matrix{Float32}:
 0.0  0.0  0.0  0.0  0.0
 0.0  0.0  0.0  0.0  0.0

Whether own should be true or false isn't clear to me. According to the comment, it says Halide doesn't own this array, does this
mean that I can let Julia manage this part of memory by setting own=true?

Halide/src/runtime/HalideRuntime.h

Lines 1444 to 1446 in cb78a6b

	/** The shape of the buffer. Halide does not own this array - you
	* must manage the memory for it yourself. */
	halide_dimension_t *dim;

[abadams]

halide_buffer_t doesn't have any reference counting or anything, so it definitely doesn't own the data

Halide::Buffer and Halide::Runtime::Buffer own the data in the cases where they do the malloc, and don't own the data in the cases where they're given a pointer. So in your example above I think Halide Buffer owns the data, and Julia just gets a view of it (but Julia owns the Halide::Buffer object itself).

If you can hide the fact that Halide::Buffer even exists from Julia users and just make everything accept and return Julia arrays, that seems like it would be even better. In that case any Halide::Buffer objects constructed under the hood would be non-owning.

[alexreinking]

If you can hide the fact that Halide::Buffer even exists from Julia users and just make everything accept and return Julia arrays, that seems like it would be even better. In that case any Halide::Buffer objects constructed under the hood would be non-owning.

Using native types is an explicit goal of this project. The Python bindings accept numpy arrays, for instance.

[johnnychen94]

Unlike Python, Julia itself has many array types(e.g., Array, SubArray, OffsetArray), so it would be more convenient to provide a Halide buffer view in Julia as a new array type, and directly applies native Julia functions on this buffer view.

I haven't figure out the way to pass arbitrary array type to Halide buffer via CxxWrap; I might end up with only support the Julia built-in Array.

[johnnychen94]

To make a real working Halide demo (e.g., minimal JIT compiled example), there're more types that needed to be wrapped. My plan is to finish the concept validation with a working demo before April 1, and then come up with a detailed proposal and send it to the mentioned mail [email protected]

Here it comes (it's just a preview so the usage and information will be definitely get improved):

julia> f = Func("Blur");

julia> x, y = Var("x"), Var("y");

# Julia doesn't support assignment operator `=`
# a workaround is to make it a 0-dimensional array and override the `setindex!`[] operator
julia> f[x, y][] = x + y;

julia> result = realize(f, (4, 4))[0];

julia> result_view = unsafe_wrap_from_buffer(Int32, result)
4×4 Matrix{Int32}:
 0  1  2  3
 1  2  3  4
 2  3  4  5
 3  4  5  6

So at least the minimal JIT example works now and I haven't met the LLVM symbolic issue so far. I'll try the AOT example tomorrow and see if it works, but I want to investigate the BinaryBuilder system in the following few days and come up with the proposal.

---

One key difference that shows up here is that Julia uses 1-based indexing, so technically, the result in Julia should be

4×4 Matrix{Int64}:
 2  3  4  5
 3  4  5  6
 4  5  6  7
 5  6  7  8

Question: Should I automatically offset the indexes to suit the Julia 1-based assumption, or just note this difference somewhere in the documentation? I prefer to do auto offset because it would work more smoothly with other Julia array types.

[johnnychen94]

So a separate repo is fine for rapid development, but it must be integrated into our repository in the end.

I still think we need one (maybe two) separate repos for the Julia bindings.

The first repo is for the Julia source codes (and potentially documentation), I still think it's necessary for the following two reasons:

• Julia's package registration system requires the Registrator bot to make PRs to the official registry. This means that separate version control is needed here and thus a separate repo. Julia adopts a rapid rolling version in the whole ecosystem so releases per several months might not be a good strategy for Halide.jl.
• The documentation for Halide.jl will be generated by https://github.com/JuliaDocs/Documenter.jl, and the docs system seems not compatible with Halide's current ones. A separate repo(can be placed together with Halide.jl) is needed to host the documentation.

The (optional) second repo is to host the generated library built by BinaryBuilder. Currently, every binary dependency is built with JuliaLang's buildbot and released in a separate repo in https://github.com/JuliaBinaryWrappers with *_jll name. I understand that we want to keep all related codes and CI workflows under halide org, so I can try to figure out how to set up our own BinaryBuilder workflow, but I'm not sure if we want to host the Halide_jll repo in halide org or in JuliaBinaryWrappers org. If we want to host it in our halide org, then one more separate repo is needed. (Julia also version controls the jll artifacts)

---

For clarification, my final goal of this project is to let users do pkg> add Halide and get everything installed without any form of manual compilation. An example of this is FFTW:

(@v1.6) pkg> add FFTW
  Installing known registries into `tmp`
       Added registry `General` to `tmp/registries/General`
   Resolving package versions...
   Installed IntelOpenMP_jll ─ v2018.0.3+2
   Installed MKL_jll ───────── v2021.1.1+1
   Installed FFTW_jll ──────── v3.3.9+7
   Installed AbstractFFTs ──── v1.0.1
   Installed Reexport ──────── v1.0.0
   Installed JLLWrappers ───── v1.2.0
   Installed FFTW ──────────── v1.3.2

[rootjalex]

@johnnychen94 Is the email address linked on your github accurate? We should probably discuss writing the proposal

[johnnychen94]

Yes, and I've just sent an incomplete draft version of my proposal to the email address [email protected]. For whoever not on that mail list and interested in this project, here it is

GSoC2021_Proposal_for_Halide.pdf

and a simple demo that utilizes both Halide and JuliaImages: