Save and reload trained network on DNNL and cuDNN

[lmtriet]

I'm reading the book DLFP and just finished the Tensors section and learning quite a lot.

The only thing that I can't find in the book is a simple way on the Deep Diamond high level API to store the trained network and reload it later ?
Could you suggest an approach ?

I tried the function transfer! to a FileChannel but it doesn't work.

(def network-blueprint (network (desc [512 1 28 28] :float :nchw)
[(fully-connected [512] :relu)
(fully-connected [10] :sigmoid)]))
(def net (init! (network-blueprint :adam)))
(train! net x-train y-train :crossentropy 30 [])

(defonce network-trained (random-access "C:/workspace/dlfp-code-0.34.0-2.0/dlfp/data/mnist/network.trained"))
(transfer! net (.getChannel network-trained))

Execution error (IllegalArgumentException) at uncomplicate.commons.core/eval2100$fn$G (core.clj:116).
No implementation of method: :bytesize* of protocol: #'uncomplicate.commons.core/Bytes found for class: uncomplicate.diamond.internal.dnnl.tensor.DnnlTensor

Thanks

[blueberry]

Hi. Storing a tensor or a network is not as simple a task as it seems, since the internal number formats of these objects is opaque. Even vectors can have strides, etc...

You can not transfer to a file channel, you can map memory to a file channel. Please see this function: https://github.com/uncomplicate/neanderthal/blob/e4a03ddca20261bf7718b7bcabc66e7ef48ee072/src/clojure/uncomplicate/neanderthal/native.clj#L405

The idea is to memory map an object (it's best to first start with vectors to see how it works), and then the changes to that vector are tied to your disk. That vector should be in a format that is simple and universally readable on other machines, but it doesn't have to be. Then, you work with it (and your changes are saved), or you transfer to/from other more complex objects when you want to load/save them from the disk.

For tensors, you can use map-tensor to load/save from disk. This function is used for loading MNIST data for example.

Networks can be transferred to channels automatically, but you have to provie a file channel. As far as I remember, network (nor tensor) does not implement .getChannel method.

[lmtriet]

Networks can be transferred to channels automatically, but you have to provie a file channel

This is what I try to do with the last two lines of the code.

(defonce random-access-file (random-access "C:/workspace/dlfp-code-0.34.0-2.0/dlfp/data/mnist/network.trained"))
(transfer! net (.getChannel random-access-file))

I have seen the implementation here :

deep-diamond/src/clojure/uncomplicate/diamond/internal/dnnl/file_channel.clj

Line 62 in bc9e424

(defmethod transfer! [SequentialNetworkTraining FileChannel]

I guess it should just work transferring a network to a file channel but it doesn't because the DnnlTensor does not implement the protocol function Bytes/bytesize*.

[blueberry]

Thanks for reporting. I'll investigate this after the weekend. It might be a bug introduced during the port to JavaCPP, but I'm not sure now without trying it out myself.

[blueberry]

BTW the book code uses the older version of deep diamond, so it shouldn't have been affected by port to JavaCPP...

[lmtriet]

I use the latest version of neanderthal and deep-diamond to code along with the book. So this might explain the error.
uncomplicate/neanderthal {:mvn/version "0.49.1"} uncomplicate/deep-diamond {:mvn/version "0.28.2"} org.bytedeco/mkl$windows-x86_64-redist {:mvn/version "2024.0-1.5.10"} org.bytedeco/cuda$windows-x86_64-redist {:mvn/version "12.3-8.9-1.5.10"}

[blueberry]

Can you please send me a notification about which chapter files work as-is, and which ones require changes to work with the new version? Most of it should be backward compatible, but some parts might need changes.

[lmtriet]

As far as I remember, there are some minor adjustments, due to stuffs moved, renamed...

• view moved to common
• train => train!
• infer => infer!
• zero does not initialize new tensor to zero, need to call entry! first (so it defeats the purpose)
• CUDA Toolkit install required even with the latest version

[blueberry]

Hey, thank you for reporting this. The fix for DNNL tensor's zero bug is in the repo and will be in the next release, which is coming soon.

[blueberry]

The network-to-channel transfer has also been fixed in the latest 0.29.4 release. You can also find a brief example here: https://github.com/uncomplicate/deep-diamond/blob/master/test/uncomplicate/diamond/functional/examples/saving-and-loading.clj

Please try it and write about whether it works for you. Please also report any other issues you stumble upon. Thanks!

[lmtriet]

Hi Dragan,
Thanks for your new fixes. I tested the new version today.

• zero works
• transfer! to file channel works on fully connected
• transfer! to file channel does not work for RNN (on Mastercard sample)

Execution error (IllegalArgumentException) at uncomplicate.diamond.internal.network/eval57593$fn (network.clj:317).
Don't know how to create ISeq from: sun.nio.ch.FileChannelImpl