Implement binary serialization

[svenstaro]

I think it would be very helpful and sensible for this library to feature binary serialization to allow for serialization to disk in custom binary formats or to send something over the network.

[jacob-carlborg]

I think so as well. There are a couple of problems with this though:

• I'm not very familiar with binary serialization
• I'm not sure if binary serialization can be effectively implemented with the way Orange works and how D works

The problem with D is the lack of runtime reflection. This has caused me to design Orange in a way that the serializer is in control instead of the archive.

The library is designed to be able to use user implemented archive types. If you want you could try and implement a binary archive.

[svenstaro]

I don't quite see why we need runtime reflection in binary serialization more than in XML serialization? Isn't the process the same? In XML you get a bunch of nodes and properties and you have to reconstruct the objects from that, from bare strings. In binary serialization you have a fixed structure that tells you how long a certain data type is each cell and what type it is and then you reassemble from that.

[jacob-carlborg]

Runtime reflection is not needed any more with binary serialization than XML serialization. But the lack of runtime reflection caused me to design Orange in a way that I'm not sure if it will be possible to efficiently implement binary serialization.

[svenstaro]

Can you give me a little rundown on what would need to be done in order to implement binary serialization?

[jacob-carlborg]

The documentation for the Archive interface has quite extensive information. All methods are documented and the documentation of the actual interface contains general information about implementing new archives; including some limitations and additional requirements that the archives must follow. Then it's basically up to the creator of the archive how he/she chooses to implement a given method.

If you don't understand something in the documentation or have other questions feel free to ask.

http://dl.dropbox.com/u/18386187/orange_docs/orange.serialization.archives.Archive.html

[svenstaro]

Alright, may I use libphobos at all? Or do I need to reimplement/copy all modules I need?

[jacob-carlborg]

Orange works both for D1 (using Tango) and D2 (using Phobos). If you want to contribute the archive back, then I prefer that it works on these "platforms". The "core" and "util" package contains code to make it easier to write code for both D1 and D2.

About remplementing or copying modules, it depends on which modules. I copied std.xml because I needed to add some additional functionality. Then I created a uniform API on top of std.xml and tango.text.xml.Document. Originally Orange only supported D1 with Tango, therefore I created the XML API to match the API used by Tango. Then with this new API I didn't have to change as much of the existing code and have it to work both for D1 (Tango) and D2.

[svenstaro]

Is that still an issue considering proposed inclusion into libphobos? Your code base might become a lot slimmer by dropping those abstractions. Anyway, I'm sure you had your reasons seeing as this library still supports D1.

I'd need std.stdio and std.stream might be cool too.

Also, I was considering using std.json for an eventual json module. How would we do that? It's probably for a different bug report though.

[jacob-carlborg]

If Orange gets added into Phobos then I would remove all those abstractions. Orange would then still be available as a separate project but only with support for Tango.

One idea could be to create different branches for D1 and D2. I actually don't know if D1 would need to be supported any more. I could create an "old" branch which would be a snapshot of how Orange currently looks like and don't add any new features. Then the master branch would only support D2. The master branch would probably be implement with Tango because the XML parser is a lot faster in Tango and the code is already implemented around the Tango API. In D2, Tango and Phobos can be used together in the same application without any conflicts so it shouldn't be any problems.

About a JSON archive, it depends on how I decide to do with the branches. As it currently looks like, and if we want to support both D1 and D2, we would most likely need to create a uniform API on top of std.json and the JSON module in Tango.

If I go with a master branch which only supports D2 then the JSON archive would be implemented using the module in Tango.