-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to serialize void[] #43
Comments
Here is something I just through together that seems to work for the serializer:
While I might continue to work on it to make it better, I think it has 2 major issues that should be address.
In fact, maybe the inlining as a base64 string should be or something better (7-bit ascii with 8 bit being an extension so no encoding or decoding is required for normal alphanumeric). Anyways, I'm not sure how to do that so I'll leave it up to you. The deserialization must also take place. |
Here is the completed serializer that will handle short arrays, inline representation, external file representation using static path. Uses id and hash of key and id to generate the file name, would better to use the full path if you can get at that(e.g., rather than just the fieldname, the class name, etc) but since id should be unique, it shouldn't matter, although changing the structure being serialized will result in orphaned files.
|
Here is the deserializer. You might want to have extend this for very long arrays of ints, floats, etc. This way one could use the serializer as a sort of one stop shop.. Could save images, etc. This would reduce the file size significantly since arrays are stored per element it seems(so an array of length N will have something like M*N the size where M is all the extra xml overhead for the element). Maybe use attributes to specify that the field is to be serialized to an external file.
|
I'm not sure that I'm that comfortable supporting serializing |
A void[] is just data, that gets serialized as bytes and back. There can be no harm in doing so because that is all there is. I'm sure you do not serialize byte[] to a file though, do you? Or any array, and that is important to me because I store audio data and image data in arrays. They make the xml file extremely large because of how inefficient arrays are. You could always add an attribute to get them to serialize. Not supporting something based on a feeling will just screw those that don't have that feeling. The fact of the matter is that now, without being able to serialize a void[], a loss of data situation is created, and that is, IMO, unacceptable(of course, it does err out, but then that leads to one having to jump through hoops to get what they want done). |
No: "There is a special type of array which acts as a wildcard that can hold arrays of any kind, declared as
No. The serialized XML would most likely end up in a file anyway. Why add another file?
I could special case ubyte/byte arrays to serialize in a different more efficient format, but not void arrays.
It's not a feeling. Serializing arbitrary void arrays won't work. What's wrong with using a type that is intended to store bytes? |
The problem is that the array can be any type. If I specify bytes then it is assumed it is a byte. The arrays are audio data and the type depends on runtime behavior(what the user selected, what the audio interface uses, etc). Your arguments are not meaningful. bytes can hold pointers, one can have an array of int's that are pointers too. So you are just picking and choosing. byte[]'s and void[] are identical as they are just logical differences, not physical. Everything is a bit in programming. If you don't like the idea of automatically serializing a void[] then at least allow a way to force it instead of making the assumption that it is bad. It is not bad in all cases so why make me jump through hoops to do what I need to do? @serializeAs(byte[]) In which it treats it as a byte[]. That kinda thing would be fine for me. But to prevent it all together is not good. Of course, I've already modified it to do what I want, so all this is moot to some degree. If I knew how to use attributes with orange I'd probably add that ability. It's true, I could just use byte[] instead and cast to whatever type I need to, but void makes it explicit to me that it's not actually meant to be a byte value(which exist for audio). When you serialize arrays one end sup with an xml tag per element. The tag is very long, about 50 times the size of the element. That ends up exploding the xml array. e.g., suppose I want to store an mp3. 5MB = 250MB. Storing it to an external file will only still be about 5MB. Same would go with images and other stuff. I'd rather have to be a little explicit in getting what I want orange to do then it prevent me from being able to do it at all. I'll probably write my own serializer at some point anyways, so I'm not too concerned with what you want to do. I've already customized orange to do what I need and it was pretty easy. Seems to work well, I might continue using it, I'm not sure. I just figured that others might benefit from having the ability to do what I wanted. Since the code is already added, you could simply make it optional or add some attributes to enable it. |
I guess nothing. You could at least try out the code
Again, you are forcing one to change their internal code to suit orange so it will serialize. The code I've added shows how to serialize data to files and works well. The reason not to realize to the same file is that large data sets could be duplicated if one uses them in multiple deserialization processes. The bigger problem is that serializing them inline makes the size an order of magnitude larger!! The reason you don't think this is a proper solution is because you have not used orange in the real work on large arrays! Even if I changed it to ubyte[] it would still increase the size 10x. But my data is void[] which can be any type so forcing ubyte requires me to constantly cast to void. There is no point. You can support the feature and make it optional by parsing an attribute that will serialize the file to an external binary file. I've already did most of the work for you! What happens is I cannot use your library as is because it is deficient for my purposes. Any updates you will then cause prevent me from easily updating. You are trying to justify not doing this when you should be finding ways to justify to do it. It is not a silly request, it is in fact very practical, but just because you haven't used orange in such a way you feel it is not but your feelings is not reality. class Item } large binary data could be 100M. Stuck in the output of the serializer it becomes 1G+ because of all the extra xml text per byte. THIS IS NOT EXCEPTABLE! Being an optional inclusion does not hurt anyone as they can opt in. This solves the problem with void being pointers or whatever.
rather than
which you seem to actually believe is acceptable! IT IS NOT! Get it through your head! Please, it is ridiculous! For a single byte you are adding over 30 bytes in the xml file! A single meg will then take 30 megs to store! Do it my way and it takes only an extra few bytes. This is not hard, basic programming! Don't be lazy! I've done 90% of the work! |
I would like to serialize a void[]. I get
Error: expression value[i] is void and has no value
when trying.I imagine this is easy to overcome since the void[] is a series of bytes. It is a very large array(audio), so I'm curious if it can be done and be done using binary to avoid bloat? While I might be able to hack it myself, it might be nice if you could add it so it would be done right. Even if it simply serialized it to a binary file and use a link in the serialized output(might be more work). I realize that the binary inside an xml text file might be considered a bad idea but in my cause it wouldn't matter. I need binary. I am using xml so I'm not sure if it will even work so you might simply have to save it to a file then make a reference in the xml to that file and load it on serialization. Maybe use a special attribute to tell it to serialize to a file or something similar.
Thanks.
The text was updated successfully, but these errors were encountered: