Addition of "properties" layer on classmate.

[garretwilson]

We have a new format, SURF, for which we want to create a marshaling system (like Jackson) for JAX-RS. (We already have reference implementation parsers/serializers in place.) We considered piggy-backing on Jackson to take advantage of the property reflection and annotations (one of our developers, @magnonasc, already communicated with @cowtowncoder on the details), but I think the impedance mismatch is too great and we would wind up rewriting a lot of things. So I've pretty much decided we'll just write everything from scratch (sort of like writing another Jackson, but for SURF).

Libraries such as Jackson, JAXB, XML encode/decode, and the like have several layers:

1. reflection / type discovery
2. property determination with optional annotations
3. actual marshaling (converting object propertis from the instance tree to the format and back)

I have written this stack myself many times over the years (an almost-two-decades old unfinished tutorial article I wrote is slightly amusing in hindsight), but only a few days ago did I realize that I could have been using Introspector to do steps 1 and 2. Then I realized I'd run into trouble with generics, and found this project, https://github.com/FasterXML/java-classmate . And I'll be glad to use it.

The problem is that ClassMate only handles step 1: using reflection to discover methods and types (albeit resolved as mush as possible with generics). I'll still need to use those methods to determine which properties (based upon the getter/setter names and annotations) the marshaled object should have. And this is the step where the wheel has been repeated over and over (my code included). Jackson (I assume) has its own code for determining which "properties" exist, based upon the accessor methods and properties. Jackson has its own proprietary annotations (although I see that Jackson can use JAXB annotations as well). This is somewhat of a shame, because really the determination of which properties to ignore, what their names should be, and whether to use methods and/or fields is in large part orthogonal to the actual format being used for marshaling (JSON in the case of Jackson, XML in the case of JAXB).

(A side note: all my historical property determination code would turn getURL()/setURL() into a url property, but just this week reading the JavaBeans specification I see that it prescribes a URL property in this case. I wonder which Jackson would use by default.)

My point of all this is that, rather than integrate step 2 (property determination with annotations) into the SURF marshaling system, I'm going to spin out that layer into a separate project this time so that it could be used by various libraries. That way we would have a consistent way of determining which properties exist for being marshaled (independent of the marshaling format), and a common set of annotations. You could then annotate your POJO one time, and it would work for SURF, JSON, XML, or whatever. (Of course that's only if Jackson would support these annotations at some point in the future. The other of both our libraries would be sharing a common codebase for property determination and annotations.

So I'm filing this issue to ask what the interest on the Jackson/ClassMate side is for such a small, self-contained library for POJO property determination with an annotation set. It's something that could be beneficial to both libraries. I'm going to have to write such a thing one way or another (and I've already started), but I'd love to collaborate. (On the other hand, if you indicate that ClassMate or another library already has this property determination layer built with a format-agnostic API, that would be even better.) I welcome any thoughts.

[cowtowncoder]

Ok, lots of good insight here. I'll start with something that probably is not surprising: I originally intended to build something more than ClassMate which would do what you suggest -- codename BeanMate. But due to sheer effort of extracting out code, and complexities thereof, I decided to start with just lower half. I won't go into all the details, but I think that effort for (2) is much bigger than (1); and part of that has to do with question of how to abstract out semantics of annotations.

Anyway I guess that is just saying that yes, having that layer would be great and should be very useful for a whole set of libraries and frameworks.

As to annotations, the reason Jackson has its own annotations is because there just wasn't (and, as far as I know, isn't) something extensive and generic enough (JAXB being too XML-specific; some of JSR ones cover only small subset of reasonable set).
I had some hopes that maybe JCP that was to define JSONB API might tackle 2 aspects of json processing I could see helped by standardization -- type resolution (~= classmate, or at least type objects better than java.lang.reflect.*) and standard property annotations -- but that did not end up happening.

[cowtowncoder]

With that as background: yes, I would very much like to see something build on top of classmate.
Question is how to go about that; probably starting from a minimal core set of features to support, extending from thereon. But ability to support different annotation sets (and some other pluggability) seems like a really important thing for me -- both given lack of standardized annotations, and due to there always being some extension sets that libs/frameworks want to define and use.
It might be worth a look to see Jackson's concept of AnnotationIntrospector for ideas; not as specific mechanism to use (it's too low level), but concept of configurable translation.
It will be necessary to think of how to scope (how extensive would canonical set of supported metadata be? Naming, inclusion/ignoral are easy but beyond that...). And perhaps question of how abstract/concrete should metadata be.

From Jackson side I think progress might be slower due to historical reasons, tight coupling of code. Classmate itself is actually not used by Jackson (alas) -- mostly because Jackson's JavaType type hierarchy is exposed as part of API, and is not 1-to-1 mappable with classmate type hierarchy. Internals of Jackson have been rewritten, however, to resemble classmate's processing quite closely (so ClassMate was extracted out of Jackson, and after delay most of improvements were ported back).

So it would be great to eventually retrofit Jackson to use possible new library, but that would be a long process and probably not worth making an initial goal. But I do think value of such library has more to do with libs/frameworks other than Jackson actually, similar to value of classmate.

Finally, question about getURL -- turns out this is one known deviation Jackson has from Bean naming. Bean naming only lower cases a single leading capital character (so it'll remain "URL"); but I did not know this when I wrote Jackson's handling and Jackson by default considers it to infer property "url". There is a setting to change this behavior, to make handling bean-compatible, and for Jackson 3.x this will become the new default.

Does above make sense?

[garretwilson]

@cowtowncoder , first of all, thank you for reading all my thoughts, processing them, and giving a response. And it made my day that you're interested in this effort.

As I mentioned earlier, this is something that I'll wind up doing whether or not anyone cares, because I'm so interested in having the SURF format succeed --- and I absolutely need this functionality for drop-in SURF JAX-RS marshaling. But it will make the whole process much more fun and even exciting if you're "on board", at least to the extent that you see the need for such a thing, and that you're watching it with an approving eye.

In an earlier incarnation of SURF (long story) I actually wrote this entire marshaling framework simply for storing configuration information (after I had written it for XML, before Java added XML encoding/decoding) for an Internet application framework (another long story). Although the property determination logic was tightly integrated into the pre-SURF version, I called the entire process PLOOP for Programming Library for Object-Oriented Persistence (with the idea that you just "ploop" your objects into the configuration files.

This new effort to create (from scratch, yet again) a self-contained, format-agnostic project for property access will be a module of PLOOP. Specifically it will live at the Maven coordinates io.ploop:ploop-pojo. The work is tracked in JIRA by PLOOP-12, and I've already started the implementation. I intend to finish the initial core functionality this week, because @magnonasc is waiting on it to create the io.urf:surf-jaxrs SURF marshaling implementation, tracked by URF-32.

The first pass will simply determine which properties to use from available getter/setter methods and public properties --- no annotations for the meantime. Later I'll add annotations for, say, indicating that a private property should be used for reading and/or writing; for ignoring properties; for changing property names; etc. Your experience will be really helpful in finding the direct path to go down. In the short term, however, I'll just be figuring out how to use ClassMate. (I could just pull out and repurpose my existing code if I wanted to use plain reflection, but I figure I should start with ClassMate from the beginning.)

The slight inconvenience (to you) is that all these projects of GlobalMentor, Inc. (my company) are managed via Bitbucket/JIRA, not Github. I would love to have your input, and I will doubtlessly have many questions not only about how to use ClassMate, but also to draw on your experience with Jackson on how best proceed in some areas. I would happily add you to our PLOOP JIRA project so that you could comment directly, or if you prefer I could file questions here on the ClassMate GitHub project for you to answer. I'm assuming that your contributing code to the project is too much to ask for, at least for the moment, but that would certainly be welcome if some day you'd like to.

Let me know how best to let you participate. Again, even just following the project and a word of encouragement and/or advice from time to time from you will be marvelous.

[garretwilson]

Bean naming only lower cases a single leading capital character (so it'll remain "URL"); but I did not know this when I wrote Jackson's handling and Jackson by default considers it to infer property "url".

Haha. Yeah, me neither. I only found out yesterday on a plane when I decided to actually read the JavaBeans spec. (That's also when I discovered Introspector and ClassMate.) All my code (e.g. com.globalmentor.java.Classes) from almost two decades ago blithely assumed that nobody had specified one way or another, and just did it the way I thought best.

[cowtowncoder]

Sounds good -- using Bitbucket should be fine. I recall cloning projects from there earlier, and I assume using git should make things pretty similar to github.
Happy to help with classmate usage; it does solve generics part at least, and that is something that can be surprisingly complicated to deal with.

I might also be interested in trying out resulting Ploop library with jackson-jr (https://github.com/FasterXML/jackson-jr). It's an alternative for full Jackson databinding, no annotations, uses Bean Introspector; but based on streaming Jackson parser/generator.
It might be fun integration project, with no real legacy requirements.

[garretwilson]

OK, this project is now well under way. The official name of the larger project is now Programming Library for Object-Oriented Persistence (Ploop), although note that this user-facing site is not yet live. The sub-project specifically for object reflection (built on top of ClassMate), property discovery, property access, and reflective object creation is ploop-introspection. The code is now live on Bitbucket at https://bitbucket.org/globalmentor/ploop-introspection . The initial release is tracked by the JIRA issue PLOOP-12.

The first task is merely to read JavaBean methods, which is tracked by PLOOP-13. If you pull up that JIRA ticket, you'll see under "Development" that there is already a branch with the initial implementation basically finished. I haven't tested a single line of code, though, so I need to create test classes and unit tests before creating a pull request. Hopefully I'll get to that in the next day or two.

@cowtowncoder , If you would like to be a code reviewer on these upcoming pull requests (and I would be honored), please let me know your Bitbucket username so I can provide you with the appropriate permissions. Being a code reviewer doesn't mean you need to actually add any comments if you're busy, but you will be notified of new pull requests and be able to add comments if you wish. (If you really get enthusiastic, of course you can create branches and pull requests too, if you wish.)

(If you want to be able to file, modify, and comment on JIRA tickets, I'll have to add you to the JIRA project manually. For that I'll need the email address you prefer to use for the GlobalMentor JIRA account. If you wish you can send me this information via email; you can find my email address by searching online for my resume. But this doesn't have to be done now; you can still participate in the pull requests separately as explained above.)

Just as important going forward will be taking advantage of your expertise in formulating the rules for Ploop introspection --- i.e. the order in which properties are discovered, which definitions take precedence (e.g. accessor methods, fields, annotations, etc.), and what the annotations mean. I want to document all this in a separate document so there will be no ambiguity, and it will allow us to provide a good test suite of the implementation.

Thanks again for the interest.

[garretwilson]

The first Ploop Introspection tests are passing and the first full request has been made, so things are going well.

I don't want to hijack your ClassMate GitHub site with Ploop development discussions, since it's a separate project, so I've created a discussion forum for Ploop:

https://discuss.ploop.io/

It would be really great if you choose to participate. There are many issues that could benefit from your expertise and experience with ClassMate and Jackson.

To kick things off, I've started an initial discussion on what minimal version of Java we'd like to use:

https://discuss.ploop.io/t/minimal-java-version/17

[cowtowncoder]

Quick note: my account at bitbucket is also @cowtowncoder.

I'll join the other mailing list; although I am ok with discussion on classmate too -- partly since it's low-volume, and partly as discussion is still somewhat relevant. But you are right it is above and beyond actual current functionality.

[garretwilson]

Just FYI I got a chance to come back and start working on this again, so I'll keep you updated.