null from Erlang to JSON and vice versa

[williamthome]

There is a confusing thing about the null literal of JSON in Euneus (and probably in any other Erlang lib that encodes/decodes JSON).

Encode contains the nulls option and decode the null_term.

When encoding, nulls is a list of Erlang terms considered to be the null literal of the JSON, e.g.:

1> euneus:encode_to_binary(#{a => null, b => undefined, c => nil}, #{nulls => [null, undefined, nil]}).
{ok,<<"{\"c\":null,\"a\":null,\"b\":null}">>}

When decoding, null_term means what term to be considered as the null literal from JSON in Erlang, e.g.:

1> euneus:decode(<<"{\"c\":null,\"a\":null,\"b\":null}">>, #{null_term => my_null_term}).
{ok,#{<<"a">> => my_null_term,
      <<"b">> => my_null_term,
      <<"c">> => my_null_term}}

Both options can be anything that you want.

Why this? Erlang and Elixir do not have a null term. What I see as a convention is the use of the atom undefined for Erlang and :nil for Elixir to represent null terms. By default, Euneus considers the Erlang way, so the default for the nulls option is [undefined] and for null_term is undefined. So, nulls and drop_nulls plugin can sound strange because them doesn't have/ignore any null term by default.

Due to this, I think we cannot use/consider the same Javascript approach, for example:

> JSON.stringify({a: undefined})
'{}'

> JSON.stringify({a: null})
'{"a":null}'

Using the drop_nulls Euneus plugin:

1> euneus:encode_to_binary(#{a => null, b => undefined, c => nil}, #{nulls => [null, undefined, nil], plugins => [drop_nulls]}).
{ok,<<"{}">>}

@leonardb said:

Sort of surprising the the drop_nulls plugin does not drop nulls by default but converts the null into a string value "null"

4> {ok, JSON} = euneus:encode_to_binary(#{a => 1, b => undefined, c => null}, #{plugins => [drop_nulls]}).
{ok,<<"{"a":1,"c":"null"}">>}
edit: Should the same behavior apply to empty maps? IE, when the recursed processing of a map results in an empty map, should that be treated as a null/undefined? I'm really not sure, but it's not what I'd 'expect'

7> f(), {ok, JSON} = euneus:encode_to_binary(#{a => 1, b => undefined, c => null, e => #{f => undefined}}, #{plugins => [drop_nulls]}).
{ok,<<"{"a":1,"c":"null","e":{}}">>}

The Principle of Least Surprise (based on the name of the plugin) would seem to dictate that the default be [null, nil].

Since the initial request that launched the feature was around handling of undefined and matching the JSON.stringify behavior, maybe that should stand alone rather than part of what is really a drop_map_keys_for_matching_values plugin. Or maybe just better renaming it?

@paulo-ferraz-oliveira said (translated from Portuguese):

I think JavaScript's consider it is as follows...
'undefined' is the "value" you get when something is not defined (which makes it strange that you can assign 'undefined' to something in JavaScript). There is no direct relationship with Erlang. It would be the equivalent of something pre-assigning a value, hence it makes sense in JavaScript; if you try to get the value of a non-existing key in a map you receive 'undefined'.
'null' is more the equivalent of 'undefined' in Erlang (hence your choice makes sense to me). You ask for a value and it is there, but it is not representative. That is, it is explicitly defined as "nothing".

@asabil said:

Actually, one important aspect for me is to make a clear distinction between null and undefined in the same way that JavaScript does.

JSON.stringify({a: 1, b: undefined})
'{"a":1}'
JSON.stringify({a: 1, b: null})
'{"a":1,"b":null}'

What I see is: there is no silver bullet.

Maybe some possibilities:

• Drop the drop_nulls from the Euneus plugins and let the user define what it considers to be removed from maps and proplists;
• Change the plugin behavior to receive an extra argument that will be the plugin options, then encode/2 and decode/2 would be encode/3 and decode/3, so these options in the drop_nulls (probably not the best name in this case) must contain the terms to be ignored;
• Keep the current Euneus approach.

Please help me by answering my question below:
What do you see as the best approach to consider null, undefined, nil, etc from Erlang to JSON and null from JSON to Erlang?

[williamthome]

Friendly ping @mworrell @mmzeeman @vkatsuba 🙂

[asabil]

Just sharing some thoughts here, but I think there are 2 issues to be addressed:

1. null and undefined are conflated.
2. drop_nulls lacks context, and exhibits feature interaction issues.

null or undefined or both?

We have 2 opposing positions that we have to choose between: is this a library for parsing and building serialised JSON structures or is this a library for mapping between Erlang terms and JSON values?

JSON doesn't really have the notion of undefined, only null exists in JSON and as far as I know is typically used to represent the missing value.

Erlang, traditionally, has used undefined, especially in records, to represent the missing value, but. Elixir on the other hand, standardised on nil for the missing value.

JavaScript, from which JSON derives, has both undefined and null, the first represent the nonexistent value while the later represents the missing value. This is conveniently used by the JSON.stringify implementation to allow encoding JavaScript objects with both missing and nonexistent values:

> JSON.stringify({a: undefined, b: null, c: 1})
'{"b":null,"c":1}'

As for Arrays, we have this 🤦🏽‍♂️:

> JSON.stringify([undefined, null, 1])
'[null,null,1]'

The question is: do we want to allow easy construction of JSON objects with nonexistent values?

This would have been a non-question if Erlang had a convenient syntax for building/matching maps with missing values, but it doesn't. So, do we want to use the same pattern as JavaScript and allow a for a special token (undefined ?) that allows us to construct JSON objects with nonexistent values?

drop_nulls naming, context and feature interactions

The naming is confusing and is a direct result of (1).

For a start, the feature should probably be scoped to express the fact that this only applies to maps, and not to arrays or plain values, for example: euneus:encode(Term, #{maps => #{skip_values => [undefined]}}) or euneus:encode(Term, #{plugins => [{map, #{skip_values => [undefined]}}]}) (I know this is not how the current API looks like, just sharing some thoughts).

In fact, maybe this shouldn't be a plugin? Maybe plugins should be called codecs instead of plugins? And they should only concern themselves with encoding/decoding values instead of changing the structural encoding logic of lists and maps?

[vkatsuba]

The undefined is not a official JSON value type - https://datatracker.ietf.org/doc/html/rfc7159#section-1 (C)

JSON can represent four primitive types (strings, numbers, booleans,
   and null) and two structured types (objects and arrays).

So, if make sense to focus on point that the only null should be converted as is. At the same time it can be configurable over plugin mechanism which will add more flexible, so, by default type null will converted to atom null, by default undefined fields will be ignore as in JavaScript - and if somebody want to keep undefined fields, well he need to do it by using plugin config stuff.

[mmzeeman]

In https://github.com/zotonic/zotonic, we usually try to avoid null values from entering the system. So we usually map nulls from postgres or json to undefined.

[leonardb]

Adding my 2c.

For my use case:

1. Our systems accept JSON payloads at the edge (undefined does not exist in a decode context, while null does)
2. We respond with JSON payloads, and the value of those payloads is generally sourced from a database query. In those of cases, undefined does not exist, but null does.

The choice to use null or undefined within code is really up to the authors, but in my case, since we're dealing with database queries/results, using null rather than undefined keeps things consistent.

From my perspective:

• undefined: this thing does not exist at all
• null: this thing has no value

So looking at it from an encoder view:

• atom null should be treated the same as JSON null
• atom undefined should drop the property from the object

Being able to drop any properties with a null value is a feature that I would find useful as in my specific case, we do not send properties with null value over the wire unless there is some weird requirement on the receiver side to receive the property with the null value.

This whole discussion has been around custom plugins to handle the cases, but I believe it's solved more simply as options for encode/decode.

To cover the spectrum:

1. decode_null_to = null :: any()
2. decode_drop_values = [] :: list(any())
3. encode_drop_values = [undefined] :: list(any())

Ending my ramble....

[paulo-ferraz-oliveira]

Here's some thoughts...

I agree with @vkatsuba's

by default type null will converted to atom null, by default undefined fields will be ignore as in JavaScript

which is consistent with what I discussed with you over Slack.

This seems to be consistent with "proposal"

euneus:encode(Term, #{maps => #{skip_values => [undefined]}})

if the second argument was actually a default for the function call, and it also seems consistent with

So looking at it from an encoder view:
atom null should be treated the same as JSON null
atom undefined should drop the property from the object

[williamthome]

Ok, so I'm planning to release a v2.0 of Euneus in #37 after these comments.
I'm currently busy, but I will explain more as soon as possible.
Nice to hear you all! Thanks for the feedback o/
If you have more suggestions, let me know.

[williamthome]

I've been working on this for a while, and last week, I decided to start from scratch and remove all the Euneus code. Now, Euneus is built on top of the new JSON module introduced in OTP 27. There are still things to do, documentation to improve, and the interface is not 100% defined. Feel free to test it out and give any suggestions. I'll close this issue soon when version 2 is released with the changes of the main branch.

[williamthome]

I'm closing this issue in favor of the v2.0 release.

Thank you to all of you who contributed here! All of the comment was extremely valuable to this version.

I've announced the version on the Erlang Forums.

Thanks again, and feel free to continue contributing o/