JSON Type

[okennedy]

Spark natively supports nested data a'la JSON. There's enough going on in Mimir that uses JSON that it might make sense to add native support for Json (i.e., TJson(), JsonPrimitive(scala.play.json.JsValue) and support for Json primitives in the SQL parser - #119).

[okennedy]

Possibly useful for supporting #212

[okennedy]

• Add a new type / primitive
• Update the typechecker
• Figure out how this plugs into Spark's handling of JSON / nested data. (e.g., we should be able to use Spark's JSON functions to interact with this new type.

[okennedy]

It's possible that what we want is actually Map/Array types instead (paralleling Spark's Map/Array types)

[willspoth]

I think the lens interface will do exactly what we want:

Create Lens mydata 
AS Select row_id, jsonCol from jsonDataset
With (row_id Int,{JSON-Schemas}); // these would handle column definitions outside of the json

Then to query mydata do SELECT row_id, $.name, $.pharmacy.* FROM mydata
This could handle un-nest, and could even use generic column regex instead of only being json syntax