How to get Text String

[ScottFreeCode]

I turned on the new --text-token option – thank you for that, by the way! – but String is the one thing still parsed into a Haskell String ([Char]) rather than Text and I have to pack it.

I suspect my case for parsing tokens as Text isn't the most common (I'm feeding them into another library that takes Text), but I would think strings commonly should be Text since they're more likely to contain human language than tokens are.

Could an option be added for Text strings? Perhaps separate from --text-token for anyone who wants Text strings but still wants bytestring tokens?

[andreasabel]

Could an option be added for Text strings? Perhaps separate from --text-token for anyone who wants Text strings but still wants bytestring tokens?

This is a good suggestion.

There are related feature request for flexibility in the representation of token types in the AST:

• Java: Java: use BigInteger and BigDecimal (?) for Integer and Double built-in tokens #59
• Java: Support "integer" category to bind to Long instead of Integer #153
• Haskell: The nonempty macro should result in the Haskell backend using the NonEmpty type? #371

Maybe #267 would be a way to in general be more flexible in the Haskell ASTs.

However, it is also thinkable to make Haskell ASTs more polymorphic in the sense that the String tokens can be represented by anything that implements the IsString class (roughly; maybe we need own classes). There would be two specializations, one for String and one for Text.