Support additional strongly-typed scalars

[groue]

Hello,

I was wondering if OpenAPI or the OpenAPI generator could define new scalar types.

New scalar types help disambiguating scalar values such as integers or strings by making them strongly typed: this is a user id, this is an ISO-8601 date, this is a number of seconds, this is an amount of cents. With strong typing, mixing apples and oranges is impossible.

OpenAPI has something that looks pretty much like this:

components:
  schemas:
    # A custom string scalar:
    URL:
      type: string
    
    PhoneNumber:
      type: object
      properties:
        localized_description:
          type: string
        # Not a string, but a URL:
        url:
          $ref: '#/components/schemas/URL'
      required:
        - localized_description
        - url

But OpenAPI Generator imports them as their raw Swift type. For example:

// Generated
public enum Components {
    public enum Schemas {
        /// - Remark: Generated from `#/components/schemas/URL`.
        public typealias URL = Swift.String
    }
}

Somehow I was expecting Components.Schemas.URL to be generated as a RawRepresentable type, as below. I thought that the definition of #/components/schemas/URL was a sufficient hint for the desire of a custom type:

// My expectation
public enum Components {
    public enum Schemas {
        /// - Remark: Generated from `#/components/schemas/URL`.
        public struct URL: Codable, Hashable, Sendable, RawRepresentable {
            public var rawValue: String

            public init(rawValue: String) {
                self.rawValue = rawValue
            }

            public init(from decoder: Decoder) throws {
                let container = try decoder.singleValueContainer()
                try self.init(rawValue: container.decode(String.self))
            }

            public func encode(to encoder: any Encoder) throws {
                var container = encoder.singleValueContainer()
                try container.encode(rawValue)
            }
        }
    }
}

I understand that OpenAPI supports a lot of features, such as string formats, that can turn a simple feature into a very complex beast.

I also understand that not everybody has a real or strong need for strongly-types scalars. They could annoy some users ("All those raw values 🙄").

My previous experience with strongly-types scalars was the Swift GraphQL library Apollo: Custom Scalars. By default, it imports custom scalars as their raw Swift representation (so the URL above with also be a typealias to Swift.String). But it allows the developer to provide a custom implementation. I was able to replace all those raw String and Int with tagged values, with much happiness.

In the end, I wonder what is the opinion of the library maintainers on such a topic?

[czechboy0]

Hi @groue, thanks for opening this interesting topic.

First, let me provide some initial context about the lines drawn between the OpenAPI specification, the generator, and Swift.

• This generator supports two OpenAPI Specification versions: 3.0.x and 3.1.0, and internally it's done by converting an input 3.0.x document into a 3.1.0 document using OpenAPIKit, so the generator mostly just works with 3.1.0 currency types.
• In OpenAPI 3.0.x, this is the list of known formats, out of which the generator already supports: int32 (as Int32), int64 (as Int64), float(asFloat), double(asDouble), byte(asBase64EncodedData), binary(asHTTPBody), date-time(asDate). There is currently no refined type for: dateandpassword, or for common formats that aren't part of the core specification, yet they're frequently used, such as uuid, email`, and others.
• In OpenAPI 3.1.0, the list has been shortened to just int32, int64, float, double, and password, out of which we support the first 4, but not the last one. It then just points to the JSON Schema specification, which defines more formats [here](https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-validation-00#section-7.3), out of which we support date-time, but not date, time, and duration. The first two unsupported ones don't have a common Swift currency type, but it'd be reasonable to start generating durationas aTimeInterval`, however that's also just a typealias, so you wouldn't get the type safety you're looking for.
• As far as I know, OpenAPI does not support any way to define custom scalars, except for the format field discussed above, and extra constraints on values, such as maxLength, minLength, pattern, and others. We don't yet support these either. However, we do plan to at least come up with a plan for supporting these constraints in a type-safe way and it's tracked by Figure out how to add more validations to string post 1.0 without API breaks #358.

All of the constraints above are defined by OpenAPI to constrain which values are successfully decoded and encoded, so that if you send a malformed string to a client expecting a date-time string (represented by Date), you get a thrown error instead of having to validate it in your application code. As I mentioned above, there is more work to be done, especially #358, but potentially also supporting commonly used format values, such as email, password, and so on. Ideas in this area are very welcome.

However, from my understanding of your question, you're not as concerned with how a value is encoded/decoded, but whether the generated code allows different typealiases of e.g. String to be used interchangeably. It's a good question, I personally haven't thought about this area very much yet, how about you, @simonjbeaumont?

There are a few ways this could go:

• you could see if there's demand for the Swift language to make typealises a bit more strict than they are today through the Swift Evolution process, at which point the generated code would get the benefits for free
• you could propose that all JSON Schema values that are just refs to other types are explicitly wrapped as you pointed out above (but keep in mind that refs also work for objects, not just for primitive types, so the question is what we'd do there)
• you could propose that an explicit wrapper is only generated for specific format values
• you could propose adopting vendor extensions for this, although that has the disadvantage that it starts making the OpenAPI document less language-agnostic and has features only useful for one language, which I'm personally not a fan of
• (something you can do today) you wrap the generated API for your users and provide these safer types yourself, adding whatever validation in encoding/decoding/converting you'd like (one could argue that even if you can't automatically "cast" one typealiased string as another, maybe there are specific rules under which you want to be able to, but only after validating the value)

As you can see, this is a pretty open-ended topic, so I encourage everyone to chime in and let's discuss here what could be done.

[groue]

However, from my understanding of your question, you're not as concerned with how a value is encoded/decoded, but whether the generated code allows different typealiases of e.g. String to be used interchangeably.

Yes, @czechboy0, this is exact. The topic of formats is very interesting as well, and it looks slightly related, but indeed this is a different track.

There are a few ways this could go:

• you could see if there's demand for the Swift language to make typealises a bit more strict than they are today through the Swift Evolution process, at which point the generated code would get the benefits for free

There has been multiple attempts (1, 2, 3, etc) at pitching related ideas on the Swift forums, without any success so far. The desired feature set is not quite clear, to be honest.

A good userland library with good ergonomics is swift-tagged (which is not based on typealiases, but instead defines new types that behave pretty much like their raw value).

I'm not sure typealiases will ever change their behavior. Today, typealias Apple = Swift.String and typealias Orange = Swift.String are all String. They are 100% interchangeable, and any modification here would be a catastrophic source break. Think about the number of apps that rely on Foundation's typealias TimeInterval = Double 😅

So I'm not sure discussing typealiases on Swift Evolution can actually be fruitful.

• you could propose that all JSON Schema values that are just refs to other types are explicitly wrapped as you pointed out above (but keep in mind that refs also work for objects, not just for primitive types, so the question is what we'd do there)

Yes, this is something that I find interesting.

The intent behind a ref is ambiguous. A ref may be actual model design, a stable concept that is part of the API. But it may also be just an artifact of the OpenAPI syntax, more akin to an implementation detail or a convenience than actual API design. In this category are refs that help avoiding tedious repetitions, refs for loose abstractions "just in case we want to change it in the future", etc. It's a little bit like public and fileprivate, but OpenAPI does not make such access levels explicit (in the current state of my knowledge).

The OpenAPI Generator generates public types for those refs (Components.Schemas.XXX), so it leans towards the "refs are public" interpretation, even for convenience refs.

In this context, I'd say that it's not too odd to help developers turn refs for scalar types into dedicated RawRepresentable types. It certainly requires an opt-in (a flag in openapi-generator-config.yaml?), because this is an "advanced" feature that not everybody expects.

I'll have a closer look.

• you could propose that an explicit wrapper is only generated for specific format values

I'm positive there is an interest for wrappers even for freely formatted values (AppleID vs OrangeId, or CentAmount vs Duration for example).

• you could propose adopting vendor extensions for this, although that has the disadvantage that it starts making the OpenAPI document less language-agnostic and has features only useful for one language, which I'm personally not a fan of

All right, so let's not do this.

• (something you can do today) you wrap the generated API for your users and provide these safer types yourself, adding whatever validation in encoding/decoding/converting you'd like (one could argue that even if you can't automatically "cast" one typealiased string as another, maybe there are specific rules under which you want to be able to, but only after validating the value)

Yes, I certainly do this already.

The types generated by OpenAPI generator are not quite made for public consumption. And OpenAPI can not express all constraints that distinguish valid from invalid payloads. It is certainly the role of some app layer to "translate" raw decoded payload to properly-designed models. This is frequently a lot of boilerplate, but certainly not the kind of boilerplate I'd complain about.

As you can see, this is a pretty open-ended topic, so I encourage everyone to chime in and let's discuss here what could be done.

Yes :-)

[czechboy0]

The types generated by OpenAPI generator are not quite made for public consumption.

Oh absolutely not, we explicitly discourage one party to publish the generated types for another party. Folks should generally just use the generated code as an implementation detail: https://swiftpackageindex.com/apple/swift-openapi-generator/0.3.4/documentation/swift-openapi-generator/api-stability-of-generated-code

[groue]

• you could see if there's demand for the Swift language to make typealises a bit more strict than they are today through the Swift Evolution process, at which point the generated code would get the benefits for free

There has been multiple attempts (1, 2, 3, etc) at pitching related ideas on the Swift forums, without any success so far. The desired feature set is not quite clear, to be honest.

One more :-) https://forums.swift.org/t/how-about-subtypealias-farenheit-double/68513

[teameh]

However, from my understanding of your question, you're not as concerned with how a value is encoded/decoded, but whether the generated code allows different typealiases of e.g. String to be used interchangeably.

Isn't the way the value is decoded is just as important here right? Being certain that a value is a valid strongly-typed scalar is one of the reasons for using this. You want the value to be decoded as URL and not as a String because:

1. You want the validation when initialising that URL, and you want the request to fail and throw when it's not a valid URL.
2. You don't want to have to guard let url = URL(string: myObject.someURL) the value when creating your own model. By that time it's ideally already too late.

Any new thoughts or updates on this since november? It would be great to support this.

[simonjbeaumont]

You want the value to be decoded as URL and not as a String because...

Narrowing the discussion to format: url, I think it's a reasonable expectation for this to map to a strongly typed URL.

When we try to generalise it we start asking questions about the target type, e.g. for format: email.

A pragmatic middle ground we could consider is to do validation on as whatever formats are well specified in the OpenAPI and JSON Schema specifications and constrain generating a strongly-typed property for just some.

Any new thoughts or updates on this since november? It would be great to support this.

To be transparent, @teameh, the issue is open because we acknowledge that we likely need improvements on this area, but it's not something we're actively staffing right now.

[teameh]

Check. Thanks.

I think there's also a distinction to be made between Swift native supported types like URL and unsupported types like email. But yeah, validation on these types sounds indeed like the best way to support them. Perhaps also behind a opt-in flag.

To be transparent, @teameh, the issue is open because we acknowledge that we likely need improvements on this area, but it's not something we're actively staffing right now.

Got it. Anything I can do help here?

Narrowing the discussion to format: url, I think it's a reasonable expectation for this to map to a strongly typed URL.

Is there something that is stopping us implementing this now? 😃 or am I too hasty?

[simonbility]

I for one would really welcome "strongly typed scalars" or something similar.

My uses case is "enum-like" types.
Meaning i have a set of defined values but specifically want to keep the option to add "cases" without introducing a breaking change.

In these cases a RawRepresentable with constants for each value is my go-to solution.

Could we somehow build a solution that does not (directly) rely on format (which seems to be subject to change)

I know it has not been done so far, and it might open the door for "too much" configurability, but one way would be to allow opting in or configuring this via vendor-extensions.

Taking the example of the initial request.

components:
  schemas:
    # A custom string scalar:
    URL:
      type: string
      format: uri
      # a few variants
      x-swift-open-api-generator-custom-format: true # generate a new raw representable type
      x-swift-open-api-generator-substitute-type: Foundation.URL # instead of generating String i can "swap in" my own type (obviously it must be importable)

[czechboy0]

I agree there is demand for improvements around typed scalars, from URL to values constrained to a specific regular expression. It's useful for us to discuss this here, but at some point it'd be good for someone to drive a concrete proposal. Note that I expect this to be split into a few smaller pieces of work, one proposal doesn't have to solve it all - but at the same time, we should at least have an idea of which direction we'd like to go overall here.

We have not used vendor extensions yet, but I'm open to that idea as well (might be in addition to the other ideas, doesn't have to be a replacement). I think that would also play nicely with #634, as one could take a language-agnostic OpenAPI doc and overlay the customization in a Swift-specific OpenAPI overlay.

All of these are just ideas though, thrown out here for further discussion 🙂

[simonbility]

Im happy to come up with a official proposal, but my timeline for this would be at the soonest in two months.

That being said i do think the "type substitution" approach has a lot of potential.
I think it could be potentially useful outside of the specific use-case of strongly typed scalars.

Our API for instance is vends geo-json https://datatracker.ietf.org/doc/html/rfc7946#section-1.4

While this has a well-defined format i think "inlining" this into our definition would bloat our spec a bit, and we do already have a type (from a library) to handle the parsing.

Right now we are not using the open-api-generator for these endpoints because of this.

If we could substitute the type we could actually use the type from our library.

components:
  schemas:
    GeoJSON:
      {} # In our case allowing everything would be acceptable, this mostly serves as a placeholder
    x-swift-open-api-generator-substitute-type: GeoJSONLibrary.GeoJSON

This would also avoid having to keep the generator in sync with the various formats (differing between versions) and put the decision/responsibility on the user.

[czechboy0]

Thank you @simonbility for describing your use case in depth. The ask sounds very reasonable. Let us discuss and think about this a bit and get back to you about how we proceed.

[simonjbeaumont]

While we have yet to do this, we've been chatting about extensions, e.g. x-swift-openapi-generated-name, but I'd like us to have really thought through the scope of these and be principled about where and how they're used. Maybe we start with something simple first to iron out the kinks.