Custom Types, Custom Serialization, Custom Validation

[augustsaintfreytag]

So I’m very close to my breaking point with this entire project, I’ve spent over a week now getting everything working, from the compiler integration, source mapping, debugging, and now actually deserializing types. I’ve been stuck at generic data models that worked 1:1 in a demo but didn’t in my big project. I think the docs are lacking when it comes to custom types.

This issue is for a use case I think is actually really trivial, effectively a subclass of Set<T>. The stripped code is:

import { serializer } from "@deepkit/type"

export class OrderedSet<Element> implements Iterable<Element> {
	// State

	private nodesByElement = new Map<Element, OrderedSetNode<Element>>()
	private root: OrderedSetNode<Element> | undefined = undefined

	// Init

	constructor(elements?: Element[]) {
		elements?.forEach(element => this.add(element))
	}

	// Read

	public get size(): number {
		// (omitted)
	}

	public get isEmpty(): boolean {
		// (omitted)
	}

	public get first(): Element | undefined {
		// (omitted)
	}

	public get last(): Element | undefined {
		// (omitted)
	}

	public has(element: Element): boolean {
		// (omitted)
	}

	// Mutation

	public add(element: Element) {
		// (omitted)
	}

	public delete(element: Element) {
		// (omitted)
	}

	public clear() {
		// (omitted)
	}

	// Iteration

	public *[Symbol.iterator](): Iterator<Element> {
		let node = this.root

		while (node) {
			yield node.element
			node = node.next
		}
	}

	public forEach(block: (element: Element) => void): void {
		// (omitted)
	}

	public map<T>(block: (element: Element) => T): T[] {
		// (omitted)
	}

	public filter(block: (element: Element) => boolean): Element[] {
		// (omitted)
	}

	// Transform

	public toArray(): Element[] {
		return [...this]
	}

	public toSet(): Set<Element> {
		return new Set(this)
	}

	// Serialization

	public toJSON(): Element[] {
		return this.toArray()
	}

	public static fromJSON<Element>(json: Element[]): OrderedSet<Element> {
		return new OrderedSet(json)
	}

	static {
		globalThis.serializerTypes.set("OrderedSet", OrderedSet)

		serializer.serializeRegistry.registerClass(OrderedSet, (type, state) => {
			state.addSetter(`${state.accessor}.toJSON()`)
		})

		serializer.deserializeRegistry.registerClass(OrderedSet, (type, state) => {
			state.addSetter(`globalThis.serializerTypes.get("OrderedSet").fromJSON(${state.accessor})`)
		})
	}
}

The steps I had a lot of issues with:

• First this type was OrderedSet<T> extends Set<T> but DeepKit didn’t serialize it like it did with Set<T>, the assumed encoded form was object-like not array-like, like it handled Set but that is what I needed.
• Next I tried to set it up with annotateClass<Set>(OrderedSet) but that didn’t deserialize to an OrderedSet, it’s been a while but I think that just decoded to an Array<T>, effectively didn’t transform anything.
• Next I tried the approach above, extending the default serializer. I wanted to declare it in a way that works on a type (i.e. in the same module), that’s why I chose a static {} block for it.
• The examples in the docs only cover types that are available globally so this didn’t work for me either. The code that’s run from addSetter does not have a scope except for a global one. As far as I can see, there is no way to pass OrderedSet to it. Serializing here would be trivial, just output as an array — but I obviously need the actual type for deserialization.
• So next I thought, okay, then I’ll brute force my way through this, set up a globally accessible type registry of my own where I can then read the type from. That worked and deserialize<T>(…) now works as expected with any type using OrderedSet internally.
• However, cast<T>(…) still throws an error. Apparently, it fails at the iterator property in the type and this is where I’m just completely lost. Where am I? serializer.validators? serializer.typeGuards? Somewhere outside serializer?

ValidationError2: Validation error for type ConsumableItem:
tags.() => Symbol.iterator(type): Not a function

So all in all, I have two plus one questions:

1. What would’ve been the expected practice to handle a custom type like this that should effectively just serialize like Set<T> and deserialize via new OrderedSet<T>(…)?
2. How do I fix casting with this custom type?
3. Why is it called ValidationError2?

[marcj]

I tried to implement the use-case you had with almost exactly that code from you and although it is possible to support it, it was not simple and also I found an interesting bug with symbol in methods names (the same you got "Symbol.iterator(type): Not a function"). So I fixed them and added a custom iterable examples in our unit tests: https://github.com/deepkit/deepkit-framework/blob/master/packages/type/tests/use-cases.spec.ts

You have to update to newest version 1.0.1-alpha.155 to make it work as shown in the link. In these examples you find two ways, one with a new function executeTypeArgumentAsArray which does most of the serialization forwarding and one manual way.

To answer your questions:

1. The expected practise in your particular case is to use registerClass and do whatever you want with the data using serializer from createSerializeFunction. This would involve reading the first template argument to know the internal collection type and then create custom serializers and iterate over it. I've added an example to the link above, see "custom iterable manual". See the example and decide on your own what to choose.
2. use createSerializeFunction
3. I don't know what ValidationError2 is. We don't have that in our code base.

[augustsaintfreytag]

Alright, thanks for the patch and the examples, that resolved the issue with decoding types that had iterator conformance. On No.3, this is definitely your ValidationError type, not sure why Vite or whatever in the chain feels the need to rename it.

This project I’m working on is built on React and I’m using both the React SWC and the Deepkit plugins, as per that one sample project. I managed to untangle from the issues I’ve described here only to run into other cases where either types just weren’t available or cast<T> wouldn’t work. The advanced compiler-level integration Deepkit and the docs make it seem like it’s a hands off, “it just works” kind of ordeal but it hasn’t been like that for me at all.

What actually are the limitations?

Do I have to explicitly specialise every generic function call? Do I have to manually append type: ReceiveType<T> as the last parameter of every call not to lose types on the way? A WrapperModel<T> works in a clean demo but not in my actual project and it’s impossible to know why. And the last one that broke me is that cast/deserialize randomly returned undefined for a fully valid model inside a useCallback block.

I’m starting to suspect it has something to do with how types are imported and exported around — but then again, many things seem to work in a clean demo.

[marcj]

What do you mean you are using React SWC and Deepkit plugins? My last knowledge of SWC was that they do not support JS plugins, so it's impossible to hook Deepkit into SWC. I can only assume you use Vite to use merge both SWC and Deepkit plugin. If you disabled minification/optimisations in Vite, you can look at the dist JS source code and see if type information were correctly embedded.

What actually are the limitations?
Do I have to explicitly specialise every generic function call? Do I have to manually append type: ReceiveType as the last parameter of every call not to lose types on the way?

That's too vague for me. Please show minimalistic fully-working source code that does not work for you. This is the most efficient way to get accurate answers.

A WrapperModel works in a clean demo but not in my actual project and it’s impossible to know why

Even in an actual project it's always possible to isolate code into a small file that contains everything needed. Once this file goes through you full build pipeline, you see the JS and can execute it. This way you see quickly what is going on. The less code you have that triggers the wrong behaviour, the better.

And the last one that broke me is that cast/deserialize randomly returned undefined for a fully valid model inside a useCallback block.

You can provide minimalistic working code here as well to get fast solutions. Feel free to join Discord, this way you get even faster help without the need to fight on your own.