EXT_mesh_features Rendering: How to visualize "interpolated" features?

[gkjohnson]

I'm looking into visualizing different feature Ids by rendering them each as different colors for the sake of both debugging and displaying feature coverage but the requirement in the specification doesn't seem easily rendered with shaders:

Implementation note: For a primitive with feature ID attributes, points in the interior of a triangle or line segment should be considered to belong to the feature associated with the nearest vertex.

As far as I understand this kind of interpolation or passing of the value from the "closest" vertex to the fragment shader isn't easily achieveable, at least in WebGL. The flat interpolation modifier only provides the varying from the provoking vertex to the fragment shader so the full triangle will be rendered with the same feature id representation.

Are there other common rendering features I may not be aware of that make this more simple?

[javagl]

I cannot speak about shader implementations in particular (maybe others can chime in here). And while I see the point of trying to visualize feature IDs for debugging and validation, this was not the main purpose behind this 'Implementation Note'. The intention behind this statement was that there may be cases where two vertices of a triangle have different feature IDs (for example, in the case that you described in #763 , but also when the IDs are assigned explicitly). And implementations will have to know how to interpret this case.

It could make sense to interpret this implementation note in the context of 'picking': When you pick an object with the mouse, then you'll intersect the picking ray with the object, and find the intersected triangle. And in order to find out which feature ID this picked point (on the surface of the object) should be associated with, you'll apply this note: The ID will be that of the vertex that is closest to this picked point.

(These are ... "corner cases", insofar that it could be hard to come up with a "sensible meaning" for this ID within the area of a triangle. I'd expect it to appear rarely in real-world applications. But if it appears, it will raise the question of how to handle it, and that's what the implementation note aimed at)

If you have ideas about how this could be improved/clarified (or when you find a way to visualize this in a shader), I'd be curious to hear about that.

The question did also come up for me, when I created the https://github.com/CesiumGS/3d-tiles-samples/tree/main/glTF/EXT_mesh_features/FeatureIdAttribute sample, and I intentionally used a structure where each triangle has a single feature ID for all its vertices - that should be the more common case after all...

[gkjohnson]

while I see the point of trying to visualize feature IDs for debugging and validation

I don't think debugging is the only use case here. I'm imagining highlighting the portion of the model that's associated with the id that the user is hovering their mouse over. Or toggling a visualization that displays everywhere a building is made with a certain material, where pipes are, where doors are, etc. I think there are some strong use cases for shading geometry associated with a specific feature.

or when you find a way to visualize this in a shader

As far as I can tell in order to render this as described you'd need the barycentric coordinate in the pixel shader (which requires a new vec3 attribute per vertex) and the feature ids for each vertex in the triangle (another vec3 attribute). This also means you'd need to de-index any indexed geometry so it's not a super straight forward thing to do.

If you have ideas about how this could be improved/clarified
...
I intentionally used a structure where each triangle has a single feature ID for all its vertices - that should be the more common case after all...

I don't have a great solution other than requiring in the spec that a triangle must have the same feature id at all vertices. Otherwise it might be worth making a note encouraging the use of textures or ensuring triangles have a single feature id when generating the features since handling this visualization use case is fairly difficult otherwise.

[lilleyse]

I don't have a great solution other than requiring in the spec that a triangle must have the same feature id at all vertices.

This sounds like a good constraint. @javagl can you think of a reason NOT to do this?

[javagl]

It sounds like a reasonable constraint for the consuming side (i.e. it makes certain functionalities easier for the renderer). But I'd hesitate to make this a strict requirement in the specification, because I think that it could just "shift" the difficulties to the producing side.

Imagine you have two adjacent 3x3 grids of vertices, meaning 2x8 triangles, with two distinct feature IDs.

A A AB B B
A A AB B B
A A AB B B

And then, some 'vertex weld' and simplification and post-processing is going on, and you end up with this...

A        B

A        B

consisting of only 2 triangles. Now... what are the IDs?

For the producing side, this would raise many questions. The most obvious one: Is it even technically possible to constrain mesh operations like weld and simplification based on the feature IDs?

(I think that this could be a very high burden, and not be possible for many or most mesh libraries at all, but that's a gut feeling for now...)

[gkjohnson]

It sounds like a reasonable constraint for the consuming side ... I think that it could just "shift" the difficulties to the producing side.

The two sides need to be compatible, though. If visualization of the features is going to be considered a use case (and I think it should) then the data needs to be structured such that it can be implemented reasonably. Right now the geometry has to be completely restructured on load in order to accurately visualize the features and that's after you've iterated over all triangles to detect whether the geometry is using these kinds of interpolated features in the first place.

And then, some 'vertex weld' and simplification and post-processing is going on, and you end up with this...

This is no different than any other vertex attribute requirement, though, is it? Disjoint UVs, normals, vertex colors, etc all have to be considered when welding or simplifying geometry. And if simplification requires removing so many details that vertex attributes can't be well represented then they should be encoded to textures such that the data can be more accurately represented on a triangle surface (see embedding higher geometry detail to texture maps, normal maps for lower lods).

[javagl]

I agree that we should try to develop some ideas or approaches here.

The visualization of the IDs themself was not a "primary goal" during the development - or at least not as much as, for example, answering the question about picking. And I would have thought that there should be way to visualize these IDs in a shader somehow. You mentioned that the flat modifier might be used to assign a single color to the whole triangle. And in many cases, that could be "good enough": Imagine two adjacent buildings, each with 1 million triangles. I could imagine that it is not crucially important which part of one triangle(!) belongs to one building. (More on that below)

This is no different than any other vertex attribute requirement, though, is it? Disjoint UVs, normals, vertex colors, etc all have to be considered when welding or simplifying geometry.

Yes. The question is whether such libraries offer the freedom to use arbitrary attributes for constraining these operations, or whether they might be limited to a fixed/known set, like POSITION/NORMAL/TEXCOORD/COLOR (but no "custom" attributes). I don't have an overview over the capabilities of these libraries in that regard.

And if simplification requires removing so many details that vertex attributes can't be well represented then they should be encoded to textures

(Referring to the point above): Indeed, if the question which part of one triangle has a certain feature ID, then one could play Devil's advocate, and say: If this is important, then one should not use feature ID attributes. The feature ID attributes allow assigning IDs on a per-vertex level. If per-texel-level IDs are required, then this can be accomplished with feature ID textures.

But that is not a satisfactory answer (just a form of evading the question). I see the point that it would be nice to have a way to visualize different feature IDs within a triangle, in a form that resembles the (picking-oriented) description in the spec. (Meaning that the triangle has to be rendered as a Voronoi diagram with 3 points...)

I'm not a shader expert, so could hardly make specific proposals. (I might try to allocate some time and play around with custom shaders in CesiumJS, but assume that a 'good', generic solution might require some deeper knowledge here...)

[gkjohnson]

You mentioned that the flat modifier might be used to assign a single color to the whole triangle. And in many cases, that could be "good enough"

My feeling and experience is that if a specification allows for something then people will use it and then complain about it when it doesn't seem to work in a client. Eg the reported id is different from the highlighted section of the model that the cursor is hovered over.

The question is whether such libraries offer the freedom to use arbitrary attributes for constraining these operations, or whether they might be limited to a fixed/known set, like POSITION/NORMAL/TEXCOORD/COLOR (but no "custom" attributes). I don't have an overview over the capabilities of these libraries in that regard.

It may be worth evaluating but if these tools are going to retain the custom attribute data then what are the other reasonable ways to handle disjoint attribute data at overlapping vertices? Discard it? Implicitly average the values for custom attributes without knowing the meaning, even for integer types?

I'm not a shader expert, so could hardly make specific proposals.

I consider myself fairly adept at shaders (at least those available in web platforms) and I don't see a reasonable way to do it:

• In order to perform this kind of manual interpolation you need the barycoord for the fragment (or something similar) which isn't possible without extra vertex information (see here, and implementation in this three.js example which uses extra attributes for wireframes). Geometry shaders are an alternative for generating these attributes necessary for barycoord information on-the-fly but they won't be supported even in WebGPU.
• Similarly we can't encode any relevant data into textures without de-indexing geometry because there's no way to get the triangle index / primitive id in a shader in WebGL or WebGPU.
• The last option is maybe compute shaders but at that point I think the only option is manually rasterizing the triangles and compute shaders aren't supported in WebGL.

It looks like there's a new VK_KHR_fragment_shader_barycentric extension that will enable getting the triangle barycoord (ref 1, ref 2) but who know's if / when this will be supported in WebGPU and even then you still need all the triangle vertex attributes to select the appropriate id to output. Compute shaders may be the only functional approach.

Ultimately this all comes down to whether you want the Ids to be guaranteed to be accurately visualizable or not (I feel they should). Once I get further along in this work I can share a demo of the effect if that helps.

[javagl]

For the pure visualization side of things, there is already an existing issue for CesiumJS to keep track of this. I just posted a few comments and a small test data set at CesiumGS/cesium#9935 (comment)

Right now, the visualization is ... rather an artifact of the specific behavior of shaders for this kind of attribute values. Finding a proper solution that nicely visualizes the areas that have certain IDs may be tricky. (I think that it could make sense to develop some ideas in the CesiumJS issue, but it might very well be that this will affect the specification as well, so it could make sense to continue the discussion either here or there)