Add color for sampling from mesh

[nikste]

Adds color to point cloud sampled from mesh
(#2483)

Type

• Bug fix (non-breaking change which fixes an issue): Fixes #
• [x ] New feature (non-breaking change which adds functionality). Resolves Sample points from textured mesh #2483
• Breaking change (fix or feature that would cause existing functionality to not work as expected) Resolves #

Motivation and Context

Checklist:

• I have run python util/check_style.py --apply to apply Open3D code style
  to my code.
• This PR changes Open3D behavior or adds new functionality.
  • Both C++ (Doxygen) and Python (Sphinx / Google style) documentation is
    updated accordingly.
  • I have added or updated C++ and / or Python unit tests OR included test
    results (e.g. screenshots or numbers) here.
• I will follow up and update the code if CI fails.
• For fork PRs, I have selected Allow edits from maintainers.

Description

[update-docs]

Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes.

[nikste]

@benjaminum
I'm not an Computer graphics expert. It seems the barycentric coordinates you were referring to (#2483 (comment)) are a sort of mixture weights for triangles (in this case it looks like a,b,c ?). Unfortunately this code seems to samples from the correct colors but from the wrong position in the texture map.

Am I making a mistake here with the uv computation or am I using the PointerAt function wrong ?
Maybe you see a mistake immediately?

picture:
result of current sampling code:

right:

[benjaminum]

Can you try (1-uv(1))*h?

[nikste]

also does not look quite right unfortunately:

[nikste]

could it be that triangle(0), triangle(1), triangle(2) in triangle_uvs_don't select adjacent triangle vertices in the texture?
I'm trying to run it on a cube with different surface colors. This is the texture map:

sampled vertices were:

if we take a weighted average of these sampled points, we might end up in the black area (no valid texture).

[nikste]

I suppose the triangle_index_generator does not take this into account, or something is unexpected in triangle_uvs_

[benjaminum]

@nikste triangle stores the indices of the vertices but we need the index to the right uv.
triangle_uvs_ if I recall correctly should have the size 3*#triangles. Can you try to get the uvs with [3tidx, 3tidx+1, 3*tidx+2]?

[nikste]

@benjaminum
Great! It works now. Thanks for your help!

Couple of questions:

1. Why is textures_ a vector, does it always has only one element ? Do we need to add some different behavior here for multiple elements in textures_?
2. Is there a case that the object could have HasVertexColors() and HasTextures() both are true?
3. Can I assume the texture always has uint8 as values, or should i template the whole sampling function too ?

images from sampled point cloud:
Uniform Sampling:

poisson sampling:

[nikste]

from my side this can be reviewed.

[benjaminum]

@benjaminum Great! It works now. Thanks for your help!

Couple of questions:

1. Why is textures_ a vector, does it always has only one element ? Do we need to add some different behavior here for multiple elements in textures_?
2. Is there a case that the object could have HasVertexColors() and HasTextures() both are true?
3. Can I assume the texture always has uint8 as values, or should i template the whole sampling function too ?

Great! Results look good. I think we now have to answer questions 1-3 and define what we want.

1. There can be multiple materials, see this line

   Open3D/cpp/open3d/geometry/TriangleMesh.h

   Line 855 in fcf98ee

   std::vector<std::pair<std::string, Material>> materials_;

   
   We can look up the material id for each triangle and use that to identify the material and the albedo texture.

2. We can prefer texture over vertex colors to keep the behavior simple.

3. Textures can have different data types