Some questions about SMIRK

[GrizzCMX]

Hello George!
     Thank you for your work and open-source contributions; it's indeed very enlightening.  I have a few simple questions about your work:
      1) Besides the common encoding of 3DMM params such as shape, expression, global pose+jaw pose, and caw, an additional 'eyelip' parameter is used in smirk. I'm not sure if this corresponds to the eye_pose parameter in the generic FLAME model. Has the FLAME model used in your open-source code been modified as shown in Figures 1 and 2, or does this parameter serve an additional purpose? Can it be used for decoding with the generic FLAME model?
      2) For the same input image (as shown in Figure 3), SMIRK's reconstruction shows a better eye closure effect compared to DECA and Emoca, as shown in Figures 4 and 5. Since SMIRK does not provide code for generating meshes, I used the decoded FLAME vertices from SMIRK combined with the generic FLAME faces to generate an OBJ file. Although SMIRK's reconstruction is closer to the input, a noticeable issue is the overlapping of eyelids and eyeballs, which is not an isolated case (as seen in the zoom in of image). What could be causing this problem, and is there a solution?

                                                                         fig1: the 3DMM params in smirk

                                                                         fig2: the 3DMM params in deca and emoca

fig3: the input image

                            fig4: the smirk result

                        fig5: the emoca result

[filby89]

Hey @GrizzCMX, inline answers follow :)

1. the eyelid parameters independently control the left and right eyelid closure. We got these from the metrical tracker of MICA: https://github.com/Zielon/metrical-tracker/tree/master/flame/blendshapes. Inside FLAME.py you can see how these extra two blendshapes are used in the model. These are essentially extra expression blendshapes because I think original space could not model independent eyelid closure.

2. This seems indeed like an erroneous reconstruction - the process you mentioned for getting the .obj file seems right. This may be attributed to the fact that we have noticed that the eyelid_parameters do have some weird interplay with the other parameters. We are going to look into it and try to reproduce it. In the meantime maybe you can try some very practical stuff such as maybe reducing just a bit the Z coordinate of the eyeball and let us know how it goes.

Btw, our FLAME.py is very similar and practically the same with the FLAME.py from DECA and EMOCA (just renaming and splitting of some variables) with the addition of the eyelids taken from the metrical tracker above.

[GrizzCMX]

Thank you for your response. I also think this issue might be caused by the 'eyelip' parameter not being completely disentangled from other 3DMM parameters. While manually adjusting the position of the eyeballs might improve this phenomenon. However, I have conducted experiments on extensive datasets (such as the MEAD dataset) and found that this issue is not incidental, so manual adjustments are quite limited. I really wish your team consider researching improvements to make this work more applicable to a broader range of downstream tasks (Because its effect is indeed impressive!). Thank you again for your work and open-source contributions.

[emlcpfx]

How do you manually reposition the the eyes?