A question about the fundamental

[drxxy98]

Can the image feature extraction module in CLIP be used as an encoder to map images from pixel space to latent space? During the training phase of VAE, it would only be necessary to use CLIP's image feature extraction module as the encoder, training it together with the decoder. The advantage of this approach, I believe, is that it would keep the text and image encodings in U-Net within the same latent space. In the current code, it is clear that CLIP and VAE are trained independently. My confusion is whether the encodings of text and images not residing in the same latent space could lead to a mismatch of information. Can U-Net effectively link the information from these two feature spaces? I look forward to your response and interpretation. Thank you very much.