Extremely high consistency score across datasets for discrete label

[Jinwoo-Yi]

Dear CEBRA team,

I am currently preparing my thesis using your excellent package, for which I am deeply grateful.

In my work, embeddings were learned based on contrasting discrete labels (including four levels). To calculate the consistency score across datasets, I set n_bins = 4, following the recent discussion here. However, I encountered an issue where all pairs' consistency scores were estimated as 1, an extremely high R-squared value.

Additionally, I observed the same result when calculating this metric using (1) pseudo-CEBRA embeddings with permuted discrete labels and (2) UMAP embeddings. I have attached the script used to compute the consistency score for your reference.

Could you assist me in identifying any potential issues here?

Thank you very much for your continuous support and active interactions with users.

Sincerely,
Jinwoo

[stes]

Hi @Jinwoo-Yi , this is due to the low number of bins, n_bins = 4. How many different discrete labels do you have? The metric becomes "harder" the higher the number of bins.

[Jinwoo-Yi]

@stes thanks for your reply. In my dataset, the auxiliary variable has four discrete labels, so I defined n_bins = 4 considering recent discussion (e.g., #125). Is it okay to set higher values as n_bins in this case? then which value will be appropriate? (P.S. In the attached screenshot, n_bins = 12 but this is a just typo. I confirmed that the results I described originated from n_bins = 4).