nr_samples

[Realfriend-Logistics]

Hi, I am enjoying my research thanks to the BERTopic you created.

I'm just posting a question because I need to check something before citing your framework in my paper.

When describing the simple algorithm of KeyBERTInspired, you said that "(...) we randomly sample a number of candidate documents per cluster which is controlled by the nr_samples parameter. (...)".

The size of each topic (cluster) can vary. Some topics might be 10, while others might be 10,000.

From a logical point of view, it seems impossible to set the parameter higher than the size of the smallest topic, and setting it lower would cause representativeness issues from a statistical point of view.

Nevertheless, setting the parameter to 500 seems to produce results for all topics, and I'm confused as to how this is possible. Am I misunderstanding the algorithm or the meaning of the parameter?

[MaartenGr]

If a cluster contains 10 documents and we want to sample 500 documents, it will simply extract all 10 documents from that specific cluster. So setting it higher poses no issues.

[Realfriend-Logistics]

Aha, so if the size of the cluster is smaller than the parameters I set, all the documents in that cluster are sampled? If that's correct, then your statement that there are no statistical issues in this case makes sense. Thanks, your framework is very nice and useful.

[MaartenGr]

That's correct! Thanks for the kind words 😄