Skip to content

Latest commit

 

History

History
17 lines (13 loc) · 837 Bytes

70_Relevance.asciidoc

File metadata and controls

17 lines (13 loc) · 837 Bytes

Stopwords and relevance

The last topic to cover before moving on from stopwords is that of relevance. Leaving stopwords in your index can potentially make the relevance calculation less accurate, especially if your documents are very long.

As we have already discussed in [bm25-saturation], the reason for this is that Term frequency/Inverse document frequency doesn’t impose an upper limit on the impact of term frequency. Very common words may have a low weight because of inverse document frequency but, in long documents, the sheer number of occurrences of stopwords in a single document may lead to their weight being artificially boosted.

You may want to consider using the Okapi BM25 similarity on long fields which include stopwords instead of the default Lucene similarity.