Future of Sphinx client-side search?

[wlach]

(continuing discussion from #12391)

Earlier this year, @jayaddison and I made various "improvements" to the client-side search implementation. Mostly this just meant fixing bugs and clearly incorrect code. For example:

#11959
#12041
#11957

Unfortunately in at least some cases this actually made client-side search results worse. In particular some of the bugs were hiding bad search results which are now being displayed to the user (as seen in the above issue).

I did a bit of spelunking through the source code again to see what's going on, and I have to say I think the implementation is just not up to modern standards for a feature like this.

The base of it is a very naive term-based search (i.e. just checking to see if a token is in the document at all) where best practice in information retrieval (for decades) is to at least use term frequency–inverse document frequency. On top of that, it looks like people have layered various things to try to augment those results with things like title and object based search but in a way that feels inconsistent and lacks cohesion-- for example, there's two implementations of title-based search (one of which involves magic numbers) which work alongside each other, yielding duplicate results:

sphinx/sphinx/themes/basic/static/searchtools.js

Line 327 in 48cbb43

const queryLower = query.toLowerCase().trim();

sphinx/sphinx/themes/basic/static/searchtools.js

Line 490 in 48cbb43

performTermsSearch: (searchTerms, excludedTerms) => {

I think part of the reason this functionality has languished for so long is due to its lack of visibility-- most Sphinx sites are hosted at readthedocs, which uses its own server-side search implementation. However, very notably the cpython docs (which many people depend on) use the client-side version included with Sphinx, so it's still important that it work correctly. Other non-public sites which don't use readthedocs also use this functionality (this is actually how I got to be interested in this area: my current employer uses Sphinx extensively for internal documentation)

To be honest I'm a little scared to change more minor things, as doing this type of thing correctly across languages can be hard. Any small change is just as likely to make things worse as it is better (as we saw above). Instead, I think a ground-up rewrite is called for.

An example of this done right might be mkdocs-material, and you can clearly see that the author of that spent probably hundreds of hours tweaking and tuning their search implementation to make it work well:

• https://squidfunk.github.io/mkdocs-material/blog/2021/09/13/search-better-faster-smaller/
• https://squidfunk.github.io/mkdocs-material/plugins/search/

I suspect the way forward is to build something new as a Sphinx extension (at least initially) so it can be tested and tuned without impacting the main repository. sphinxcontrib-lunrsearch is probably not exactly what we want (it just provides "search as you type" functionality in the search box) but does show that this type of approach is possible.

[chrisjsewell]

Thanks for the great write up @wlach!

• Is it your intent to create such an extension and do you envision you will have time to do so?
• Is there any changes to sphinx-core required to enable this?
• How much more do you intend to do with the current PRs you are working on here?
  I see there is still work ongoing in HTML Search: omit anchor reference from document titles in the search index. #12047

Also @jayaddison what are your general thoughts?

[wlach]

Thanks for the great write up @wlach!

* Is it your intent to create such an extension and do you envision you will have time to do so?

I think so. A bit of an open question as to when I'll have time to start, have been pretty swamped with $DAY_JOB stuff lately. My guess is the next couple of months. If someone else wants to do something similar, don't let me stop you! Would be happy to contribute to another effort built on similar principles

* Is there any changes to sphinx-core required to enable this?

Not that I know of right now, based on what I've seen out there (sphinx-lunrsearch e.g.) I think there are existing mechanisms I can plug into.

* How much more do you intend to do with the current PRs you are working on here?
  I see there is still work ongoing in [HTML Search: omit anchor reference from document titles in the search index. #12047](https://github.com/sphinx-doc/sphinx/pull/12047)

I'm still happy to help with reviewing/helping with things as I'm able, though I'm personally feeling less inclined to do much more work here for the reasons I outline above. In the case of #12047, I would expect that to be superseded by @jayaddison's #12441

[jayaddison]

Thanks for the great write up @wlach!

👍

I'll admit to being less skeptical of Sphinx client-side search; I think it's one of the better client-side search implementations from what I've seen - but this is still a useful writeup, and yep, the recent regressions are a problem.

Also @jayaddison what are your general thoughts?

As suggested by @wlach, I think that developing an extension and building support for that is a good route to take, in terms of minimizing possible disruption. Figuring out how to ensure that an extension of that kind would receive sufficient real-world evaluation is something I'm wondering about.

Possibly-related: it might be helpful when building an alternative to be able to run the existing test cases from sphinx.git. That should be fine licensing-wise I think; Sphinx is permissively BSD-2 licensed, so the current (and any additional) test cases should be available for comparative checks (not the same as practical human evaluation, but perhaps useful anyway). The tests themselves could also be adapted if the opinions in the implementation change.

Another thought I have at the moment is a non-technical one: a suggestion to be mindful of work/personal balance if using Sphinx for projects at work while also developing a feature for it in your own time. Some of that could be about avoiding later claims from your employer that they own intellectual property contained within it, and some of it is about avoiding doing additional unpaid work (knowingly, unwittingly, or in some kind of gray area in-between).

[jayaddison]

I'm still happy to help with reviewing/helping with things as I'm able, though I'm personally feeling less inclined to do much more work here for the reasons I outline above. In the case of #12047, I would expect that to be superseded by @jayaddison's #12441

Nitpicking a bit here: I think #12047 and #12441 could be merged individually or combined (without superseding each other).

[jayaddison]

From some other research: the Sphinx Extension Survey (last updated Y2020) is a nice project and includes a few search-related extensions; however they appear to be unmaintained nowadays: https://sphinxext-survey.readthedocs.io/en/latest/searchfind.html

[jayaddison]

I mentioned that I think the search functionality is fairly good, especially in comparison to other in-browser engines, but I didn't mention why. I think that:

• Having multiple categories of index (for document text, titles and objects) is a useful feature for technical projects.
• A reasonably large number of stemmers are implemented - and that's a useful internationalization feature.
• The search index is static for each build of the project -- so the index creation cost is paid only once when a project is built, instead of every time the browser loads the page.

I do generally prefer incremental solutions -- and very reluctant to drop any existing functionality -- but I also think that there are some opportunities to improve client-side search performance (ref #12045).

At the moment I'm curious about https://github.com/tinysearch/tinysearch/ and wonder whether anyone else has thoughts/experience with that?

[AA-Turner]

I recently saw tantivy, which might also be of interest.

A

[AA-Turner]

I believe that @kaycebasques has been doing some work on search usability, though I'm not sure it goes as far as the (needed) rewrite that Will mentions.

A