Search Filter Expansion

[Sherlouk]

With what is currently merged in, we currently support the following search filters.

last_activity
last_commit
stars
license
keyword      (keyword:test)
author       (author:apple author:sherlouk)

With pull requests currently ongoing, we will hopefully soon support the following.

Here are some initial ideas which I had that could be extremely helpful.

last_release (last_release:>2021-08-08)
releases     (releases:>5)
dependencies (dependencies:<5)
dependants   (dependants:>100)
platform     (platform:ios platform:!macos)
swift        (swift:5 swift:>4.2)
libraries    (libraries:>0)
executables  (executables:>0)

[Sherlouk]

In theory a keyword or author search with no terms is the same as heading to the existing author page and keyword page. The benefit being that they can add additional filters to that search, or add a search term - ultimately providing more tools for the user to find the package they really want.

[Sherlouk]

Worth noting that there is a minor difference in respect of sorting, but the actual contents is the same. Implemented in #1413.

[Sherlouk]

compatible is an interesting one because it's actually two different filters with potentially the same key? Alternatively we could split it up. What do people think?

I think the distinction is Swift compatibiliy, and OS compatibility.

swift:>5 // Something a version newer than Swift 5
swift:4.2 // Supports Swift 4.2

OR

compatible:>swift-5
compatible:swift-4.2

OS compatibility is a strict match or no match (no greater/less than comparison).

compatible:ios // does support ios
compatible:!linux // doesn't support linux

Remember all queries are "AND" conditions. Do if you want iOS and macOS compatibility you would do compatible:ios compatible:macos.

[daveverwer]

last_release (last_release:>2021-08-08)
releases     (releases:>5)
dependencies (dependencies:<5)
dependants   (dependants:>100)
compatible   (compatible:>swift-5 compatible:linux)
libraries    (libraries:>0)
executables  (executables:>0)
keyword      (keyword:test)
author       (author:apple author:sherlouk)

I'd vote yes for all of these even though I don't see people using things like libraries and executables very often (ever?) as long as they do not add too much complexity to the filtering system. Right now what we have is fairly simple but every one of these that we add brings some kind of maintenance burden with it. As long as that doesn't get out of control, I think all these are good.

Alternatively we could split it up. What do people think?

I think this should be split up into swift and platform:

• swift:5.1
• swift:>5.1
• platform:ios
• swift:>5.5 platform:macos

[Sherlouk]

Note taken on platform and swift. I think that's good naming.

I think executables as a filter will hopefully get more value as people write more CLI tools. Agreed low priority for now though until we see a shift there.

[daveverwer]

Consolidated from #1229 where @Sherlouk suggested:

#1226 (comment)

These are just ideas, and not instructions.

• Support for >= and <=
• Support for OR clauses platform:ios|macos
• Support for grouped clauses ((platform:ios stars:>5) OR (platform:macos stars:<5))
• Support for strings license:"MIT 2.0"

I think >= and <= are no brainers and should be fairly easy to implement. I'm less convinced of the need for the others until we see some usage of filters. This brings up a very tricky point, we don't currently track any use of search more than "someone visited some search results" (which is a good thing) but we also don't see what features of search people are using. We could potentially add some instrumentation around whether filters are being used, and what type of filters, etc...

Something like counters for:

• Every search (so we can calculate percentages)
• Does the search include a filter?
• Does the search include an X filter? (Where X is all the different fields we support) This would need to be multiple different counts so we could still calculate percentages.
• Number of terms in a search
• Number of filters in a search

[Sherlouk]

There's a bit of an assumption that filter operators are one character, not impossible to tweak but needs a bit more testing if we want to add >= and <= operators. Not the highest of priorities though since it's not really unlocking anything new. (>=1 is the same as >0 🤷)

Agree on the search metrics though. I'll get working on that.

[daveverwer]

Agree on the search metrics though. I'll get working on that.

You'll want to run a plan for this past @finestructure as I'd imagine these will go into the Loki/Prom/Grafana system as they're server-side.

[Sherlouk]

Metrics and >= <= operators have been added in #1411.

[finestructure]

I've had a closer look at what it would mean to support platform filters. The SQL query for a single package is as follows:

-- package compatibility
select b.platform
from packages p
join versions v on v.package_id = p.id
join builds b on b.version_id = v.id
where p.url = 'https://github.com/soto-project/soto-core.git'
and v.latest is not null
and b.status = 'ok'
group by b.platform
having count(*) > 0
order by b.platform

This is fast (< 30ms on the amazing M1).

However, in order to support search, we'd need to include it in the materialised view:

            CREATE MATERIALIZED VIEW search AS
            SELECT
              p.id AS package_id,
              ...
              v.package_name,
              array(
				select _b.platform
				from versions _v
				join builds _b on _b.version_id = _v.id
				where _v.package_id = p.id
				and _v.latest is not null
				and _b.status = 'ok'
				group by _b.platform
				having count(*) > 0
				order by _b.platform
              ) AS platform_compatibility
            FROM packages p
              ...

yielding

(We may to consider folding all the macos-* variants into a macos compatibility for sake of simplicity, but that's a detail.)

The problem with this is that creating/refreshing the materialised view takes 24s (again on the M1), and we refresh the view after each analysis pass (running ~every 30s-1min or so). So this would increase the analysis runtime considerably.

There are a couple of things we could do here:

1. Break the search refresh out of the analysis pass and run it on a slower cycle.
2. Add a field platform_compatibility to package and roll up the result of the query when we refresh packages in analysis.
3. Same as 2. but refresh the package when we receive a new build report.

The cons are:

• Running the search refresh on a slower cycle means that newly added packages won't be searchable even though they're reachable for some time, depending on how slow the refresh is.
• We're not looking at builds at all in analysis at the moment so refreshing the package there means loading builds in and updating on a cycle that isn't related to build changes. And we'd be refreshing it much more often than necessary.
• Refresh on receiving a build report means we'd not only add/update a single build record as we do now but also load all others (i.e. dozens) for a given package.

So I think the way to go would be 3. as that's pretty much the least amount of extra work we'd need to do. We'd still need to add the column to the search but it's a straight 1:1 copy from packages with a fixed amount of data.

A very similar mechanism would also work for Swift versions (but I'd take it a step at a time).

[Sherlouk]

Platform and Swift compatibility are both extremely helpful filters (I personally think dependency count is too but clearly that's still up for discussion). I'm trying to process the above message so I can understand the next steps and how to break it down, but will follow up once I have a bit more time (on a work trip for the rest of this week).

[finestructure]

Oh, don't worry. I'll probably start looking into the refreshing and the search schema update tomorrow and we can merge that in first independently before using it in any filters.

[Sherlouk]

With #1432 going through, are you following up with the filter or want me to sort that out? (Happy either way)

Thanks for getting the complex stuff done at least, wasn't looking forward to figuring all those queries out 😂

[finestructure]

:) If you fancy doing the rest that's fine by me. But I can do it as well - I believe you said you're busy this week? Just let me know!

[Sherlouk]

Aye in hotels away from personal laptop until Saturday evening, if there's a desire to speed it through before then, then go ahead! Looking forward to this one 🙌