Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make 'list::sort' quick by using natural mergesort #711

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

jiribenes
Copy link
Contributor

@jiribenes jiribenes commented Nov 23, 2024

Please see issue #710 for context and analysis.
(Resolves #710)

It would be nice if somebody benchmarked this properly: the old quicksort, this new quicksort on JS and LLVM + with JS native sort, but I can notice the improvement as this version can suddenly quickly sort huge random lists.

  • As far as I can tell when trying to use it, it's about 2-3x slower than a JS native sort -- I blame the optimiser 😇

Other considerations:

  • we could do some hybrid sort for smaller lists (and it would probably speed it up!), but I don't want the mental overhead of two different sorting algorithms
  • I'm pretty limited here by Effekt not allowing mutually recursive subfunctions -- this means that I need to pass compare everywhere manually.
  • not stacksafe (but works fine for ~5M random elements, so it's good enough for me!)
    • EDIT: I still want to be able to sort a, let's say, ~10M list without getting a OOM :(
  • technically this breaks the previous interface: now the function wants a >= instead of a >
  • it uses a lot of mutual recursion, but I'm willing to optimise stuff a bit more
  • doesn't seem to work well in LLVM

See issue #710 for context and analysis
@jiribenes
Copy link
Contributor Author

jiribenes commented Nov 23, 2024

It seems that this change makes Valgrind very unhappy with the LLVM backend :)

https://github.com/effekt-lang/effekt/actions/runs/11990743350/job/33428584571#step:11:164
Screenshot 2024-11-23 at 23 07 43

Comment on lines 667 to 672
/// When in an ascending sequence, try to add `current` to `run` (if possible)
def ascending[A](current: A, rest: List[A]) { runDiff: List[A] => List[A] } { compare: (A, A) => Bool }: List[List[A]] = rest match {
case Cons(next, tail) and compare(current, next) =>
ascending(next, tail) { diffRest => runDiff(Cons(current, diffRest)) } {compare}
case _ => Cons(runDiff([current]), sequences(rest) {compare})
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function usually uses a difference list (we need to append to the end)
It would probably be interesting to compare it (perf-wise) with:

  • prepending and then .reverse-ing
  • some yield effect

What would be the most idiomatic solution here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still really want to have Daan-style contexts. That would help here, right?

@jiribenes
Copy link
Contributor Author

I tried improving both the performance and the stack safety here by making certain parts (mergeAll) more imperative, but it didn't really work (actually made it measurably slower!)

@jiribenes
Copy link
Contributor Author

In a "real" application (and by that I mean AoC), this change saves 25-50% of the run time:

Before
✓ 1: JS [6.44ms]
✓ 2: JS [103.46ms]
✓ 1: LLVM [0.27ms]
✓ 2: LLVM [17.80ms]

After
✓ 1: JS [5.66ms]
✓ 2: JS [80.25ms]
✓ 1: LLVM [0.12ms]
✓ 2: LLVM [8.72ms]


/// Check if a list is sorted according to the given comparison function.
/// Check if a list is sorted according to the given comparison function (less-or-equal).
///
/// O(N)
def isSortedBy[A](list: List[A]) { compare: (A, A) => Bool }: Bool = {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Consider changing the names of comparison functions from compare to something like lteq or lessThanOrEqual even, just to express properly what we expect of them...

@jiribenes jiribenes added the blocked Blocked on other issues / PRs label Dec 17, 2024
@jiribenes
Copy link
Contributor Author

Blocked on resolving the leak on the LLVM backend. ^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:stdlib blocked Blocked on other issues / PRs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

'list::sort' is very slow
2 participants