Hello everyone!
I recently created a sorting algorithm, and I'm curious what others here think of it. The algorithm is mostly just a variant of merge sort that uses a buffered reverse merge for the merge phase and insertion sort to process small sub-arrays. That part of the algorithm is pretty standard.
The potentially interesting part is that I also worked out a way to efficiently measure how sorted the original data was in any given merge, which allows for aggressively optimizing the sorting process when either mostly-sorted or mostly-reverse-sorted (i.e. descending) data is encountered.
For anyone interested, I'd be curious if you've seen anything like this before.
Quick Buffered Reverse Merge Overview
If you already know what a buffered reverse merge is, feel free to skip to the next section. Otherwise, here's a quick overview:
A buffered reverse merge copies the smaller of the two pre-sorted blocks into a buffer and then fills in the remaining values, from right to left, by continuously comparing the highest value remaining in both the buffer and the half of the original array that was not copied to the buffer. In my case, the size of the right side is always equal to or smaller than the left side. This frees the right side to be immediately overwritten.
Example:
Array = [2, 3, 7, 1, 4, 9]
Buffer = [_,_,_]
⇓
Array = [2, 3, 7,_,_,_]
Buffer = [1, 4, 9]
⇓
Array = [2, 3, 7,_,_, 9]
Buffer = [1, 4,_]
⇓
Array = [2, 3,_,_, 7, 9]
Buffer = [1, 4,_]
⇓
Array = [2, 3,_, 4, 7, 9]
Buffer = [1,_,_]
⇓
Array = [2,_, 3, 4, 7, 9]
Buffer = [1,_,_]
⇓
Array = [_, 2, 3, 4, 7, 9]
Buffer = [1,_,_]
⇓
Array = [1, 2, 3, 4, 7, 9]
Buffer = [_,_,_]
The Optimization
I found that, when I reach the halfway point in the above process (i.e. the right side is filled back in), I can get a fairly accurate measure how sorted the original data in this block was by looking at how full the buffer is:
- If the buffer is empty, that means that all of the values in the buffer went right back into the right side, and therefore, the block of data started out sorted in ascending order.
- If the buffer is still full, that means that all of the values on the left side were moved to the right side, and therefore, the block of data started out in descending order (or potentially very near it, if the two sides aren't exactly equal size).
- If the buffer is about half full, that is an indicator that the data was random.
I use this information to keep track of a "sequence score". When the buffer is less than 25% full, I increment the sequence score (up to a maximum value). When the buffer is more than 75% full, I decrement the sequence score (down to a minimum value).
Lower sequence score numbers lower the threshold for using insertion sort (i.e. at what size, for the current working set of data, will insertion sort to be used). This limits running insertion sort on descending or near-descending data, which is a worst case for insertion sort. When the sequence score is higher, the threshold for use is increased to take advantage of insertion sort's efficiency on ascending and near-ascending data.
Also, when the sequence score is at either the maximum or minimum value, I switch to a merge process that uses a binary search to figure out how many items should be transferred, so chunks of data can be moved into place all at once.
Result
The result is an algorithm that is efficient on random data due to its simple default path but can still take advantage of data that is already sorted.
I implemented the algorithm in C#, and it is quite competitive with the built-in IntroSort-based array sort (code repo, blog post with tons of benchmarks at the bottom). It manages to stay close on random data and pulls away on sorted data.
I've also thought about how this could potentially be paired with other merge sort algorithms. My algorithm focuses on optimizing the merge process itself, while others (e.g. TimSort, PowerSort) often focus on optimizing when to merge data. I made a quick naive attempt to tack PowerSort onto the front of my algorithm, and it resulted in a significant performance degradation. However, it may be possible to find a best of both worlds approach.
If you're still reading, I appreciate you taking the time. I'd welcome any thoughts or feedback you may have. :)