StringZilla 4.0! #201
Open
StringZilla 4.0! #201
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`sz_checksum`, `sz_hash`, `sz_edit_distance_utf8`, `sz_edit_distance_bounded`, `sz_edit_distance_utf8_bounded`.
…lla/small_string.h
…lla/small_string.h
…lla/small_string.h
Benchmarks on Sapphire Rapids suggest: - For 8.3 M words in Leipzig1M.txt of length ~5 -- `std::sort` is 2 seconds -- `sz_sort_serial` is 0.6 seconds -- `qsort_r` is 3.2 seconds - For 268 M words in XLSum.csv of length ~8 -- `std::sort` is 147 seconds -- `sz_sort_serial` is 29 seconds -- `qsort_r` is 192 seconds
Makes it easier to differentiate stable `sz_msort`
This huge commit brings many new sorting APIs, as well as a new naming convention to differentiate inplace sorting helpers from "argsort" operations. Also refactors the testing and micro-benchmarking helpers.
It yields no noticeable performance improvements
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR entirely refactors the codebase and separates the single-header implementation into separate headers. Moreover, it brings faster kernels for:
And more community contributions:
basic_charsetinitialization are discarded #200Why Split the Files? Matching SimSIMD Design
Sadly, most of the modern software development tooling stinks. VS Code is just as slow and unresponsive as the older Atom and the other web-based technologies, while LSP implementations for C++ are equally slow and completely mess up code highlighting for files over 5,000 Lines Of Code (LOCs). So, I've unbundled the single-header solution into multiple headers, similar to SimSIMD.
Also, similar to SimSIMD, CPU feature detection has been reworked to separate serial implementations, Haswell, Skylake, Ice Lake, NEON, and SVE.
Faster Sorting
Our old algorithm didn't perform any memory allocations and tried to fit too much into the provided buffers. The new breaking change in the API allows passing a memory allocator, making the implementation more flexible. It now works fine on 32-bit systems as well.
The new serial algorithm is often 5x faster than the
std::sortof C++ Standard Templates Library for a vector of strings. It's also often 10x faster than theqsort_rin the GNU C library. There are even faster versions available for Ice Lake CPUs with AVX-512 and Arm CPUs with SVE.Faster Sequence Alignment & Scoring
Faster Hashing Algorithms
Multi-Pattern Search