From c4792b674688959f1be0538de76eaf1dda1faad6 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 27 Jul 2020 14:23:23 -0700 Subject: [PATCH 001/187] updated benchmark results on README.md with newest measurements featuring XXH3 on recent systems --- README.md | 128 +++++++++++++++++++++++++----------------------------- 1 file changed, 60 insertions(+), 68 deletions(-) diff --git a/README.md b/README.md index 01637f49..ecf96520 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ + xxHash - Extremely fast hash algorithm ====================================== - xxHash is an Extremely fast Hash algorithm, running at RAM speed limits. It successfully completes the [SMHasher](https://code.google.com/p/smhasher/wiki/SMHasher) test suite which evaluates collision, dispersion and randomness qualities of hash functions. -Code is highly portable, and hashes are identical on all platforms (little / big endian). +Code is highly portable, and hashes are identical across all platforms (little / big endian). |Branch |Status | |------------|---------| @@ -13,50 +13,70 @@ Code is highly portable, and hashes are identical on all platforms (little / big |dev | [![Build Status](https://travis-ci.org/Cyan4973/xxHash.svg?branch=dev)](https://travis-ci.org/Cyan4973/xxHash?branch=dev) | - Benchmarks ------------------------- -The benchmark uses SMHasher speed test, compiled with Visual 2010 on a Windows Seven 32-bit box. -The reference system uses a Core 2 Duo @3GHz - - -| Name | Speed | Quality | Author | -|---------------|--------------------|:-------:|-------------------| -| [xxHash] | 5.4 GB/s | 10 | Y.C. | -| MurmurHash 3a | 2.7 GB/s | 10 | Austin Appleby | -| SBox | 1.4 GB/s | 9 | Bret Mulvey | -| Lookup3 | 1.2 GB/s | 9 | Bob Jenkins | -| CityHash64 | 1.05 GB/s | 10 | Pike & Alakuijala | -| FNV | 0.55 GB/s | 5 | Fowler, Noll, Vo | -| CRC32 | 0.43 GB/s † | 9 | | -| MD5-32 | 0.33 GB/s | 10 | Ronald L.Rivest | -| SHA1-32 | 0.28 GB/s | 10 | | - -[xxHash]: https://www.xxhash.com +The benchmark is compiled with clang v10.0 and run on Ubuntu x64 20.04. +The reference system uses Intel i7-9700K + +| Hash Name | Width | Bandwidth (GB/s) | Small Data Velocity | Quality | Comment | +| --------- | ----- | ----------------- | ----- | --- | --- | +| __XXH3__ (SSE2) | 64 | 31.5 GB/s | 133.1 | 10 +| __XXH128__ (SSE2) | 128 | 29.6 GB/s | 118.1 | 10 +| _RAM sequential read_ | N/A | 28.0 GB/s | N/A | N/A +| City64 | 64 | 22.0 GB/s | 76.6 | 10 +| T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collision ratio] +| City128 | 128 | 21.7 GB/s | 57.7 | 10 +| __XXH64__ | 64 | 19.4 GB/s | 71.0 | 10 +| SpookyHash | 64 | 19.3 GB/s | 53.2 | 10 +| Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collision ratio] +| __XXH32__ | 32 | 9.7 GB/s | 71.9 | 10 +| City32 | 32 | 9.1 GB/s | 66.0 | 10 +| Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 +| SipHash | 64 | 3.0 GB/s | 43.2 | 10 +| HighwayHash | 64 | 1.4 GB/s | 6.0 | 10 +| FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties +| Blake2 | 128 | 1.1 GB/s | 5.1 | 10 + +[collision ratio]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study + +note: some algorithms feature _faster than RAM_ speed. In which case, they can only reach their full speed when input data is already in CPU cache (L3 or better). Otherwise, they max out on RAM speed limit. + +### Small data +Performance on large data is only one part of the picture. +Hashing is also very useful in constructions like hash tables and bloom filters. +In these use cases, it's frequent to hash a lot of small data (starting at a few bytes). +Algorithm's performance can be very different for such scenarios, since parts of the algorithm, +such as initialization or finalization, become fixed cost. +The impact of branch mis-prediction also becomes much more present. + +XXH3 has been designed for excellent performance on both long and small inputs, +which can be observed in the following graph: -Note †: SMHasher's CRC32 implementation is known to be slow. Faster implementations exist. +![XXH3, latency, random size](https://user-images.githubusercontent.com/750081/61976089-aedeab00-af9f-11e9-9239-e5375d6c080f.png) -Q.Score is a measure of quality of the hash function. -It depends on successfully passing SMHasher test set. -10 is a perfect score. -Algorithms with a score < 5 are not listed on this table. +For a more detailed analysis, visit the wiki : +https://github.com/Cyan4973/xxHash/wiki/Performance-comparison#benchmarks-concentrating-on-small-data- -A more recent version, XXH64, has been created thanks to [Mathias Westerdahl](https://github.com/JCash), -which offers superior speed and dispersion for 64-bit systems. -Note however that 32-bit applications will still run faster using the 32-bit version. +Quality +------------------------- -SMHasher speed test, compiled using GCC 4.8.2, on Linux Mint 64-bit. -The reference system uses a Core i5-3340M @2.7GHz +Speed is not the only property that matters. +Produced hash values must respect excellent dispersion and randomness properties, +so that any sub-section of it can be used to maximally spread out a table or index, +as well as reduce the amount of collisions to the minimal theoretical level, following the [birthday paradox]. -| Version | Speed on 64-bit | Speed on 32-bit | -|------------|------------------|------------------| -| XXH64 | 13.8 GB/s | 1.9 GB/s | -| XXH32 | 6.8 GB/s | 6.0 GB/s | +`xxHash` has been tested with Austin Appleby's excellent SMHasher test suite, +and passes all tests, ensuring reasonable quality levels. +It also passes extended tests from [newer forks of SMHasher], featuring additional scenarios and conditions. -This project also includes a command line utility, named `xxhsum`, offering similar features to `md5sum`, -thanks to [Takayuki Matsuoka](https://github.com/t-mat)'s contributions. +Finally, xxHash provides its own [massive collision tester](https://github.com/Cyan4973/xxHash/tree/dev/tests/collisions), +able to generate and compare billions of hash to test the limits of 64-bit hash algorithms. +On this front too, xxHash features good results, in line with the [birthday paradox]. +A more detailed analysis is documented [in the wiki](https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison). +[birthday paradox]: https://en.wikipedia.org/wiki/Birthday_problem +[newer forks of SMHasher]: https://github.com/rurban/smhasher ### License @@ -64,30 +84,6 @@ The library files `xxhash.c` and `xxhash.h` are BSD licensed. The utility `xxhsum` is GPL licensed. -### New hash algorithms - -Starting with `v0.7.0`, the library includes a new algorithm named `XXH3`, -which is able to generate 64 and 128-bit hashes. - -The new algorithm is much faster than its predecessors for both long and small inputs, -which can be observed in the following graphs: - -![XXH3, bargraph](https://user-images.githubusercontent.com/750081/61976096-b3a35f00-af9f-11e9-8229-e0afc506c6ec.png) - -![XXH3, latency, random size](https://user-images.githubusercontent.com/750081/61976089-aedeab00-af9f-11e9-9239-e5375d6c080f.png) - -To access these new prototypes, one needs to unlock their declaration, using the build macro `XXH_STATIC_LINKING_ONLY`. - -The algorithm is currently in development, meaning its return values might still change in future versions. -However, the API is stable, and can be used in production, -typically for generation of ephemeral hashes (produced and consumed in same session). - -`XXH3` has now reached "release candidate" status. -If everything remains fine, its format will be "frozen" and become final. -After which, return values of `XXH3` and `XXH128` will no longer change in future versions. -`XXH3`'s return values will be officially finalized upon reaching `v0.8.0`. - - ### Build modifiers The following macros can be set at compilation time to modify libxxhash's behavior. They are generally disabled by default. @@ -216,11 +212,7 @@ thanks to many great contributors. They are [listed here](https://www.xxhash.com/#other-languages). -### Branch Policy - -> - The "master" branch is considered stable, at all times. -> - The "dev" branch is the one where all contributions must be merged - before being promoted to master. -> + If you plan to propose a patch, please commit into the "dev" branch, - or its own feature branch. - Direct commit to "master" are not permitted. +### Special Thanks +Takayuki Matsuoka, aka @t-mat, for creating `xxhsum -c` and general support during early xxh releases +Mathias Westerdahl, aka @JCash, for introducing the first version of `XXH64` +Devin Hussey, aka @easyaspi314, for excellent low-level optimizations on `XXH3` and `XXH128` From 69b894eadeeecfaa429f8dabaf4d62b0e3f0de91 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 27 Jul 2020 14:29:39 -0700 Subject: [PATCH 002/187] fix minor formatting issues especially in the "Thanks" section --- README.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index ecf96520..a4832ab1 100644 --- a/README.md +++ b/README.md @@ -23,13 +23,13 @@ The reference system uses Intel i7-9700K | --------- | ----- | ----------------- | ----- | --- | --- | | __XXH3__ (SSE2) | 64 | 31.5 GB/s | 133.1 | 10 | __XXH128__ (SSE2) | 128 | 29.6 GB/s | 118.1 | 10 -| _RAM sequential read_ | N/A | 28.0 GB/s | N/A | N/A +| _RAM sequential read_ | N/A | 28.0 GB/s | N/A | N/A | _for reference_ | City64 | 64 | 22.0 GB/s | 76.6 | 10 -| T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collision ratio] +| T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collisions] | City128 | 128 | 21.7 GB/s | 57.7 | 10 | __XXH64__ | 64 | 19.4 GB/s | 71.0 | 10 | SpookyHash | 64 | 19.3 GB/s | 53.2 | 10 -| Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collision ratio] +| Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collisions] | __XXH32__ | 32 | 9.7 GB/s | 71.9 | 10 | City32 | 32 | 9.1 GB/s | 66.0 | 10 | Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 @@ -38,9 +38,11 @@ The reference system uses Intel i7-9700K | FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties | Blake2 | 128 | 1.1 GB/s | 5.1 | 10 -[collision ratio]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study +[collisions]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study -note: some algorithms feature _faster than RAM_ speed. In which case, they can only reach their full speed when input data is already in CPU cache (L3 or better). Otherwise, they max out on RAM speed limit. +note 1: Small data velocity is a rough evaluation of algorithm's efficiency on small data. For more detailed information, please refer to next paragraph. + +note 2: some algorithms feature _faster than RAM_ speed. In which case, they can only reach their full speed when input data is already in CPU cache (L3 or better). Otherwise, they max out on RAM speed limit. ### Small data Performance on large data is only one part of the picture. @@ -209,10 +211,13 @@ XXH64_hash_t calcul_hash_streaming(FileHandler fh) Aside from the C reference version, xxHash is also available in many different programming languages, thanks to many great contributors. -They are [listed here](https://www.xxhash.com/#other-languages). +They are [listed here](http://www.xxhash.com/#other-languages). ### Special Thanks + Takayuki Matsuoka, aka @t-mat, for creating `xxhsum -c` and general support during early xxh releases + Mathias Westerdahl, aka @JCash, for introducing the first version of `XXH64` + Devin Hussey, aka @easyaspi314, for excellent low-level optimizations on `XXH3` and `XXH128` From b4dbf5fefc37b8a5f80b3bbc6b8ff639cf5f0ec2 Mon Sep 17 00:00:00 2001 From: Mattias Ellert Date: Tue, 28 Jul 2020 08:01:00 +0200 Subject: [PATCH 003/187] Fix empty version in .pc file --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index ef24e94c..da1ce068 100644 --- a/Makefile +++ b/Makefile @@ -437,7 +437,7 @@ libxxhash.pc: libxxhash.pc.in -e 's|@EXECPREFIX@|$(PCEXECDIR)|' \ -e 's|@LIBDIR@|$(PCLIBDIR)|' \ -e 's|@INCLUDEDIR@|$(PCINCDIR)|' \ - -e 's|@VERSION@|$(VERSION)|' \ + -e 's|@VERSION@|$(LIBVER)|' \ $< > $@ From 50d5774bcf3a2173d85aab6a2d080e854ed49ca9 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 2 Aug 2020 02:05:57 -0700 Subject: [PATCH 004/187] removed highwayhash from benchmark summary the summary is merely there to provide some rough guideline providing comparative performance figures from well known hash algorithms. highway was just added to please a 3rd party request, but results are contested by another 3rd party, this kind of debate is outside the scope of xxhash repository which merely aims at presenting xxhash in context. On the other hand, maybe I should consider adding some very well known algorithms such as md5 or sha1 which are used very often in multiple contexts in order to provide perspective. --- README.md | 72 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 38 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index a4832ab1..a7884c06 100644 --- a/README.md +++ b/README.md @@ -16,35 +16,36 @@ Code is highly portable, and hashes are identical across all platforms (little / Benchmarks ------------------------- -The benchmark is compiled with clang v10.0 and run on Ubuntu x64 20.04. -The reference system uses Intel i7-9700K +The reference system uses an Intel i7-9700K cpu, and runs Ubuntu x64 20.04. +The [open source benchmark program] is compiled with `clang` v10.0 using `-O3` flag. | Hash Name | Width | Bandwidth (GB/s) | Small Data Velocity | Quality | Comment | -| --------- | ----- | ----------------- | ----- | --- | --- | -| __XXH3__ (SSE2) | 64 | 31.5 GB/s | 133.1 | 10 -| __XXH128__ (SSE2) | 128 | 29.6 GB/s | 118.1 | 10 -| _RAM sequential read_ | N/A | 28.0 GB/s | N/A | N/A | _for reference_ -| City64 | 64 | 22.0 GB/s | 76.6 | 10 -| T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collisions] -| City128 | 128 | 21.7 GB/s | 57.7 | 10 -| __XXH64__ | 64 | 19.4 GB/s | 71.0 | 10 -| SpookyHash | 64 | 19.3 GB/s | 53.2 | 10 -| Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collisions] -| __XXH32__ | 32 | 9.7 GB/s | 71.9 | 10 -| City32 | 32 | 9.1 GB/s | 66.0 | 10 -| Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 -| SipHash | 64 | 3.0 GB/s | 43.2 | 10 -| HighwayHash | 64 | 1.4 GB/s | 6.0 | 10 -| FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties -| Blake2 | 128 | 1.1 GB/s | 5.1 | 10 - +| --------- | ----- | ---------------- | ----- | --- | --- | +| __XXH3__ (SSE2) | 64 | 31.5 GB/s | 133.1 | 10 +| __XXH128__ (SSE2) | 128 | 29.6 GB/s | 118.1 | 10 +| _RAM sequential read_ | N/A | 28.0 GB/s | N/A | N/A | _for reference_ +| City64 | 64 | 22.0 GB/s | 76.6 | 10 +| T1ha2 | 64 | 22.0 GB/s | 99.0 | 9 | Slightly worse [collisions] +| City128 | 128 | 21.7 GB/s | 57.7 | 10 +| __XXH64__ | 64 | 19.4 GB/s | 71.0 | 10 +| SpookyHash | 64 | 19.3 GB/s | 53.2 | 10 +| Mum | 64 | 18.0 GB/s | 67.0 | 9 | Slightly worse [collisions] +| __XXH32__ | 32 | 9.7 GB/s | 71.9 | 10 +| City32 | 32 | 9.1 GB/s | 66.0 | 10 +| Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 +| SipHash | 64 | 3.0 GB/s | 43.2 | 10 +| FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties +| Blake2 | 128 | 1.1 GB/s | 5.1 | 10 + +[open source benchmark program]: https://github.com/Cyan4973/xxHash/tree/release/tests/bench [collisions]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study -note 1: Small data velocity is a rough evaluation of algorithm's efficiency on small data. For more detailed information, please refer to next paragraph. +note 1: Small data velocity is a _rough_ evaluation of algorithm's efficiency on small data. For more detailed analysis, please refer to next paragraph. note 2: some algorithms feature _faster than RAM_ speed. In which case, they can only reach their full speed when input data is already in CPU cache (L3 or better). Otherwise, they max out on RAM speed limit. ### Small data + Performance on large data is only one part of the picture. Hashing is also very useful in constructions like hash tables and bloom filters. In these use cases, it's frequent to hash a lot of small data (starting at a few bytes). @@ -80,11 +81,6 @@ A more detailed analysis is documented [in the wiki](https://github.com/Cyan4973 [birthday paradox]: https://en.wikipedia.org/wiki/Birthday_problem [newer forks of SMHasher]: https://github.com/rurban/smhasher -### License - -The library files `xxhash.c` and `xxhash.h` are BSD licensed. -The utility `xxhsum` is GPL licensed. - ### Build modifiers @@ -155,7 +151,8 @@ The xxHash port in vcpkg is kept up to date by Microsoft team members and commun ### Example -Calling xxhash 64-bit variant from a C program: +The simplest example calls xxhash 64-bit variant as a one-shot function +generating a hash value from a single buffer, and invoked from a C/C++ program: ```C #include "xxhash.h" @@ -165,7 +162,8 @@ Calling xxhash 64-bit variant from a C program: } ``` -Using streaming variant is more involved, but makes it possible to provide data incrementally: +Streaming variant is more involved, but makes it possible to provide data incrementally: + ```C #include "stdlib.h" /* abort() */ #include "xxhash.h" @@ -187,17 +185,17 @@ XXH64_hash_t calcul_hash_streaming(FileHandler fh) /* Feed the state with input data, any size, any number of times */ (...) - while ( /* any condition */ ) { + while ( /* some data left */ ) { size_t const length = get_more_data(buffer, bufferSize, fh); if (XXH64_update(state, buffer, length) == XXH_ERROR) abort(); (...) } (...) - /* Get the hash */ + /* Produce the final hash value */ XXH64_hash_t const hash = XXH64_digest(state); - /* State can be re-used; in this example, it is simply freed */ + /* State could be re-used; but in this example, it is simply freed */ free(buffer); XXH64_freeState(state); @@ -206,11 +204,17 @@ XXH64_hash_t calcul_hash_streaming(FileHandler fh) ``` +### License + +The library files `xxhash.c` and `xxhash.h` are BSD licensed. +The utility `xxhsum` is GPL licensed. + + ### Other programming languages -Aside from the C reference version, -xxHash is also available in many different programming languages, -thanks to many great contributors. +Beyond the C reference version, +xxHash is also available from many different programming languages, +thanks to great contributors. They are [listed here](http://www.xxhash.com/#other-languages). From 95014352caf0da5162bb3677ba5a7c0f7ddcd895 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 2 Aug 2020 02:57:25 -0700 Subject: [PATCH 005/187] added MD5 benchmark results --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a7884c06..78978826 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,8 @@ The [open source benchmark program] is compiled with `clang` v10.0 using `-O3` f | Murmur3 | 32 | 3.9 GB/s | 56.1 | 10 | SipHash | 64 | 3.0 GB/s | 43.2 | 10 | FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties -| Blake2 | 128 | 1.1 GB/s | 5.1 | 10 +| Blake2 | 256 | 1.1 GB/s | 5.1 | 10 | Cryptographic +| MD5 | 128 | 0.6 GB/s | 7.8 | 10 | Cryptographic but broken [open source benchmark program]: https://github.com/Cyan4973/xxHash/tree/release/tests/bench [collisions]: https://github.com/Cyan4973/xxHash/wiki/Collision-ratio-comparison#collision-study From b2a1eba413e67426b7b177288f033ac5c1e3b805 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sun, 2 Aug 2020 03:19:21 -0700 Subject: [PATCH 006/187] added SHA1 results --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 78978826..1207969a 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,7 @@ The [open source benchmark program] is compiled with `clang` v10.0 using `-O3` f | SipHash | 64 | 3.0 GB/s | 43.2 | 10 | FNV64 | 64 | 1.2 GB/s | 62.7 | 5 | Poor avalanche properties | Blake2 | 256 | 1.1 GB/s | 5.1 | 10 | Cryptographic +| SHA1 | 160 | 0.8 GB/s | 5.6 | 10 | Cryptographic but broken | MD5 | 128 | 0.6 GB/s | 7.8 | 10 | Cryptographic but broken [open source benchmark program]: https://github.com/Cyan4973/xxHash/tree/release/tests/bench From 7e1e4a7d6f7577d7850d3912c235ebf740b19517 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Thu, 13 Aug 2020 00:43:59 -0700 Subject: [PATCH 007/187] minor fix url in pkgconfig since https doesn't work anymore ... --- libxxhash.pc.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libxxhash.pc.in b/libxxhash.pc.in index 0a52dde2..28c16448 100644 --- a/libxxhash.pc.in +++ b/libxxhash.pc.in @@ -9,7 +9,7 @@ libdir=${exec_prefix}/@LIBDIR@ Name: xxhash Description: extremely fast hash algorithm -URL: https://www.xxhash.com/ +URL: http://www.xxhash.com/ Version: @VERSION@ Libs: -L${libdir} -lxxhash Cflags: -I${includedir} From 9481f42a629f315dacf6df8a6029451c1f9e3974 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20Hamb=C3=BCchen?= Date: Thu, 13 Aug 2020 18:34:19 +0200 Subject: [PATCH 008/187] xxhash.h: Update "still in development" comments v0.8.0 marked XXH3 as stable. --- xxhash.h | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/xxhash.h b/xxhash.h index 2d56d23c..400d3a21 100644 --- a/xxhash.h +++ b/xxhash.h @@ -440,19 +440,14 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src * * The 128-bit version adds additional strength, but it is slightly slower. * - * The XXH3 algorithm is still in development. - * The results it produces may still change in future versions. + * Return values of XXH3 and XXH128 are officially finalized starting + * with v0.8.0 and will no longer change in future versions. + * Avoid storing values from before that release in long-term storage. * * Results produced by v0.7.x are not comparable with results from v0.7.y. * However, the API is completely stable, and it can safely be used for * ephemeral data (local sessions). * - * Avoid storing values in long-term storage until the algorithm is finalized. - * XXH3's return values will be officially finalized upon reaching v0.8.0. - * - * After which, return values of XXH3 and XXH128 will no longer change in - * future versions. - * * The API supports one-shot hashing, streaming mode, and custom secrets. */ From 96bd569adc71e01df0930836792de36cd7742883 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 2 Sep 2020 10:11:36 -0400 Subject: [PATCH 009/187] [WIP] Start prefixing most symbols in xxhsum.c Make most symbols start with XSUM_ to prepare for future refactoring. Some macros are no longer SCREAMING_SNAKE_CASE, they will be converted to functions in the refactor. --- xxhsum.c | 767 +++++++++++++++++++++++++++---------------------------- 1 file changed, 378 insertions(+), 389 deletions(-) diff --git a/xxhsum.c b/xxhsum.c index 565eb998..4f790a1a 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -97,20 +97,20 @@ || defined(__DJGPP__) \ || defined(__MSYS__) # include /* isatty */ -# define IS_CONSOLE(stdStream) isatty(fileno(stdStream)) +# define XSUM_isConsole(stdStream) isatty(fileno(stdStream)) #elif defined(MSDOS) || defined(OS2) # include /* _isatty */ -# define IS_CONSOLE(stdStream) _isatty(_fileno(stdStream)) +# define XSUM_isConsole(stdStream) _isatty(_fileno(stdStream)) #elif defined(WIN32) || defined(_WIN32) # include /* _isatty */ # include /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */ # include /* FILE */ -static __inline int IS_CONSOLE(FILE* stdStream) { +static __inline int XSUM_isConsole(FILE* stdStream) { DWORD dummy; return _isatty(_fileno(stdStream)) && GetConsoleMode((HANDLE)_get_osfhandle(_fileno(stdStream)), &dummy); } #else -# define IS_CONSOLE(stdStream) 0 +# define XSUM_isConsole(stdStream) 0 #endif #if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) @@ -137,7 +137,7 @@ static __inline int IS_CONSOLE(FILE* stdStream) { * Converts a UTF-8 string to UTF-16. Acts like strdup. The string must be freed afterwards. * This version allows keeping the output length. */ -static wchar_t* utf8_to_utf16_len(const char* str, int* lenOut) +static wchar_t* XSUM_widenString(const char* str, int* lenOut) { int const len = MultiByteToWideChar(CP_UTF8, 0, str, -1, NULL, 0); if (lenOut != NULL) *lenOut = len; @@ -152,17 +152,11 @@ static wchar_t* utf8_to_utf16_len(const char* str, int* lenOut) } } -/* Converts a UTF-8 string to UTF-16. Acts like strdup. The string must be freed afterwards. */ -static wchar_t* utf8_to_utf16(const char *str) -{ - return utf8_to_utf16_len(str, NULL); -} - /* * Converts a UTF-16 string to UTF-8. Acts like strdup. The string must be freed afterwards. * This version allows keeping the output length. */ -static char* utf16_to_utf8_len(const wchar_t *str, int *lenOut) +static char* XSUM_narrowString(const wchar_t *str, int *lenOut) { int len = WideCharToMultiByte(CP_UTF8, 0, str, -1, NULL, 0, NULL, NULL); if (lenOut != NULL) *lenOut = len; @@ -177,12 +171,6 @@ static char* utf16_to_utf8_len(const wchar_t *str, int *lenOut) } } -/* Converts a UTF-16 string to UTF-8. Acts like strdup. The string must be freed afterwards. */ -static char *utf16_to_utf8(const wchar_t *str) -{ - return utf16_to_utf8_len(str, NULL); -} - /* * fopen wrapper that supports UTF-8 * @@ -190,9 +178,9 @@ static char *utf16_to_utf8(const wchar_t *str) * * In order to open a Unicode filename, we need to convert filenames to UTF-16 and use _wfopen. */ -static FILE* XXH_fopen_wrapped(const char *filename, const wchar_t *mode) +static FILE* XSUM_fopen_wrapped(const char *filename, const wchar_t *mode) { - wchar_t* const wide_filename = utf8_to_utf16(filename); + wchar_t* const wide_filename = XSUM_widenString(filename, NULL); if (wide_filename == NULL) return NULL; { FILE* const f = _wfopen(wide_filename, mode); free(wide_filename); @@ -219,7 +207,7 @@ static FILE* XXH_fopen_wrapped(const char *filename, const wchar_t *mode) * * Credit to t-mat: https://github.com/t-mat/xxHash/commit/5691423 */ -static int fprintf_utf8(FILE *stream, const char *format, ...) +static int XSUM_fprintf_utf8(FILE *stream, const char *format, ...) { int result; va_list args; @@ -251,7 +239,7 @@ static int fprintf_utf8(FILE *stream, const char *format, ...) u8_str[nchar - 1] = '\0'; if (result > 0) { /* - * Check if we are outputting to a console. Don't use IS_CONSOLE + * Check if we are outputting to a console. Don't use XSUM_isConsole * directly -- we don't need to call _get_osfhandle twice. */ int fileNb = _fileno(stream); @@ -269,7 +257,7 @@ static int fprintf_utf8(FILE *stream, const char *format, ...) * default msvcrt.dll. */ int len; - wchar_t *const u16_buf = utf8_to_utf16_len(u8_str, &len); + wchar_t *const u16_buf = XSUM_widenString(u8_str, &len); if (u16_buf == NULL) { result = -1; } else { @@ -298,9 +286,9 @@ static int fprintf_utf8(FILE *stream, const char *format, ...) * Since we always use literals in the "mode" argument, it is just easier to append "L" to * the string to make it UTF-16 and avoid the hassle of a second manual conversion. */ -# define XXH_fopen(filename, mode) XXH_fopen_wrapped(filename, L##mode) +# define XSUM_fopen(filename, mode) XSUM_fopen_wrapped(filename, L##mode) #else -# define XXH_fopen(filename, mode) fopen(filename, mode) +# define XSUM_fopen(filename, mode) fopen(filename, mode) #endif /* ************************************ @@ -323,7 +311,7 @@ static int fprintf_utf8(FILE *stream, const char *format, ...) typedef unsigned long long U64; #endif /* not C++/C99 */ -static unsigned BMK_isLittleEndian(void) +static unsigned XSUM_isLittleEndian(void) { const union { U32 u; U8 c[4]; } one = { 1 }; /* don't use static: performance detrimental */ return one.c[0]; @@ -449,7 +437,7 @@ static unsigned BMK_isLittleEndian(void) static const int g_nbBits = (int)(sizeof(void*)*8); static const char g_lename[] = "little endian"; static const char g_bename[] = "big endian"; -#define ENDIAN_NAME (BMK_isLittleEndian() ? g_lename : g_bename) +#define ENDIAN_NAME (XSUM_isLittleEndian() ? g_lename : g_bename) static const char author[] = "Yann Collet"; #define WELCOME_MESSAGE(exename) "%s %s by %s \n", exename, PROGRAM_VERSION, author #define FULL_WELCOME_MESSAGE(exename) "%s %s by %s \n" \ @@ -461,7 +449,7 @@ static const char author[] = "Yann Collet"; #define MB *( 1<<20) #define GB *(1U<<30) -static size_t XXH_DEFAULT_SAMPLE_SIZE = 100 KB; +static size_t XSUM_DEFAULT_SAMPLE_SIZE = 100 KB; #define NBLOOPS 3 /* Default number of benchmark iterations */ #define TIMELOOP_S 1 #define TIMELOOP (TIMELOOP_S * CLOCKS_PER_SEC) /* target timing per iteration */ @@ -473,7 +461,7 @@ static size_t XXH_DEFAULT_SAMPLE_SIZE = 100 KB; static const char stdinName[] = "-"; typedef enum { algo_xxh32=0, algo_xxh64=1, algo_xxh128=2 } AlgoSelected; -static AlgoSelected g_defaultAlgo = algo_xxh64; /* required within main() & usage() */ +static AlgoSelected g_defaultAlgo = algo_xxh64; /* required within main() & XSUM_usage() */ /* <16 hex char> <'\0'> * '4096' is typical Linux PATH_MAX configuration. */ @@ -487,14 +475,14 @@ static AlgoSelected g_defaultAlgo = algo_xxh64; /* required within main() & u * Display macros **************************************/ #ifdef _WIN32 -#define DISPLAY(...) fprintf_utf8(stderr, __VA_ARGS__) -#define DISPLAYRESULT(...) fprintf_utf8(stdout, __VA_ARGS__) +#define XSUM_log(...) XSUM_fprintf_utf8(stderr, __VA_ARGS__) +#define XSUM_output(...) XSUM_fprintf_utf8(stdout, __VA_ARGS__) #else -#define DISPLAY(...) fprintf(stderr, __VA_ARGS__) -#define DISPLAYRESULT(...) fprintf(stdout, __VA_ARGS__) +#define XSUM_log(...) fprintf(stderr, __VA_ARGS__) +#define XSUM_output(...) fprintf(stdout, __VA_ARGS__) #endif -#define DISPLAYLEVEL(l, ...) do { if (g_displayLevel>=l) DISPLAY(__VA_ARGS__); } while (0) +#define XSUM_logVerbose(l, ...) do { if (g_displayLevel>=l) XSUM_log(__VA_ARGS__); } while (0) static int g_displayLevel = 2; @@ -507,13 +495,13 @@ static U32 g_nbIterations = NBLOOPS; /* ************************************ * Benchmark Functions **************************************/ -static clock_t BMK_clockSpan( clock_t start ) +static clock_t XSUM_clockSpan( clock_t start ) { return clock() - start; /* works even if overflow; Typical max span ~ 30 mn */ } -static size_t BMK_findMaxMem(U64 requiredMem) +static size_t XSUM_findMaxMem(U64 requiredMem) { size_t const step = 64 MB; void* testmem = NULL; @@ -537,7 +525,7 @@ static size_t BMK_findMaxMem(U64 requiredMem) } -static U64 BMK_GetFileSize(const char* infilename) +static U64 XSUM_GetFileSize(const char* infilename) { int r; #if defined(_MSC_VER) @@ -555,7 +543,7 @@ static U64 BMK_GetFileSize(const char* infilename) * Allocates a string containing s1 and s2 concatenated. Acts like strdup. * The result must be freed. */ -static char* XXH_strcatDup(const char* s1, const char* s2) +static char* XSUM_strcatDup(const char* s1, const char* s2) { assert(s1 != NULL); assert(s2 != NULL); @@ -582,7 +570,7 @@ static char* XXH_strcatDup(const char* s1, const char* s2) * * This is used in the sanity check - its values must not be changed. */ -static void BMK_fillTestBuffer(U8* buffer, size_t len) +static void XSUM_fillTestBuffer(U8* buffer, size_t len) { U64 byteGen = PRIME32; size_t i; @@ -713,20 +701,20 @@ static const char k_testIDs_default[NB_TESTFUNC] = { 0, 1 /*XXH128*/ }; #define HASHNAME_MAX 29 -static void BMK_benchHash(hashFunction h, const char* hName, int testID, +static void XSUM_benchHash(hashFunction h, const char* hName, int testID, const void* buffer, size_t bufferSize) { U32 nbh_perIteration = (U32)((300 MB) / (bufferSize+1)) + 1; /* first iteration conservatively aims for 300 MB/s */ unsigned iterationNb, nbIterations = g_nbIterations + !g_nbIterations /* min 1 */; double fastestH = 100000000.; assert(HASHNAME_MAX > 2); - DISPLAYLEVEL(2, "\r%80s\r", ""); /* Clean display line */ + XSUM_logVerbose(2, "\r%80s\r", ""); /* Clean display line */ for (iterationNb = 1; iterationNb <= nbIterations; iterationNb++) { U32 r=0; clock_t cStart; - DISPLAYLEVEL(2, "%2u-%-*.*s : %10u ->\r", + XSUM_logVerbose(2, "%2u-%-*.*s : %10u ->\r", iterationNb, HASHNAME_MAX, HASHNAME_MAX, hName, (unsigned)bufferSize); @@ -738,9 +726,9 @@ static void BMK_benchHash(hashFunction h, const char* hName, int testID, for (u=0; u0.) { /* avoid div by zero */ - DISPLAYLEVEL(2, "%2u-%-*.*s : %10u -> %8.0f it/s (%7.1f MB/s) \r", + XSUM_logVerbose(2, "%2u-%-*.*s : %10u -> %8.0f it/s (%7.1f MB/s) \r", iterationNb, HASHNAME_MAX, HASHNAME_MAX, hName, (unsigned)bufferSize, @@ -793,27 +781,27 @@ static void BMK_benchHash(hashFunction h, const char* hName, int testID, nbh_perIteration = (U32)nbh_perSecond; } } - DISPLAYLEVEL(1, "%2i#%-*.*s : %10u -> %8.0f it/s (%7.1f MB/s) \n", + XSUM_logVerbose(1, "%2i#%-*.*s : %10u -> %8.0f it/s (%7.1f MB/s) \n", testID, HASHNAME_MAX, HASHNAME_MAX, hName, (unsigned)bufferSize, (double)1 / fastestH, ((double)bufferSize / (1 MB)) / fastestH); if (g_displayLevel<1) - DISPLAYLEVEL(0, "%u, ", (unsigned)((double)1 / fastestH)); + XSUM_logVerbose(0, "%u, ", (unsigned)((double)1 / fastestH)); } /*! - * BMK_benchMem(): + * XSUM_benchMem(): * buffer: Must be 16-byte aligned. * The real allocated size of buffer is supposed to be >= (bufferSize+3). * returns: 0 on success, 1 if error (invalid mode selected) */ -static void BMK_benchMem(const void* buffer, size_t bufferSize) +static void XSUM_benchMem(const void* buffer, size_t bufferSize) { assert((((size_t)buffer) & 15) == 0); /* ensure alignment */ - BMK_fillTestBuffer(g_benchSecretBuf, sizeof(g_benchSecretBuf)); + XSUM_fillTestBuffer(g_benchSecretBuf, sizeof(g_benchSecretBuf)); { int i; for (i = 1; i < NB_TESTFUNC; i++) { int const hashFuncID = (i-1) / 2; @@ -821,51 +809,51 @@ static void BMK_benchMem(const void* buffer, size_t bufferSize) if (g_testIDs[i] == 0) continue; /* aligned */ if ((i % 2) == 1) { - BMK_benchHash(g_hashesToBench[hashFuncID].func, g_hashesToBench[hashFuncID].name, i, buffer, bufferSize); + XSUM_benchHash(g_hashesToBench[hashFuncID].func, g_hashesToBench[hashFuncID].name, i, buffer, bufferSize); } /* unaligned */ if ((i % 2) == 0) { /* Append "unaligned". */ - char* const hashNameBuf = XXH_strcatDup(g_hashesToBench[hashFuncID].name, " unaligned"); + char* const hashNameBuf = XSUM_strcatDup(g_hashesToBench[hashFuncID].name, " unaligned"); assert(hashNameBuf != NULL); - BMK_benchHash(g_hashesToBench[hashFuncID].func, hashNameBuf, i, ((const char*)buffer)+3, bufferSize); + XSUM_benchHash(g_hashesToBench[hashFuncID].func, hashNameBuf, i, ((const char*)buffer)+3, bufferSize); free(hashNameBuf); } } } } -static size_t BMK_selectBenchedSize(const char* fileName) +static size_t XSUM_selectBenchedSize(const char* fileName) { - U64 const inFileSize = BMK_GetFileSize(fileName); - size_t benchedSize = (size_t) BMK_findMaxMem(inFileSize); + U64 const inFileSize = XSUM_GetFileSize(fileName); + size_t benchedSize = (size_t) XSUM_findMaxMem(inFileSize); if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) { - DISPLAY("Not enough memory for '%s' full size; testing %i MB only...\n", fileName, (int)(benchedSize>>20)); + XSUM_log("Not enough memory for '%s' full size; testing %i MB only...\n", fileName, (int)(benchedSize>>20)); } return benchedSize; } -static int BMK_benchFiles(const char*const* fileNamesTable, int nbFiles) +static int XSUM_benchFiles(const char*const* fileNamesTable, int nbFiles) { int fileIdx; for (fileIdx=0; fileIdx 10 KB) { - DISPLAYLEVEL(1, "%u KB", (unsigned)(keySize >> 10)); + XSUM_logVerbose(1, "%u KB", (unsigned)(keySize >> 10)); } else { - DISPLAYLEVEL(1, "%u bytes", (unsigned)keySize); + XSUM_logVerbose(1, "%u bytes", (unsigned)keySize); } - DISPLAYLEVEL(1, "... \n"); + XSUM_logVerbose(1, "... \n"); - BMK_benchMem(alignedBuffer, keySize); + XSUM_benchMem(alignedBuffer, keySize); free(buffer); } return 0; @@ -919,50 +907,50 @@ static int BMK_benchInternal(size_t keySize) * ensure results consistency accross platforms *********************************************** */ -static void BMK_checkResult32(XXH32_hash_t r1, XXH32_hash_t r2) +static void XSUM_checkResult32(XXH32_hash_t r1, XXH32_hash_t r2) { static int nbTests = 1; if (r1!=r2) { - DISPLAY("\rError: 32-bit hash test %i: Internal sanity check failed!\n", nbTests); - DISPLAY("\rGot 0x%08X, expected 0x%08X.\n", (unsigned)r1, (unsigned)r2); - DISPLAY("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in BMK_sanityCheck.\n"); + XSUM_log("\rError: 32-bit hash test %i: Internal sanity check failed!\n", nbTests); + XSUM_log("\rGot 0x%08X, expected 0x%08X.\n", (unsigned)r1, (unsigned)r2); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily comment out the tests in XSUM_sanityCheck.\n"); exit(1); } nbTests++; } -static void BMK_checkResult64(XXH64_hash_t r1, XXH64_hash_t r2) +static void XSUM_checkResult64(XXH64_hash_t r1, XXH64_hash_t r2) { static int nbTests = 1; if (r1!=r2) { - DISPLAY("\rError: 64-bit hash test %i: Internal sanity check failed!\n", nbTests); - DISPLAY("\rGot 0x%08X%08XULL, expected 0x%08X%08XULL.\n", + XSUM_log("\rError: 64-bit hash test %i: Internal sanity check failed!\n", nbTests); + XSUM_log("\rGot 0x%08X%08XULL, expected 0x%08X%08XULL.\n", (unsigned)(r1>>32), (unsigned)r1, (unsigned)(r2>>32), (unsigned)r2); - DISPLAY("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in BMK_sanityCheck.\n"); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily comment out the tests in XSUM_sanityCheck.\n"); exit(1); } nbTests++; } -static void BMK_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) +static void XSUM_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) { static int nbTests = 1; if ((r1.low64 != r2.low64) || (r1.high64 != r2.high64)) { - DISPLAY("\rError: 128-bit hash test %i: Internal sanity check failed.\n", nbTests); - DISPLAY("\rGot { 0x%08X%08XULL, 0x%08X%08XULL }, expected { 0x%08X%08XULL, 0x%08X%08XULL } \n", + XSUM_log("\rError: 128-bit hash test %i: Internal sanity check failed.\n", nbTests); + XSUM_log("\rGot { 0x%08X%08XULL, 0x%08X%08XULL }, expected { 0x%08X%08XULL, 0x%08X%08XULL } \n", (unsigned)(r1.low64>>32), (unsigned)r1.low64, (unsigned)(r1.high64>>32), (unsigned)r1.high64, (unsigned)(r2.low64>>32), (unsigned)r2.low64, (unsigned)(r2.high64>>32), (unsigned)r2.high64 ); - DISPLAY("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in BMK_sanityCheck.\n"); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily comment out the tests in XSUM_sanityCheck.\n"); exit(1); } nbTests++; } -static void BMK_testXXH32(const void* data, size_t len, U32 seed, U32 Nresult) +static void XSUM_testXXH32(const void* data, size_t len, U32 seed, U32 Nresult) { XXH32_state_t *state = XXH32_createState(); size_t pos; @@ -970,20 +958,20 @@ static void BMK_testXXH32(const void* data, size_t len, U32 seed, U32 Nresult) assert(state != NULL); if (len>0) assert(data != NULL); - BMK_checkResult32(XXH32(data, len, seed), Nresult); + XSUM_checkResult32(XXH32(data, len, seed), Nresult); (void)XXH32_reset(state, seed); (void)XXH32_update(state, data, len); - BMK_checkResult32(XXH32_digest(state), Nresult); + XSUM_checkResult32(XXH32_digest(state), Nresult); (void)XXH32_reset(state, seed); for (pos=0; pos0) assert(data != NULL); - BMK_checkResult64(XXH64(data, len, seed), Nresult); + XSUM_checkResult64(XXH64(data, len, seed), Nresult); (void)XXH64_reset(state, seed); (void)XXH64_update(state, data, len); - BMK_checkResult64(XXH64_digest(state), Nresult); + XSUM_checkResult64(XXH64_digest(state), Nresult); (void)XXH64_reset(state, seed); for (pos=0; pos0) assert(data != NULL); { U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); - BMK_checkResult64(Dresult, Nresult); + XSUM_checkResult64(Dresult, Nresult); } /* check that the no-seed variant produces same result as seed==0 */ if (seed == 0) { U64 const Dresult = XXH3_64bits(data, len); - BMK_checkResult64(Dresult, Nresult); + XSUM_checkResult64(Dresult, Nresult); } /* streaming API test */ @@ -1032,19 +1020,19 @@ void BMK_testXXH3(const void* data, size_t len, U64 seed, U64 Nresult) /* single ingestion */ (void)XXH3_64bits_reset_withSeed(state, seed); (void)XXH3_64bits_update(state, data, len); - BMK_checkResult64(XXH3_64bits_digest(state), Nresult); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* random ingestion */ { size_t p = 0; (void)XXH3_64bits_reset_withSeed(state, seed); while (p < len) { size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(BMK_rand()) % modulo; + size_t l = (size_t)(XSUM_rand()) % modulo; if (p + l > len) l = len - p; (void)XXH3_64bits_update(state, (const char*)data+p, l); p += l; } - BMK_checkResult64(XXH3_64bits_digest(state), Nresult); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); } /* byte by byte ingestion */ @@ -1052,18 +1040,18 @@ void BMK_testXXH3(const void* data, size_t len, U64 seed, U64 Nresult) (void)XXH3_64bits_reset_withSeed(state, seed); for (pos=0; pos0) assert(data != NULL); { U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); - BMK_checkResult64(Dresult, Nresult); + XSUM_checkResult64(Dresult, Nresult); } /* streaming API test */ @@ -1071,19 +1059,19 @@ void BMK_testXXH3_withSecret(const void* data, size_t len, const void* secret, s assert(state != NULL); (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); (void)XXH3_64bits_update(state, data, len); - BMK_checkResult64(XXH3_64bits_digest(state), Nresult); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* random ingestion */ { size_t p = 0; (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); while (p < len) { size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(BMK_rand()) % modulo; + size_t l = (size_t)(XSUM_rand()) % modulo; if (p + l > len) l = len - p; (void)XXH3_64bits_update(state, (const char*)data+p, l); p += l; } - BMK_checkResult64(XXH3_64bits_digest(state), Nresult); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); } /* byte by byte ingestion */ @@ -1091,27 +1079,27 @@ void BMK_testXXH3_withSecret(const void* data, size_t len, const void* secret, s (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); for (pos=0; pos 2 ? len : 2; - size_t l = (size_t)(BMK_rand()) % modulo; + size_t l = (size_t)(XSUM_rand()) % modulo; if (p + l > len) l = len - p; (void)XXH3_128bits_update(state, (const char*)data+p, l); p += l; } - BMK_checkResult128(XXH3_128bits_digest(state), Nresult); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); } /* byte by byte ingestion */ @@ -1141,18 +1129,18 @@ void BMK_testXXH128(const void* data, size_t len, U64 seed, XXH128_hash_t Nresul (void)XXH3_128bits_reset_withSeed(state, seed); for (pos=0; pos0) assert(data != NULL); { XXH128_hash_t const Dresult = XXH3_128bits_withSecret(data, len, secret, secretSize); - BMK_checkResult128(Dresult, Nresult); + XSUM_checkResult128(Dresult, Nresult); } /* streaming API test */ @@ -1160,19 +1148,19 @@ void BMK_testXXH128_withSecret(const void* data, size_t len, const void* secret, assert(state != NULL); (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); (void)XXH3_128bits_update(state, data, len); - BMK_checkResult128(XXH3_128bits_digest(state), Nresult); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); /* random ingestion */ { size_t p = 0; (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); while (p < len) { size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(BMK_rand()) % modulo; + size_t l = (size_t)(XSUM_rand()) % modulo; if (p + l > len) l = len - p; (void)XXH3_128bits_update(state, (const char*)data+p, l); p += l; } - BMK_checkResult128(XXH3_128bits_digest(state), Nresult); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); } /* byte by byte ingestion */ @@ -1180,7 +1168,7 @@ void BMK_testXXH128_withSecret(const void* data, size_t len, const void* secret, (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); for (pos=0; pos= 7 + secretSize); - BMK_testXXH3_withSecret(NULL, 0, secret, secretSize, 0x3559D64878C5C66CULL); /* empty string */ - BMK_testXXH3_withSecret(sanityBuffer, 1, secret, secretSize, 0x8A52451418B2DA4DULL); /* 1 - 3 */ - BMK_testXXH3_withSecret(sanityBuffer, 6, secret, secretSize, 0x82C90AB0519369ADULL); /* 4 - 8 */ - BMK_testXXH3_withSecret(sanityBuffer, 12, secret, secretSize, 0x14631E773B78EC57ULL); /* 9 - 16 */ - BMK_testXXH3_withSecret(sanityBuffer, 24, secret, secretSize, 0xCDD5542E4A9D9FE8ULL); /* 17 - 32 */ - BMK_testXXH3_withSecret(sanityBuffer, 48, secret, secretSize, 0x33ABD54D094B2534ULL); /* 33 - 64 */ - BMK_testXXH3_withSecret(sanityBuffer, 80, secret, secretSize, 0xE687BA1684965297ULL); /* 65 - 96 */ - BMK_testXXH3_withSecret(sanityBuffer, 195, secret, secretSize, 0xA057273F5EECFB20ULL); /* 129-240 */ + XSUM_testXXH3_withSecret(NULL, 0, secret, secretSize, 0x3559D64878C5C66CULL); /* empty string */ + XSUM_testXXH3_withSecret(sanityBuffer, 1, secret, secretSize, 0x8A52451418B2DA4DULL); /* 1 - 3 */ + XSUM_testXXH3_withSecret(sanityBuffer, 6, secret, secretSize, 0x82C90AB0519369ADULL); /* 4 - 8 */ + XSUM_testXXH3_withSecret(sanityBuffer, 12, secret, secretSize, 0x14631E773B78EC57ULL); /* 9 - 16 */ + XSUM_testXXH3_withSecret(sanityBuffer, 24, secret, secretSize, 0xCDD5542E4A9D9FE8ULL); /* 17 - 32 */ + XSUM_testXXH3_withSecret(sanityBuffer, 48, secret, secretSize, 0x33ABD54D094B2534ULL); /* 33 - 64 */ + XSUM_testXXH3_withSecret(sanityBuffer, 80, secret, secretSize, 0xE687BA1684965297ULL); /* 65 - 96 */ + XSUM_testXXH3_withSecret(sanityBuffer, 195, secret, secretSize, 0xA057273F5EECFB20ULL); /* 129-240 */ - BMK_testXXH3_withSecret(sanityBuffer, 403, secret, secretSize, 0x14546019124D43B8ULL); /* one block, last stripe is overlapping */ - BMK_testXXH3_withSecret(sanityBuffer, 512, secret, secretSize, 0x7564693DD526E28DULL); /* one block, finishing at stripe boundary */ - BMK_testXXH3_withSecret(sanityBuffer,2048, secret, secretSize, 0xD32E975821D6519FULL); /* >= 2 blocks, at least one scrambling */ - BMK_testXXH3_withSecret(sanityBuffer,2367, secret, secretSize, 0x293FA8E5173BB5E7ULL); /* >= 2 blocks, at least one scrambling, last stripe unaligned */ + XSUM_testXXH3_withSecret(sanityBuffer, 403, secret, secretSize, 0x14546019124D43B8ULL); /* one block, last stripe is overlapping */ + XSUM_testXXH3_withSecret(sanityBuffer, 512, secret, secretSize, 0x7564693DD526E28DULL); /* one block, finishing at stripe boundary */ + XSUM_testXXH3_withSecret(sanityBuffer,2048, secret, secretSize, 0xD32E975821D6519FULL); /* >= 2 blocks, at least one scrambling */ + XSUM_testXXH3_withSecret(sanityBuffer,2367, secret, secretSize, 0x293FA8E5173BB5E7ULL); /* >= 2 blocks, at least one scrambling, last stripe unaligned */ - BMK_testXXH3_withSecret(sanityBuffer,64*10*3, secret, secretSize, 0x751D2EC54BC6038BULL); /* exactly 3 full blocks, not a multiple of 256 */ + XSUM_testXXH3_withSecret(sanityBuffer,64*10*3, secret, secretSize, 0x751D2EC54BC6038BULL); /* exactly 3 full blocks, not a multiple of 256 */ } /* XXH128 */ { XXH128_hash_t const expected = { 0x6001C324468D497FULL, 0x99AA06D3014798D8ULL }; - BMK_testXXH128(NULL, 0, 0, expected); /* empty string */ + XSUM_testXXH128(NULL, 0, 0, expected); /* empty string */ } { XXH128_hash_t const expected = { 0x5444F7869C671AB0ULL, 0x92220AE55E14AB50ULL }; - BMK_testXXH128(NULL, 0, PRIME32, expected); + XSUM_testXXH128(NULL, 0, PRIME32, expected); } { XXH128_hash_t const expected = { 0xC44BDFF4074EECDBULL, 0xA6CD5E9392000F6AULL }; - BMK_testXXH128(sanityBuffer, 1, 0, expected); /* 1-3 */ + XSUM_testXXH128(sanityBuffer, 1, 0, expected); /* 1-3 */ } { XXH128_hash_t const expected = { 0xB53D5557E7F76F8DULL, 0x89B99554BA22467CULL }; - BMK_testXXH128(sanityBuffer, 1, PRIME32, expected); /* 1-3 */ + XSUM_testXXH128(sanityBuffer, 1, PRIME32, expected); /* 1-3 */ } { XXH128_hash_t const expected = { 0x3E7039BDDA43CFC6ULL, 0x082AFE0B8162D12AULL }; - BMK_testXXH128(sanityBuffer, 6, 0, expected); /* 4-8 */ + XSUM_testXXH128(sanityBuffer, 6, 0, expected); /* 4-8 */ } { XXH128_hash_t const expected = { 0x269D8F70BE98856EULL, 0x5A865B5389ABD2B1ULL }; - BMK_testXXH128(sanityBuffer, 6, PRIME32, expected); /* 4-8 */ + XSUM_testXXH128(sanityBuffer, 6, PRIME32, expected); /* 4-8 */ } { XXH128_hash_t const expected = { 0x061A192713F69AD9ULL, 0x6E3EFD8FC7802B18ULL }; - BMK_testXXH128(sanityBuffer, 12, 0, expected); /* 9-16 */ + XSUM_testXXH128(sanityBuffer, 12, 0, expected); /* 9-16 */ } { XXH128_hash_t const expected = { 0x9BE9F9A67F3C7DFBULL, 0xD7E09D518A3405D3ULL }; - BMK_testXXH128(sanityBuffer, 12, PRIME32, expected); /* 9-16 */ + XSUM_testXXH128(sanityBuffer, 12, PRIME32, expected); /* 9-16 */ } { XXH128_hash_t const expected = { 0x1E7044D28B1B901DULL, 0x0CE966E4678D3761ULL }; - BMK_testXXH128(sanityBuffer, 24, 0, expected); /* 17-32 */ + XSUM_testXXH128(sanityBuffer, 24, 0, expected); /* 17-32 */ } { XXH128_hash_t const expected = { 0xD7304C54EBAD40A9ULL, 0x3162026714A6A243ULL }; - BMK_testXXH128(sanityBuffer, 24, PRIME32, expected); /* 17-32 */ + XSUM_testXXH128(sanityBuffer, 24, PRIME32, expected); /* 17-32 */ } { XXH128_hash_t const expected = { 0xF942219AED80F67BULL, 0xA002AC4E5478227EULL }; - BMK_testXXH128(sanityBuffer, 48, 0, expected); /* 33-64 */ + XSUM_testXXH128(sanityBuffer, 48, 0, expected); /* 33-64 */ } { XXH128_hash_t const expected = { 0x7BA3C3E453A1934EULL, 0x163ADDE36C072295ULL }; - BMK_testXXH128(sanityBuffer, 48, PRIME32, expected); /* 33-64 */ + XSUM_testXXH128(sanityBuffer, 48, PRIME32, expected); /* 33-64 */ } { XXH128_hash_t const expected = { 0x5E8BAFB9F95FB803ULL, 0x4952F58181AB0042ULL }; - BMK_testXXH128(sanityBuffer, 81, 0, expected); /* 65-96 */ + XSUM_testXXH128(sanityBuffer, 81, 0, expected); /* 65-96 */ } { XXH128_hash_t const expected = { 0x703FBB3D7A5F755CULL, 0x2724EC7ADC750FB6ULL }; - BMK_testXXH128(sanityBuffer, 81, PRIME32, expected); /* 65-96 */ + XSUM_testXXH128(sanityBuffer, 81, PRIME32, expected); /* 65-96 */ } { XXH128_hash_t const expected = { 0xF1AEBD597CEC6B3AULL, 0x337E09641B948717ULL }; - BMK_testXXH128(sanityBuffer, 222, 0, expected); /* 129-240 */ + XSUM_testXXH128(sanityBuffer, 222, 0, expected); /* 129-240 */ } { XXH128_hash_t const expected = { 0xAE995BB8AF917A8DULL, 0x91820016621E97F1ULL }; - BMK_testXXH128(sanityBuffer, 222, PRIME32, expected); /* 129-240 */ + XSUM_testXXH128(sanityBuffer, 222, PRIME32, expected); /* 129-240 */ } { XXH128_hash_t const expected = { 0xCDEB804D65C6DEA4ULL, 0x1B6DE21E332DD73DULL }; - BMK_testXXH128(sanityBuffer, 403, 0, expected); /* one block, last stripe is overlapping */ + XSUM_testXXH128(sanityBuffer, 403, 0, expected); /* one block, last stripe is overlapping */ } { XXH128_hash_t const expected = { 0x6259F6ECFD6443FDULL, 0xBED311971E0BE8F2ULL }; - BMK_testXXH128(sanityBuffer, 403, PRIME64, expected); /* one block, last stripe is overlapping */ + XSUM_testXXH128(sanityBuffer, 403, PRIME64, expected); /* one block, last stripe is overlapping */ } { XXH128_hash_t const expected = { 0x617E49599013CB6BULL, 0x18D2D110DCC9BCA1ULL }; - BMK_testXXH128(sanityBuffer, 512, 0, expected); /* one block, finishing at stripe boundary */ + XSUM_testXXH128(sanityBuffer, 512, 0, expected); /* one block, finishing at stripe boundary */ } { XXH128_hash_t const expected = { 0x3CE457DE14C27708ULL, 0x925D06B8EC5B8040ULL }; - BMK_testXXH128(sanityBuffer, 512, PRIME64, expected); /* one block, finishing at stripe boundary */ + XSUM_testXXH128(sanityBuffer, 512, PRIME64, expected); /* one block, finishing at stripe boundary */ } { XXH128_hash_t const expected = { 0xDD59E2C3A5F038E0ULL, 0xF736557FD47073A5ULL }; - BMK_testXXH128(sanityBuffer,2048, 0, expected); /* two blocks, finishing at block boundary */ + XSUM_testXXH128(sanityBuffer,2048, 0, expected); /* two blocks, finishing at block boundary */ } { XXH128_hash_t const expected = { 0x230D43F30206260BULL, 0x7FB03F7E7186C3EAULL }; - BMK_testXXH128(sanityBuffer,2048, PRIME32, expected); /* two blocks, finishing at block boundary */ + XSUM_testXXH128(sanityBuffer,2048, PRIME32, expected); /* two blocks, finishing at block boundary */ } { XXH128_hash_t const expected = { 0x6E73A90539CF2948ULL, 0xCCB134FBFA7CE49DULL }; - BMK_testXXH128(sanityBuffer,2240, 0, expected); /* two blocks, ends at stripe boundary */ + XSUM_testXXH128(sanityBuffer,2240, 0, expected); /* two blocks, ends at stripe boundary */ } { XXH128_hash_t const expected = { 0xED385111126FBA6FULL, 0x50A1FE17B338995FULL }; - BMK_testXXH128(sanityBuffer,2240, PRIME32, expected); /* two blocks, ends at stripe boundary */ + XSUM_testXXH128(sanityBuffer,2240, PRIME32, expected); /* two blocks, ends at stripe boundary */ } { XXH128_hash_t const expected = { 0xCB37AEB9E5D361EDULL, 0xE89C0F6FF369B427ULL }; - BMK_testXXH128(sanityBuffer,2367, 0, expected); /* two blocks, last stripe is overlapping */ + XSUM_testXXH128(sanityBuffer,2367, 0, expected); /* two blocks, last stripe is overlapping */ } { XXH128_hash_t const expected = { 0x6F5360AE69C2F406ULL, 0xD23AAE4B76C31ECBULL }; - BMK_testXXH128(sanityBuffer,2367, PRIME32, expected); /* two blocks, last stripe is overlapping */ + XSUM_testXXH128(sanityBuffer,2367, PRIME32, expected); /* two blocks, last stripe is overlapping */ } /* XXH128 with custom Secret */ @@ -1378,38 +1366,38 @@ static void BMK_sanityCheck(void) assert(sizeof(sanityBuffer) >= 7 + secretSize); { XXH128_hash_t const expected = { 0x005923CCEECBE8AEULL, 0x5F70F4EA232F1D38ULL }; - BMK_testXXH128_withSecret(NULL, 0, secret, secretSize, expected); /* empty string */ + XSUM_testXXH128_withSecret(NULL, 0, secret, secretSize, expected); /* empty string */ } { XXH128_hash_t const expected = { 0x8A52451418B2DA4DULL, 0x3A66AF5A9819198EULL }; - BMK_testXXH128_withSecret(sanityBuffer, 1, secret, secretSize, expected); /* 1-3 */ + XSUM_testXXH128_withSecret(sanityBuffer, 1, secret, secretSize, expected); /* 1-3 */ } { XXH128_hash_t const expected = { 0x0B61C8ACA7D4778FULL, 0x376BD91B6432F36DULL }; - BMK_testXXH128_withSecret(sanityBuffer, 6, secret, secretSize, expected); /* 4-8 */ + XSUM_testXXH128_withSecret(sanityBuffer, 6, secret, secretSize, expected); /* 4-8 */ } { XXH128_hash_t const expected = { 0xAF82F6EBA263D7D8ULL, 0x90A3C2D839F57D0FULL }; - BMK_testXXH128_withSecret(sanityBuffer, 12, secret, secretSize, expected); /* 9-16 */ + XSUM_testXXH128_withSecret(sanityBuffer, 12, secret, secretSize, expected); /* 9-16 */ } } /* secret generator */ { verifSample_t const expected = { { 0xB8, 0x26, 0x83, 0x7E } }; - BMK_testSecretGenerator(NULL, 0, expected); + XSUM_testSecretGenerator(NULL, 0, expected); } { verifSample_t const expected = { { 0xA6, 0x16, 0x06, 0x7B } }; - BMK_testSecretGenerator(sanityBuffer, 1, expected); + XSUM_testSecretGenerator(sanityBuffer, 1, expected); } { verifSample_t const expected = { { 0xDA, 0x2A, 0x12, 0x11 } }; - BMK_testSecretGenerator(sanityBuffer, XXH3_SECRET_SIZE_MIN - 1, expected); + XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_SIZE_MIN - 1, expected); } { verifSample_t const expected = { { 0x7E, 0x48, 0x0C, 0xA7 } }; - BMK_testSecretGenerator(sanityBuffer, XXH3_SECRET_DEFAULT_SIZE + 500, expected); + XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_DEFAULT_SIZE + 500, expected); } - DISPLAYLEVEL(3, "\r%70s\r", ""); /* Clean display line */ - DISPLAYLEVEL(3, "Sanity check -- all tests ok\n"); + XSUM_logVerbose(3, "\r%70s\r", ""); /* Clean display line */ + XSUM_logVerbose(3, "Sanity check -- all tests ok\n"); } @@ -1440,20 +1428,20 @@ int XSUM_isDirectory(const char* infilename) } /* for support of --little-endian display mode */ -static void BMK_display_LittleEndian(const void* ptr, size_t length) +static void XSUM_display_LittleEndian(const void* ptr, size_t length) { const U8* const p = (const U8*)ptr; size_t idx; for (idx=length-1; idx <' ('> <') = '> <'\0'> */ -static ParseLineResult parseLine(ParsedLine* parsedLine, char* line, int rev) +static ParseLineResult XSUM_parseLine(ParsedLine* parsedLine, char* line, int rev) { char* const firstSpace = strchr(line, ' '); const char* hash_ptr; @@ -1895,7 +1883,7 @@ static ParseLineResult parseLine(ParsedLine* parsedLine, char* line, int rev) { case 8: { XXH32_canonical_t* xxh32c = &parsedLine->canonical.xxh32; - if (canonicalFromString(xxh32c->digest, sizeof(xxh32c->digest), hash_ptr, rev) + if (XSUM_canonicalFromString(xxh32c->digest, sizeof(xxh32c->digest), hash_ptr, rev) != CanonicalFromString_ok) { return ParseLine_invalidFormat; } @@ -1905,7 +1893,7 @@ static ParseLineResult parseLine(ParsedLine* parsedLine, char* line, int rev) case 16: { XXH64_canonical_t* xxh64c = &parsedLine->canonical.xxh64; - if (canonicalFromString(xxh64c->digest, sizeof(xxh64c->digest), hash_ptr, rev) + if (XSUM_canonicalFromString(xxh64c->digest, sizeof(xxh64c->digest), hash_ptr, rev) != CanonicalFromString_ok) { return ParseLine_invalidFormat; } @@ -1915,7 +1903,7 @@ static ParseLineResult parseLine(ParsedLine* parsedLine, char* line, int rev) case 32: { XXH128_canonical_t* xxh128c = &parsedLine->canonical.xxh128; - if (canonicalFromString(xxh128c->digest, sizeof(xxh128c->digest), hash_ptr, rev) + if (XSUM_canonicalFromString(xxh128c->digest, sizeof(xxh128c->digest), hash_ptr, rev) != CanonicalFromString_ok) { return ParseLine_invalidFormat; } @@ -1938,10 +1926,10 @@ static ParseLineResult parseLine(ParsedLine* parsedLine, char* line, int rev) /*! * Parse xxHash checksum file. */ -static void parseFile1(ParseFileArg* parseFileArg, int rev) +static void XSUM_parseFile1(ParseFileArg* XSUM_parseFileArg, int rev) { - const char* const inFileName = parseFileArg->inFileName; - ParseFileReport* const report = &parseFileArg->report; + const char* const inFileName = XSUM_parseFileArg->inFileName; + ParseFileReport* const report = &XSUM_parseFileArg->report; unsigned long lineNumber = 0; memset(report, 0, sizeof(*report)); @@ -1954,46 +1942,46 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) lineNumber++; if (lineNumber == 0) { /* This is unlikely happen, but md5sum.c has this error check. */ - DISPLAY("%s: Error: Too many checksum lines\n", inFileName); + XSUM_log("%s: Error: Too many checksum lines\n", inFileName); report->quit = 1; break; } - { GetLineResult const getLineResult = getLine(&parseFileArg->lineBuf, - &parseFileArg->lineMax, - parseFileArg->inFile); - if (getLineResult != GetLine_ok) { - if (getLineResult == GetLine_eof) break; + { GetLineResult const XSUM_getLineResult = XSUM_getLine(&XSUM_parseFileArg->lineBuf, + &XSUM_parseFileArg->lineMax, + XSUM_parseFileArg->inFile); + if (XSUM_getLineResult != GetLine_ok) { + if (XSUM_getLineResult == GetLine_eof) break; - switch (getLineResult) + switch (XSUM_getLineResult) { case GetLine_ok: case GetLine_eof: - /* These cases never happen. See above getLineResult related "if"s. + /* These cases never happen. See above XSUM_getLineResult related "if"s. They exist just for make gcc's -Wswitch-enum happy. */ assert(0); break; default: - DISPLAY("%s:%lu: Error: Unknown error.\n", inFileName, lineNumber); + XSUM_log("%s:%lu: Error: Unknown error.\n", inFileName, lineNumber); break; case GetLine_exceedMaxLineLength: - DISPLAY("%s:%lu: Error: Line too long.\n", inFileName, lineNumber); + XSUM_log("%s:%lu: Error: Line too long.\n", inFileName, lineNumber); break; case GetLine_outOfMemory: - DISPLAY("%s:%lu: Error: Out of memory.\n", inFileName, lineNumber); + XSUM_log("%s:%lu: Error: Out of memory.\n", inFileName, lineNumber); break; } report->quit = 1; break; } } - if (parseLine(&parsedLine, parseFileArg->lineBuf, rev) != ParseLine_ok) { + if (XSUM_parseLine(&parsedLine, XSUM_parseFileArg->lineBuf, rev) != ParseLine_ok) { report->nImproperlyFormattedLines++; - if (parseFileArg->warn) { - DISPLAY("%s:%lu: Error: Improperly formatted checksum line.\n", + if (XSUM_parseFileArg->warn) { + XSUM_log("%s:%lu: Error: Improperly formatted checksum line.\n", inFileName, lineNumber); } continue; @@ -2002,7 +1990,7 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) report->nProperlyFormattedLines++; do { - FILE* const fp = XXH_fopen(parsedLine.filename, "rb"); + FILE* const fp = XSUM_fopen(parsedLine.filename, "rb"); if (fp == NULL) { lineStatus = LineStatus_failedToOpen; break; @@ -2011,21 +1999,21 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) switch (parsedLine.xxhBits) { case 32: - { Multihash const xxh = XSUM_hashStream(fp, algo_xxh32, parseFileArg->blockBuf, parseFileArg->blockSize); + { Multihash const xxh = XSUM_hashStream(fp, algo_xxh32, XSUM_parseFileArg->blockBuf, XSUM_parseFileArg->blockSize); if (xxh.xxh32 == XXH32_hashFromCanonical(&parsedLine.canonical.xxh32)) { lineStatus = LineStatus_hashOk; } } break; case 64: - { Multihash const xxh = XSUM_hashStream(fp, algo_xxh64, parseFileArg->blockBuf, parseFileArg->blockSize); + { Multihash const xxh = XSUM_hashStream(fp, algo_xxh64, XSUM_parseFileArg->blockBuf, XSUM_parseFileArg->blockSize); if (xxh.xxh64 == XXH64_hashFromCanonical(&parsedLine.canonical.xxh64)) { lineStatus = LineStatus_hashOk; } } break; case 128: - { Multihash const xxh = XSUM_hashStream(fp, algo_xxh128, parseFileArg->blockBuf, parseFileArg->blockSize); + { Multihash const xxh = XSUM_hashStream(fp, algo_xxh128, XSUM_parseFileArg->blockBuf, XSUM_parseFileArg->blockSize); if (XXH128_isEqual(xxh.xxh128, XXH128_hashFromCanonical(&parsedLine.canonical.xxh128))) { lineStatus = LineStatus_hashOk; } } @@ -2040,14 +2028,14 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) switch (lineStatus) { default: - DISPLAY("%s: Error: Unknown error.\n", inFileName); + XSUM_log("%s: Error: Unknown error.\n", inFileName); report->quit = 1; break; case LineStatus_failedToOpen: report->nOpenOrReadFailures++; - if (!parseFileArg->statusOnly) { - DISPLAYRESULT("%s:%lu: Could not open or read '%s': %s.\n", + if (!XSUM_parseFileArg->statusOnly) { + XSUM_output("%s:%lu: Could not open or read '%s': %s.\n", inFileName, lineNumber, parsedLine.filename, strerror(errno)); } break; @@ -2057,13 +2045,13 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) { int b = 1; if (lineStatus == LineStatus_hashOk) { /* If --quiet is specified, don't display "OK" */ - if (parseFileArg->quiet) b = 0; + if (XSUM_parseFileArg->quiet) b = 0; } else { report->nMismatchedChecksums++; } - if (b && !parseFileArg->statusOnly) { - DISPLAYRESULT("%s: %s\n", parsedLine.filename + if (b && !XSUM_parseFileArg->statusOnly) { + XSUM_output("%s: %s\n", parsedLine.filename , lineStatus == LineStatus_hashOk ? "OK" : "FAILED"); } } break; @@ -2087,18 +2075,18 @@ static void parseFile1(ParseFileArg* parseFileArg, int rev) * - All hash values match with its content. * - (strict mode) All lines in checksum file are consistent and well formatted. */ -static int checkFile(const char* inFileName, - const Display_endianess displayEndianess, - U32 strictMode, - U32 statusOnly, - U32 warn, - U32 quiet) +static int XSUM_checkFile(const char* inFileName, + const Display_endianess displayEndianess, + U32 strictMode, + U32 statusOnly, + U32 warn, + U32 quiet) { int result = 0; FILE* inFile = NULL; - ParseFileArg parseFileArgBody; - ParseFileArg* const parseFileArg = &parseFileArgBody; - ParseFileReport* const report = &parseFileArg->report; + ParseFileArg XSUM_parseFileArgBody; + ParseFileArg* const XSUM_parseFileArg = &XSUM_parseFileArgBody; + ParseFileReport* const report = &XSUM_parseFileArg->report; /* note: stdinName is special constant pointer. It is not a string. */ if (inFileName == stdinName) { @@ -2109,54 +2097,54 @@ static int checkFile(const char* inFileName, inFileName = "stdin"; inFile = stdin; } else { - inFile = XXH_fopen( inFileName, "rt" ); + inFile = XSUM_fopen( inFileName, "rt" ); } if (inFile == NULL) { - DISPLAY("Error: Could not open '%s': %s\n", inFileName, strerror(errno)); + XSUM_log("Error: Could not open '%s': %s\n", inFileName, strerror(errno)); return 0; } - parseFileArg->inFileName = inFileName; - parseFileArg->inFile = inFile; - parseFileArg->lineMax = DEFAULT_LINE_LENGTH; - parseFileArg->lineBuf = (char*) malloc((size_t)parseFileArg->lineMax); - parseFileArg->blockSize = 64 * 1024; - parseFileArg->blockBuf = (char*) malloc(parseFileArg->blockSize); - parseFileArg->strictMode = strictMode; - parseFileArg->statusOnly = statusOnly; - parseFileArg->warn = warn; - parseFileArg->quiet = quiet; - - if ( (parseFileArg->lineBuf == NULL) - || (parseFileArg->blockBuf == NULL) ) { - DISPLAY("Error: : memory allocation failed \n"); + XSUM_parseFileArg->inFileName = inFileName; + XSUM_parseFileArg->inFile = inFile; + XSUM_parseFileArg->lineMax = DEFAULT_LINE_LENGTH; + XSUM_parseFileArg->lineBuf = (char*) malloc((size_t)XSUM_parseFileArg->lineMax); + XSUM_parseFileArg->blockSize = 64 * 1024; + XSUM_parseFileArg->blockBuf = (char*) malloc(XSUM_parseFileArg->blockSize); + XSUM_parseFileArg->strictMode = strictMode; + XSUM_parseFileArg->statusOnly = statusOnly; + XSUM_parseFileArg->warn = warn; + XSUM_parseFileArg->quiet = quiet; + + if ( (XSUM_parseFileArg->lineBuf == NULL) + || (XSUM_parseFileArg->blockBuf == NULL) ) { + XSUM_log("Error: : memory allocation failed \n"); exit(1); } - parseFile1(parseFileArg, displayEndianess != big_endian); + XSUM_parseFile1(XSUM_parseFileArg, displayEndianess != big_endian); - free(parseFileArg->blockBuf); - free(parseFileArg->lineBuf); + free(XSUM_parseFileArg->blockBuf); + free(XSUM_parseFileArg->lineBuf); if (inFile != stdin) fclose(inFile); /* Show error/warning messages. All messages are copied from md5sum.c */ if (report->nProperlyFormattedLines == 0) { - DISPLAY("%s: no properly formatted xxHash checksum lines found\n", inFileName); + XSUM_log("%s: no properly formatted xxHash checksum lines found\n", inFileName); } else if (!statusOnly) { if (report->nImproperlyFormattedLines) { - DISPLAYRESULT("%lu %s improperly formatted\n" + XSUM_output("%lu %s improperly formatted\n" , report->nImproperlyFormattedLines , report->nImproperlyFormattedLines == 1 ? "line is" : "lines are"); } if (report->nOpenOrReadFailures) { - DISPLAYRESULT("%lu listed %s could not be read\n" + XSUM_output("%lu listed %s could not be read\n" , report->nOpenOrReadFailures , report->nOpenOrReadFailures == 1 ? "file" : "files"); } if (report->nMismatchedChecksums) { - DISPLAYRESULT("%lu computed %s did NOT match\n" + XSUM_output("%lu computed %s did NOT match\n" , report->nMismatchedChecksums , report->nMismatchedChecksums == 1 ? "checksum" : "checksums"); } } @@ -2172,23 +2160,23 @@ static int checkFile(const char* inFileName, } -static int checkFiles(const char*const* fnList, int fnTotal, - const Display_endianess displayEndianess, - U32 strictMode, - U32 statusOnly, - U32 warn, - U32 quiet) +static int XSUM_checkFiles(const char*const* fnList, int fnTotal, + const Display_endianess displayEndianess, + U32 strictMode, + U32 statusOnly, + U32 warn, + U32 quiet) { int ok = 1; /* Special case for stdinName "-", * note: stdinName is not a string. It's special pointer. */ if (fnTotal==0) { - ok &= checkFile(stdinName, displayEndianess, strictMode, statusOnly, warn, quiet); + ok &= XSUM_checkFile(stdinName, displayEndianess, strictMode, statusOnly, warn, quiet); } else { int fnNb; for (fnNb=0; fnNb Date: Wed, 2 Sep 2020 10:26:26 -0400 Subject: [PATCH 010/187] Fix typo --- xxhsum.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xxhsum.c b/xxhsum.c index 4f790a1a..748a403d 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -2455,7 +2455,7 @@ static char** XSUM_convertArgv(int argc, const wchar_t* const utf16_argv[]) return utf8_argv; } /* Frees arguments returned by XSUM_convertArgv */ -static void freeargv(int argc, char** argv) +static void XSUM_freeArgv(int argc, char** argv) { int i; if (argv == NULL) { From 67d35c7d34e867f35dbb493972c4c1cf048f829b Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 9 Sep 2020 13:59:29 -0400 Subject: [PATCH 011/187] [xxhsum] Begin splitting xxhsum.c Starting with the macros Still need to implement the Windows feature tests. --- .gitignore | 1 + Makefile | 3 +- programs/xxhsum/xsum_arch.h | 153 +++++++++++++++++++++++++++++ programs/xxhsum/xsum_config.h | 169 ++++++++++++++++++++++++++++++++ xxhsum.c | 175 ++-------------------------------- 5 files changed, 333 insertions(+), 168 deletions(-) create mode 100644 programs/xxhsum/xsum_arch.h create mode 100644 programs/xxhsum/xsum_config.h diff --git a/.gitignore b/.gitignore index d0ce9aac..a2209534 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,7 @@ xxh32sum xxh64sum xxh128sum xxhsum +!programs/xxhsum xxhsum32 xxhsum_privateXXH xxhsum_inlinedXXH diff --git a/Makefile b/Makefile index da1ce068..42e23b82 100644 --- a/Makefile +++ b/Makefile @@ -98,7 +98,8 @@ dispatch: xxhash.o xxh_x86dispatch.o xxhsum.c $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) xxhash.o: xxhash.c xxhash.h -xxhsum.o: xxhsum.c xxhash.h xxh_x86dispatch.h +xxhsum.o: xxhsum.c programs/xxhsum/xsum_config.h programs/xxhsum/xsum_arch.h \ + xxhash.h xxh_x86dispatch.h xxh_x86dispatch.o: xxh_x86dispatch.c xxh_x86dispatch.h xxhash.h .PHONY: xxhsum_and_links diff --git a/programs/xxhsum/xsum_arch.h b/programs/xxhsum/xsum_arch.h new file mode 100644 index 00000000..1fb9a634 --- /dev/null +++ b/programs/xxhsum/xsum_arch.h @@ -0,0 +1,153 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +/* + * Checks for predefined macros by the compiler to try and get both the arch + * and the compiler version. + */ +#ifndef XSUM_ARCH_H +#define XSUM_ARCH_H + +#include "xsum_config.h" + +#define XSUM_LIB_VERSION XXH_VERSION_MAJOR.XXH_VERSION_MINOR.XXH_VERSION_RELEASE +#define XSUM_QUOTE(str) #str +#define XSUM_EXPAND_AND_QUOTE(str) XSUM_QUOTE(str) +#define XSUM_PROGRAM_VERSION XSUM_EXPAND_AND_QUOTE(XSUM_LIB_VERSION) + + +/* Show compiler versions in WELCOME_MESSAGE. XSUM_CC_VERSION_FMT will return the printf specifiers, + * and VERSION will contain the comma separated list of arguments to the XSUM_CC_VERSION_FMT string. */ +#if defined(__clang_version__) +/* Clang does its own thing. */ +# ifdef __apple_build_version__ +# define XSUM_CC_VERSION_FMT "Apple Clang %s" +# else +# define XSUM_CC_VERSION_FMT "Clang %s" +# endif +# define XSUM_CC_VERSION __clang_version__ +#elif defined(__VERSION__) +/* GCC and ICC */ +# define XSUM_CC_VERSION_FMT "%s" +# ifdef __INTEL_COMPILER /* icc adds its prefix */ +# define XSUM_CC_VERSION __VERSION__ +# else /* assume GCC */ +# define XSUM_CC_VERSION "GCC " __VERSION__ +# endif +#elif defined(_MSC_FULL_VER) && defined(_MSC_BUILD) +/* + * MSVC + * "For example, if the version number of the Visual C++ compiler is + * 15.00.20706.01, the _MSC_FULL_VER macro evaluates to 150020706." + * + * https://docs.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=vs-2017 + */ +# define XSUM_CC_VERSION_FMT "MSVC %02i.%02i.%05i.%02i" +# define XSUM_CC_VERSION _MSC_FULL_VER / 10000000 % 100, _MSC_FULL_VER / 100000 % 100, _MSC_FULL_VER % 100000, _MSC_BUILD +#elif defined(_MSC_VER) /* old MSVC */ +# define XSUM_CC_VERSION_FMT "MSVC %02i.%02i" +# define XSUM_CC_VERSION _MSC_VER / 100, _MSC_VER % 100 +#elif defined(__TINYC__) +/* tcc stores its version in the __TINYC__ macro. */ +# define XSUM_CC_VERSION_FMT "tcc %i.%i.%i" +# define XSUM_CC_VERSION __TINYC__ / 10000 % 100, __TINYC__ / 100 % 100, __TINYC__ % 100 +#else +# define XSUM_CC_VERSION_FMT "%s" +# define XSUM_CC_VERSION "unknown compiler" +#endif + +/* makes the next part easier */ +#if defined(__x86_64__) || defined(_M_AMD64) || defined(_M_X64) +# define XSUM_ARCH_X64 1 +# define XSUM_ARCH_X86 "x86_64" +#elif defined(__i386__) || defined(_M_IX86) || defined(_M_IX86_FP) +# define XSUM_ARCH_X86 "i386" +#endif + +/* Try to detect the architecture. */ +#if defined(ARCH_X86) +# if defined(XXHSUM_DISPATCH) +# define XSUM_ARCH XSUM_ARCH_X86 " autoVec" +# elif defined(__AVX512F__) +# define XSUM_ARCH XSUM_ARCH_X86 " + AVX512" +# elif defined(__AVX2__) +# define XSUM_ARCH XSUM_ARCH_X86 " + AVX2" +# elif defined(__AVX__) +# define XSUM_ARCH XSUM_ARCH_X86 " + AVX" +# elif defined(_M_X64) || defined(_M_AMD64) || defined(__x86_64__) \ + || defined(__SSE2__) || (defined(_M_IX86_FP) && _M_IX86_FP == 2) +# define XSUM_ARCH XSUM_ARCH_X86 " + SSE2" +# else +# define XSUM_ARCH XSUM_ARCH_X86 +# endif +#elif defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64) +# define XSUM_ARCH "aarch64 + NEON" +#elif defined(__arm__) || defined(__thumb__) || defined(__thumb2__) || defined(_M_ARM) +/* ARM has a lot of different features that can change xxHash significantly. */ +# if defined(__thumb2__) || (defined(__thumb__) && (__thumb__ == 2 || __ARM_ARCH >= 7)) +# define XSUM_ARCH_THUMB " Thumb-2" +# elif defined(__thumb__) +# define XSUM_ARCH_THUMB " Thumb-1" +# else +# define XSUM_ARCH_THUMB "" +# endif +/* ARMv7 has unaligned by default */ +# if defined(__ARM_FEATURE_UNALIGNED) || __ARM_ARCH >= 7 || defined(_M_ARMV7VE) +# define XSUM_ARCH_UNALIGNED " + unaligned" +# else +# define XSUM_ARCH_UNALIGNED "" +# endif +# if defined(__ARM_NEON) || defined(__ARM_NEON__) +# define XSUM_ARCH_NEON " + NEON" +# else +# define XSUM_ARCH_NEON "" +# endif +# define XSUM_ARCH "ARMv" XSUM_EXPAND_AND_QUOTE(__ARM_ARCH) XSUM_ARCH_THUMB XSUM_ARCH_NEON XSUM_ARCH_UNALIGNED +#elif defined(__powerpc64__) || defined(__ppc64__) || defined(__PPC64__) +# if defined(__GNUC__) && defined(__POWER9_VECTOR__) +# define XSUM_ARCH "ppc64 + POWER9 vector" +# elif defined(__GNUC__) && defined(__POWER8_VECTOR__) +# define XSUM_ARCH "ppc64 + POWER8 vector" +# else +# define XSUM_ARCH "ppc64" +# endif +#elif defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) +# define XSUM_ARCH "ppc" +#elif defined(__AVR) +# define XSUM_ARCH "AVR" +#elif defined(__mips64) +# define XSUM_ARCH "mips64" +#elif defined(__mips) +# define XSUM_ARCH "mips" +#elif defined(__s390x__) +# define XSUM_ARCH "s390x" +#elif defined(__s390__) +# define XSUM_ARCH "s390" +#else +# define XSUM_ARCH "unknown" +#endif + + +#endif /* XSUM_ARCH_H */ diff --git a/programs/xxhsum/xsum_config.h b/programs/xxhsum/xsum_config.h new file mode 100644 index 00000000..1f28f9cb --- /dev/null +++ b/programs/xxhsum/xsum_config.h @@ -0,0 +1,169 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +/* + * This contains various configuration parameters and feature detection for + * xxhsum. + * + * Similar to config.h in Autotools, this should be the first header included. + */ + +#ifndef XSUM_CONFIG_H +#define XSUM_CONFIG_H + + +/* ************************************ + * Compiler Options + **************************************/ +/* + * Disable Visual C's warnings when using the "insecure" CRT functions instead + * of the "secure" _s functions. + * + * These functions are not portable, and aren't necessary if you are using the + * original functions properly. + */ +#if defined(_MSC_VER) || defined(_WIN32) +# ifndef _CRT_SECURE_NO_WARNINGS +# define _CRT_SECURE_NO_WARNINGS +# endif +#endif + +/* Under Linux at least, pull in the *64 commands */ +#ifndef _LARGEFILE64_SOURCE +# define _LARGEFILE64_SOURCE +#endif + +/* + * So we can use __attribute__((__format__)) + */ +#ifdef __GNUC__ +# define XSUM_ATRRIBUTE(x) __attribute__(x) +#else +# define XSUM_ATTRIBUTE(x) +#endif + +#if !defined(_WIN32) && (defined(__unix__) || defined(__unix) || (defined(__APPLE__) && defined(__MACH__)) /* UNIX-like OS */ \ + || defined(__midipix__) || defined(__VMS)) +# if (defined(__APPLE__) && defined(__MACH__)) || defined(__SVR4) || defined(_AIX) || defined(__hpux) /* POSIX.1-2001 (SUSv3) conformant */ \ + || defined(__DragonFly__) || defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) /* BSD distros */ +# define XSUM_PLATFORM_POSIX_VERSION 200112L +# else +# if defined(__linux__) || defined(__linux) +# ifndef _POSIX_C_SOURCE +# define _POSIX_C_SOURCE 200112L /* use feature test macro */ +# endif +# endif +# include /* declares _POSIX_VERSION */ +# if defined(_POSIX_VERSION) /* POSIX compliant */ +# define XSUM_PLATFORM_POSIX_VERSION _POSIX_VERSION +# else +# define XSUM_PLATFORM_POSIX_VERSION 0 +# endif +# endif +#endif +#if !defined(XSUM_PLATFORM_POSIX_VERSION) +# define XSUM_PLATFORM_POSIX_VERSION -1 +#endif + +#if !defined(S_ISREG) +# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) +#endif + + +/* ************************************ + * Windows helpers + **************************************/ + +/* + * Whether to use the Windows UTF-16 APIs instead of the portable libc 8-bit + * ("ANSI") APIs. + * + * Windows is not UTF-8 clean by default, and the only way to access every file + * on the OS is to use UTF-16. + * + * Do note that xxhsum uses UTF-8 internally and only uses UTF-16 for command + * line arguments, console I/O, and opening files. + * + * Additionally, this guarantees all piped output is UTF-8. + */ +#if defined(XSUM_WIN32_USE_WCHAR) && !defined(_WIN32) +/* We use Windows APIs, only use this on Windows. */ +# undef XSUM_WIN32_USE_WCHAR +#endif + +#ifndef XSUM_WIN32_USE_WCHAR +# if defined(_WIN32) +# include +# if WCHAR_MAX == 0xFFFFU /* UTF-16 wchar_t */ +# define XSUM_WIN32_USE_WCHAR 1 +# else +# define XSUM_WIN32_USE_WCHAR 0 +# endif +# else +# define XSUM_WIN32_USE_WCHAR 0 +# endif +#endif + +#if !XSUM_WIN32_USE_WCHAR +/* + * It doesn't make sense to have one without the other. + * Due to XSUM_WIN32_USE_WCHAR being undef'd, this also handles + * non-WIN32 platforms. + */ +# undef XSUM_WIN32_USE_WMAIN +# define XSUM_WIN32_USE_WMAIN 0 +#else +/* + * Whether to use wmain() or main(). + * + * wmain() is preferred because we don't have to mess with internal hidden + * APIs. + * + * It always works on MSVC, but in MinGW, it only works on MinGW-w64 with the + * -municode flag. + * + * Therefore we have to use main() -- there is no better option. + */ +# ifndef XSUM_WIN32_USE_WMAIN +# if defined(_UNICODE) || defined(UNICODE) /* MinGW -municode */ \ + || defined(_MSC_VER) /* MSVC */ +# define XSUM_WIN32_USE_WMAIN 1 +# else +# define XSUM_WIN32_USE_WMAIN 0 +# endif +# endif +/* + * It is always good practice to define these to prevent accidental use of the + * ANSI APIs, even if the program primarily uses UTF-8. + */ +# ifndef _UNICODE +# define _UNICODE +# endif +# ifndef UNICODE +# define UNICODE +# endif +#endif /* XSUM_WIN32_USE_WCHAR */ + +#endif /* XSUM_CONFIG_H */ diff --git a/xxhsum.c b/xxhsum.c index 748a403d..0f906ae1 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -29,21 +29,8 @@ * Display convention is Big Endian, for both 32 and 64 bits algorithms */ - -/* ************************************ - * Compiler Options - **************************************/ -/* MS Visual */ -#if defined(_MSC_VER) || defined(_WIN32) -# ifndef _CRT_SECURE_NO_WARNINGS -# define _CRT_SECURE_NO_WARNINGS /* removes visual warnings */ -# endif -#endif - -/* Under Linux at least, pull in the *64 commands */ -#ifndef _LARGEFILE64_SOURCE -# define _LARGEFILE64_SOURCE -#endif +#include "programs/xxhsum/xsum_config.h" +#include "programs/xxhsum/xsum_arch.h" /* ************************************ * Includes @@ -65,35 +52,8 @@ # include "xxh_x86dispatch.h" #endif - -/* ************************************ - * OS-Specific Includes - **************************************/ -#if !defined(_WIN32) && (defined(__unix__) || defined(__unix) || (defined(__APPLE__) && defined(__MACH__)) /* UNIX-like OS */ \ - || defined(__midipix__) || defined(__VMS)) -# if (defined(__APPLE__) && defined(__MACH__)) || defined(__SVR4) || defined(_AIX) || defined(__hpux) /* POSIX.1-2001 (SUSv3) conformant */ \ - || defined(__DragonFly__) || defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) /* BSD distros */ -# define PLATFORM_POSIX_VERSION 200112L -# else -# if defined(__linux__) || defined(__linux) -# ifndef _POSIX_C_SOURCE -# define _POSIX_C_SOURCE 200112L /* use feature test macro */ -# endif -# endif -# include /* declares _POSIX_VERSION */ -# if defined(_POSIX_VERSION) /* POSIX compliant */ -# define PLATFORM_POSIX_VERSION _POSIX_VERSION -# else -# define PLATFORM_POSIX_VERSION 0 -# endif -# endif -#endif -#if !defined(PLATFORM_POSIX_VERSION) -# define PLATFORM_POSIX_VERSION -1 -#endif - -#if (defined(__linux__) && (PLATFORM_POSIX_VERSION >= 1)) \ - || (PLATFORM_POSIX_VERSION >= 200112L) \ +#if (defined(__linux__) && (XSUM_PLATFORM_POSIX_VERSION >= 1)) \ + || (XSUM_PLATFORM_POSIX_VERSION >= 200112L) \ || defined(__DJGPP__) \ || defined(__MSYS__) # include /* isatty */ @@ -127,10 +87,6 @@ static __inline int XSUM_isConsole(FILE* stdStream) { # define SET_BINARY_MODE(file) #endif -#if !defined(S_ISREG) -# define S_ISREG(x) (((x) & S_IFMT) == S_IFREG) -#endif - /* Unicode helpers for Windows to make UTF-8 act as it should. */ #ifdef _WIN32 /* @@ -318,132 +274,17 @@ static unsigned XSUM_isLittleEndian(void) } -/* ************************************* - * Constants - ***************************************/ -#define LIB_VERSION XXH_VERSION_MAJOR.XXH_VERSION_MINOR.XXH_VERSION_RELEASE -#define QUOTE(str) #str -#define EXPAND_AND_QUOTE(str) QUOTE(str) -#define PROGRAM_VERSION EXPAND_AND_QUOTE(LIB_VERSION) - -/* Show compiler versions in WELCOME_MESSAGE. CC_VERSION_FMT will return the printf specifiers, - * and VERSION will contain the comma separated list of arguments to the CC_VERSION_FMT string. */ -#if defined(__clang_version__) -/* Clang does its own thing. */ -# ifdef __apple_build_version__ -# define CC_VERSION_FMT "Apple Clang %s" -# else -# define CC_VERSION_FMT "Clang %s" -# endif -# define CC_VERSION __clang_version__ -#elif defined(__VERSION__) -/* GCC and ICC */ -# define CC_VERSION_FMT "%s" -# ifdef __INTEL_COMPILER /* icc adds its prefix */ -# define CC_VERSION __VERSION__ -# else /* assume GCC */ -# define CC_VERSION "GCC " __VERSION__ -# endif -#elif defined(_MSC_FULL_VER) && defined(_MSC_BUILD) -/* - * MSVC - * "For example, if the version number of the Visual C++ compiler is - * 15.00.20706.01, the _MSC_FULL_VER macro evaluates to 150020706." - * - * https://docs.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=vs-2017 - */ -# define CC_VERSION_FMT "MSVC %02i.%02i.%05i.%02i" -# define CC_VERSION _MSC_FULL_VER / 10000000 % 100, _MSC_FULL_VER / 100000 % 100, _MSC_FULL_VER % 100000, _MSC_BUILD -#elif defined(__TINYC__) -/* tcc stores its version in the __TINYC__ macro. */ -# define CC_VERSION_FMT "tcc %i.%i.%i" -# define CC_VERSION __TINYC__ / 10000 % 100, __TINYC__ / 100 % 100, __TINYC__ % 100 -#else -# define CC_VERSION_FMT "%s" -# define CC_VERSION "unknown compiler" -#endif - -/* makes the next part easier */ -#if defined(__x86_64__) || defined(_M_AMD64) || defined(_M_X64) -# define ARCH_X64 1 -# define ARCH_X86 "x86_64" -#elif defined(__i386__) || defined(_M_IX86) || defined(_M_IX86_FP) -# define ARCH_X86 "i386" -#endif - -/* Try to detect the architecture. */ -#if defined(ARCH_X86) -# if defined(XXHSUM_DISPATCH) -# define ARCH ARCH_X86 " autoVec" -# elif defined(__AVX512F__) -# define ARCH ARCH_X86 " + AVX512" -# elif defined(__AVX2__) -# define ARCH ARCH_X86 " + AVX2" -# elif defined(__AVX__) -# define ARCH ARCH_X86 " + AVX" -# elif defined(_M_X64) || defined(_M_AMD64) || defined(__x86_64__) \ - || defined(__SSE2__) || (defined(_M_IX86_FP) && _M_IX86_FP == 2) -# define ARCH ARCH_X86 " + SSE2" -# else -# define ARCH ARCH_X86 -# endif -#elif defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64) -# define ARCH "aarch64 + NEON" -#elif defined(__arm__) || defined(__thumb__) || defined(__thumb2__) || defined(_M_ARM) -/* ARM has a lot of different features that can change xxHash significantly. */ -# if defined(__thumb2__) || (defined(__thumb__) && (__thumb__ == 2 || __ARM_ARCH >= 7)) -# define ARCH_THUMB " Thumb-2" -# elif defined(__thumb__) -# define ARCH_THUMB " Thumb-1" -# else -# define ARCH_THUMB "" -# endif -/* ARMv7 has unaligned by default */ -# if defined(__ARM_FEATURE_UNALIGNED) || __ARM_ARCH >= 7 || defined(_M_ARMV7VE) -# define ARCH_UNALIGNED " + unaligned" -# else -# define ARCH_UNALIGNED "" -# endif -# if defined(__ARM_NEON) || defined(__ARM_NEON__) -# define ARCH_NEON " + NEON" -# else -# define ARCH_NEON "" -# endif -# define ARCH "ARMv" EXPAND_AND_QUOTE(__ARM_ARCH) ARCH_THUMB ARCH_NEON ARCH_UNALIGNED -#elif defined(__powerpc64__) || defined(__ppc64__) || defined(__PPC64__) -# if defined(__GNUC__) && defined(__POWER9_VECTOR__) -# define ARCH "ppc64 + POWER9 vector" -# elif defined(__GNUC__) && defined(__POWER8_VECTOR__) -# define ARCH "ppc64 + POWER8 vector" -# else -# define ARCH "ppc64" -# endif -#elif defined(__powerpc__) || defined(__ppc__) || defined(__PPC__) -# define ARCH "ppc" -#elif defined(__AVR) -# define ARCH "AVR" -#elif defined(__mips64) -# define ARCH "mips64" -#elif defined(__mips) -# define ARCH "mips" -#elif defined(__s390x__) -# define ARCH "s390x" -#elif defined(__s390__) -# define ARCH "s390" -#else -# define ARCH "unknown" -#endif static const int g_nbBits = (int)(sizeof(void*)*8); static const char g_lename[] = "little endian"; static const char g_bename[] = "big endian"; #define ENDIAN_NAME (XSUM_isLittleEndian() ? g_lename : g_bename) static const char author[] = "Yann Collet"; -#define WELCOME_MESSAGE(exename) "%s %s by %s \n", exename, PROGRAM_VERSION, author +#define WELCOME_MESSAGE(exename) "%s %s by %s \n", exename, XSUM_PROGRAM_VERSION, author #define FULL_WELCOME_MESSAGE(exename) "%s %s by %s \n" \ - "compiled as %i-bit %s %s with " CC_VERSION_FMT " \n", \ - exename, PROGRAM_VERSION, author, \ - g_nbBits, ARCH, ENDIAN_NAME, CC_VERSION + "compiled as %i-bit %s %s with " XSUM_CC_VERSION_FMT " \n", \ + exename, XSUM_PROGRAM_VERSION, author, \ + g_nbBits, XSUM_ARCH, ENDIAN_NAME, XSUM_CC_VERSION #define KB *( 1<<10) #define MB *( 1<<20) From 7ee691763bf55669217d8b74846e4992a558fe9c Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Tue, 15 Sep 2020 11:21:58 -0700 Subject: [PATCH 012/187] refactor xxh3 presentation in code comments hopefully clarifying #449 --- xxhash.h | 43 ++++++++++++++++++------------------------- 1 file changed, 18 insertions(+), 25 deletions(-) diff --git a/xxhash.h b/xxhash.h index 400d3a21..2ba034a5 100644 --- a/xxhash.h +++ b/xxhash.h @@ -406,7 +406,7 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src ************************************************************************/ /* ************************************************************************ - * XXH3 is a new hash algorithm featuring: + * XXH3 is a more recent hash algorithm featuring: * - Improved speed for both small and large inputs * - True 64-bit and 128-bit outputs * - SIMD acceleration @@ -416,38 +416,31 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src * * https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html * - * In general, expect XXH3 to run about ~2x faster on large inputs and >3x - * faster on small ones compared to XXH64, though exact differences depend on - * the platform. + * Compared to XXH64, expect XXH3 to run approximately + * ~2x faster on large inputs and >3x faster on small ones, + * exact differences vary depending on platform. * - * The algorithm is portable: Like XXH32 and XXH64, it generates the same hash - * on all platforms. - * - * It benefits greatly from SIMD and 64-bit arithmetic, but does not require it. - * - * Almost all 32-bit and 64-bit targets that can run XXH32 smoothly can run - * XXH3 at competitive speeds, even if XXH64 runs slowly. Further details are - * explained in the implementation. + * XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, + * but does not require it. + * Any 32-bit and 64-bit targets that can run XXH32 smoothly + * can run XXH3 at competitive speeds, even without vector support. + * Further details are explained in the implementation. * * Optimized implementations are provided for AVX512, AVX2, SSE2, NEON, POWER8, - * ZVector and scalar targets. This can be controlled with the XXH_VECTOR macro. + * ZVector and scalar targets. This can be controlled via the XXH_VECTOR macro. + * + * XXH3 implementation is portable: + * it has a generic C90 formulation that can be compiled on any platform, + * all implementations generage exactly the same hash value on all platforms. + * Starting from v0.8.0, it's also labelled "stable", meaning that + * any future version will also generate the same hash value. * * XXH3 offers 2 variants, _64bits and _128bits. - * When only 64 bits are needed, prefer calling the _64bits variant, as it - * reduces the amount of mixing, resulting in faster speed on small inputs. * + * When only 64 bits are needed, prefer invoking the _64bits variant, as it + * reduces the amount of mixing, resulting in faster speed on small inputs. * It's also generally simpler to manipulate a scalar return type than a struct. * - * The 128-bit version adds additional strength, but it is slightly slower. - * - * Return values of XXH3 and XXH128 are officially finalized starting - * with v0.8.0 and will no longer change in future versions. - * Avoid storing values from before that release in long-term storage. - * - * Results produced by v0.7.x are not comparable with results from v0.7.y. - * However, the API is completely stable, and it can safely be used for - * ephemeral data (local sessions). - * * The API supports one-shot hashing, streaming mode, and custom secrets. */ From 1c5402c4f5ca1a1a97b5df794cf5aafe8afc883c Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 21:30:07 -0400 Subject: [PATCH 013/187] xxhsum: Split most OS/wrapper functions Most, if not all platform-specific code is in its own file now. Also, minor, fix Windows ANSI API usage on XSUM_isDirectory() Didn't test Windows yet, waiting for CI. A new change is that XSUM_fprintf_utf8 (now XSUM_vfprintf) separates the printing and the formatting routines, instead using a vasprintf implementation. Functionally the same, but much cleaner now. --- Makefile | 15 +- cmake_unofficial/CMakeLists.txt | 5 +- programs/xxhsum/xsum_config.h | 2 +- programs/xxhsum/xsum_os_specific.c | 467 +++++++++++++++++++++++++++++ programs/xxhsum/xsum_os_specific.h | 84 ++++++ programs/xxhsum/xsum_output.c | 67 +++++ programs/xxhsum/xsum_output.h | 62 ++++ xxhsum.c | 397 +----------------------- 8 files changed, 708 insertions(+), 391 deletions(-) create mode 100644 programs/xxhsum/xsum_os_specific.c create mode 100644 programs/xxhsum/xsum_os_specific.h create mode 100644 programs/xxhsum/xsum_output.c create mode 100644 programs/xxhsum/xsum_output.h diff --git a/Makefile b/Makefile index 42e23b82..63349695 100644 --- a/Makefile +++ b/Makefile @@ -70,7 +70,13 @@ else endif LIBXXH = libxxhash.$(SHARED_EXT_VER) - +XXHSUM_OBJS = xxhsum.o \ + programs/xxhsum/xsum_os_specific.o \ + programs/xxhsum/xsum_output.o +XXHSUM_HEADERS = programs/xxhsum/xsum_config.h \ + programs/xxhsum/xsum_arch.h \ + programs/xxhsum/xsum_os_specific.h \ + programs/xxhsum/xsum_output.h ## generate CLI and libraries in release mode (default for `make`) .PHONY: default @@ -85,7 +91,7 @@ ifeq ($(DISPATCH),1) xxhsum: CPPFLAGS += -DXXHSUM_DISPATCH=1 xxhsum: xxh_x86dispatch.o endif -xxhsum: xxhash.o xxhsum.o +xxhsum: xxhash.o $(XXHSUM_OBJS) $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) xxhsum32: CFLAGS += -m32 ## generate CLI in 32-bits mode @@ -98,7 +104,7 @@ dispatch: xxhash.o xxh_x86dispatch.o xxhsum.c $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) xxhash.o: xxhash.c xxhash.h -xxhsum.o: xxhsum.c programs/xxhsum/xsum_config.h programs/xxhsum/xsum_arch.h \ +xxhsum.o: xxhsum.c $(XXHSUM_HEADERS) \ xxhash.h xxh_x86dispatch.h xxh_x86dispatch.o: xxh_x86dispatch.c xxh_x86dispatch.h xxhash.h @@ -158,9 +164,10 @@ help: ## list documented targets .PHONY: clean clean: ## remove all build artifacts $(Q)$(RM) -r *.dSYM # Mac OS-X specific - $(Q)$(RM) core *.o *.$(SHARED_EXT) *.$(SHARED_EXT).* *.a libxxhash.pc + $(Q)$(RM) core *.o *.obj *.$(SHARED_EXT) *.$(SHARED_EXT).* *.a libxxhash.pc $(Q)$(RM) xxhsum$(EXT) xxhsum32$(EXT) xxhsum_inlinedXXH$(EXT) dispatch$(EXT) $(Q)$(RM) xxh32sum$(EXT) xxh64sum$(EXT) xxh128sum$(EXT) + $(Q)$(RM) programs/xxhsum/*.o programs/xxhsum/*.obj @echo cleaning completed diff --git a/cmake_unofficial/CMakeLists.txt b/cmake_unofficial/CMakeLists.txt index d9a8636f..3a5086c8 100644 --- a/cmake_unofficial/CMakeLists.txt +++ b/cmake_unofficial/CMakeLists.txt @@ -83,7 +83,10 @@ set_target_properties(xxhash PROPERTIES if(XXHASH_BUILD_XXHSUM) # xxhsum - add_executable(xxhsum "${XXHASH_DIR}/xxhsum.c") + add_executable(xxhsum "${XXHASH_DIR}/xxhsum.c" + "${XXHASH_DIR}/programs/xxhsum/xsum_os_specific.c" + "${XXHASH_DIR}/programs/xxhsum/xsum_output.c" + ) add_executable(${PROJECT_NAME}::xxhsum ALIAS xxhsum) target_link_libraries(xxhsum PRIVATE xxhash) diff --git a/programs/xxhsum/xsum_config.h b/programs/xxhsum/xsum_config.h index 1f28f9cb..f49bf394 100644 --- a/programs/xxhsum/xsum_config.h +++ b/programs/xxhsum/xsum_config.h @@ -59,7 +59,7 @@ * So we can use __attribute__((__format__)) */ #ifdef __GNUC__ -# define XSUM_ATRRIBUTE(x) __attribute__(x) +# define XSUM_ATTRIBUTE(x) __attribute__(x) #else # define XSUM_ATTRIBUTE(x) #endif diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c new file mode 100644 index 00000000..45a5896f --- /dev/null +++ b/programs/xxhsum/xsum_os_specific.c @@ -0,0 +1,467 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#include "xsum_config.h" +#include "xsum_os_specific.h" +#include +#include +#include +#include /* struct stat / __wstat64 */ +#include /* stat() / _stat64() */ + +/* + * This file contains all of the ugly boilerplate to make xxhsum work across + * platforms. + */ +#if defined(_MSC_VER) + typedef struct __stat64 stat_t; + typedef int mode_t; +#else + typedef struct stat stat_t; +#endif + +#if (defined(__linux__) && (XSUM_PLATFORM_POSIX_VERSION >= 1)) \ + || (XSUM_PLATFORM_POSIX_VERSION >= 200112L) \ + || defined(__DJGPP__) \ + || defined(__MSYS__) +# include /* isatty */ +# define XSUM_IS_CONSOLE(stdStream) isatty(fileno(stdStream)) +#elif defined(MSDOS) || defined(OS2) +# include /* _isatty */ +# define XSUM_IS_CONSOLE(stdStream) _isatty(_fileno(stdStream)) +#elif defined(WIN32) || defined(_WIN32) +# include /* _isatty */ +# include /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */ +# include /* FILE */ +static __inline int XSUM_IS_CONSOLE(FILE* stdStream) +{ + DWORD dummy; + return _isatty(_fileno(stdStream)) && GetConsoleMode((HANDLE)_get_osfhandle(_fileno(stdStream)), &dummy); +} +#else +# define XSUM_IS_CONSOLE(stdStream) 0 +#endif + +#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) +# include /* _O_BINARY */ +# include /* _setmode, _fileno, _get_osfhandle */ +# if !defined(__DJGPP__) +# include /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */ +# include /* FSCTL_SET_SPARSE */ +# define XSUM_SET_BINARY_MODE(file) { int const unused=_setmode(_fileno(file), _O_BINARY); (void)unused; } +# else +# define XSUM_SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY) +# endif +#else +# define XSUM_SET_BINARY_MODE(file) ((void)file) +#endif + +int XSUM_isConsole(FILE* stream) +{ + return XSUM_IS_CONSOLE(stream); +} + +void XSUM_setBinaryMode(FILE* stream) +{ + XSUM_SET_BINARY_MODE(stream); +} + +#if !XSUM_WIN32_USE_WCHAR + +FILE* XSUM_fopen(const char* filename, const char* mode) +{ + return fopen(filename, mode); +} +XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) +int XSUM_vfprintf(FILE* stream, const char* format, va_list ap) +{ + return vfprintf(stream, format, ap); +} + +int XSUM_isDirectory(const char* infilename) +{ + stat_t statbuf; +#if defined(_MSC_VER) + int const r = _stat64(infilename, &statbuf); + if (!r && (statbuf.st_mode & _S_IFDIR)) return 1; +#else + int const r = stat(infilename, &statbuf); + if (!r && S_ISDIR(statbuf.st_mode)) return 1; +#endif + return 0; +} + +#ifndef XSUM_NO_MAIN +int main(int argc, char* argv[]) +{ + return XSUM_main(argc, argv); +} +#endif + +/* Unicode helpers for Windows to make UTF-8 act as it should. */ +#else +# include +# include + +/***************************************************************************** + * Unicode conversion tools + *****************************************************************************/ + +/* + * Converts a UTF-8 string to UTF-16. Acts like strdup. The string must be freed afterwards. + * This version allows keeping the output length. + */ +static wchar_t* XSUM_widenString(const char* str, int* lenOut) +{ + int const len = MultiByteToWideChar(CP_UTF8, 0, str, -1, NULL, 0); + if (lenOut != NULL) *lenOut = len; + if (len == 0) return NULL; + { wchar_t* buf = (wchar_t*)malloc((size_t)len * sizeof(wchar_t)); + if (buf != NULL) { + if (MultiByteToWideChar(CP_UTF8, 0, str, -1, buf, len) == 0) { + free(buf); + return NULL; + } } + return buf; + } +} + +/* + * Converts a UTF-16 string to UTF-8. Acts like strdup. The string must be freed afterwards. + * This version allows keeping the output length. + */ +static char* XSUM_narrowString(const wchar_t *str, int *lenOut) +{ + int len = WideCharToMultiByte(CP_UTF8, 0, str, -1, NULL, 0, NULL, NULL); + if (lenOut != NULL) *lenOut = len; + if (len == 0) return NULL; + { char* const buf = (char*)malloc((size_t)len * sizeof(char)); + if (buf != NULL) { + if (WideCharToMultiByte(CP_UTF8, 0, str, -1, buf, len, NULL, NULL) == 0) { + free(buf); + return NULL; + } } + return buf; + } +} + + + +/***************************************************************************** + * File helpers + *****************************************************************************/ +/* + * fopen wrapper that supports UTF-8 + * + * fopen will only accept ANSI filenames, which means that we can't open Unicode filenames. + * + * In order to open a Unicode filename, we need to convert filenames to UTF-16 and use _wfopen. + */ +FILE* XSUM_fopen(const char* filename, const char* mode) +{ + FILE* f = NULL; + wchar_t* const wide_filename = XSUM_widenString(filename, NULL); + if (wide_filename != NULL) { + wchar_t* const wide_mode = XSUM_widenString(mode, NULL); + if (wide_mode != NULL) { + f = _wfopen(wide_filename, wide_mode); + free(wide_mode); + } + free(wide_filename); + } + return f; +} + +/* + * Determines whether the file at path is a directory. + * + * Accepts UTF-8 filenames, unlike _stat64. + */ +int XSUM_isDirectory(const char* filename) +{ + struct __stat64 statbuf; + int result = 0; + wchar_t* const wide_filename = XSUM_widenString(filename, NULL); + if (wide_filename != NULL) { + if (_wstat64(wide_filename, &statbuf) == 0 /* stat fail is ok */ + && (statbuf.st_mode & _S_IFDIR)) { + result = 1; + } + free(wide_filename); + } + return result; +} + +/* + * In case it isn't available, this is what MSVC 2019 defines in stdarg.h. + */ +#if defined(_MSC_VER) && !defined(__clang__) && !defined(va_copy) +# define XSUM_va_copy(destination, source) ((destination) = (source)) +#else +# define XSUM_va_copy(destination, source) va_copy(destination, source) +#endif + +/* + * vasprintf for Windows. + */ +static int XSUM_vasprintf(char** strp, const char* format, va_list ap) +{ + int ret; + int size; + va_list copy; + /* + * To be safe, make a va_copy. + * + * Note that Microsoft doesn't use va_copy in its sample code: + * https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/vsprintf-vsprintf-l-vswprintf-vswprintf-l-vswprintf-l?view=vs-2019 + */ + XSUM_va_copy(copy, ap); + /* Calculate how many characters we need */ + size = _vscprintf(format, ap); + va_end(copy); + + if (size < 0) { + *strp = NULL; + return size; + } else { + *strp = (char*) malloc((size_t)size + 1); + if (*strp == NULL) { + return -1; + } + /* vsprintf into the new buffer */ + ret = vsprintf(*strp, format, ap); + if (ret < 0) { + free(*strp); + *strp = NULL; + } + return ret; + } +} + +/* + * fprintf wrapper that supports UTF-8. + * + * fprintf doesn't properly handle Unicode on Windows. + * + * Additionally, it is codepage sensitive on console and may crash the program. + * + * Instead, we use vsnprintf, and either print with fwrite or convert to UTF-16 + * for console output and use the codepage-independent WriteConsoleW. + * + * Credit to t-mat: https://github.com/t-mat/xxHash/commit/5691423 + */ +XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) +int XSUM_vfprintf(FILE *stream, const char *format, va_list ap) +{ + int result; + char* u8_str = NULL; + + /* + * Generate the UTF-8 output string with vasprintf. + */ + result = XSUM_vasprintf(&u8_str, format, ap); + + if (result >= 0) { + const size_t nchar = (size_t)result + 1; + + /* + * Check if we are outputting to a console. Don't use XSUM_isConsole + * directly -- we don't need to call _get_osfhandle twice. + */ + int fileNb = _fileno(stream); + intptr_t handle_raw = _get_osfhandle(fileNb); + HANDLE handle = (HANDLE)handle_raw; + DWORD dwTemp; + + if (handle_raw < 0) { + result = -1; + } else if (_isatty(fileNb) && GetConsoleMode(handle, &dwTemp)) { + /* + * Convert to UTF-16 and output with WriteConsoleW. + * + * This is codepage independent and works on Windows XP's default + * msvcrt.dll. + */ + int len; + wchar_t* const u16_buf = XSUM_widenString(u8_str, &len); + if (u16_buf == NULL) { + result = -1; + } else { + if (WriteConsoleW(handle, u16_buf, (DWORD)len - 1, &dwTemp, NULL)) { + result = (int)dwTemp; + } else { + result = -1; + } + free(u16_buf); + } + } else { + /* fwrite the UTF-8 string if we are printing to a file */ + result = (int)fwrite(u8_str, 1, nchar - 1, stream); + if (result == 0) { + result = -1; + } + } + free(u8_str); + } + return result; +} + +#ifndef XSUM_NO_MAIN +/***************************************************************************** + * Command Line argument parsing + *****************************************************************************/ + +/* Converts a UTF-16 argv to UTF-8. */ +static char** XSUM_convertArgv(int argc, wchar_t* utf16_argv[]) +{ + char** const utf8_argv = (char**)malloc((size_t)(argc + 1) * sizeof(char*)); + if (utf8_argv != NULL) { + int i; + for (i = 0; i < argc; i++) { + utf8_argv[i] = XSUM_narrowString(utf16_argv[i], NULL); + if (utf8_argv[i] == NULL) { + /* Out of memory, whoops. */ + while (i-- > 0) { + free(utf8_argv[i]); + } + free(utf8_argv); + return NULL; + } + } + utf8_argv[argc] = NULL; + } + return utf8_argv; +} +/* Frees arguments returned by XSUM_convertArgv */ +static void XSUM_freeArgv(int argc, char** argv) +{ + int i; + if (argv == NULL) { + return; + } + for (i = 0; i < argc; i++) { + free(argv[i]); + } + free(argv); +} + +static int XSUM_wmain(int argc, wchar_t* utf16_argv[]) +{ + /* Convert the UTF-16 arguments to UTF-8. */ + char** utf8_argv = XSUM_convertArgv(argc, utf16_argv); + + if (utf8_argv == NULL) { + /* An unfortunate but incredibly unlikely error. */ + fprintf(stderr, "xxhsum: error converting command line arguments!\n"); + abort(); + } else { + int ret; + + /* + * MinGW's terminal uses full block buffering for stderr. + * + * This is nonstandard behavior and causes text to not display until + * the buffer fills. + * + * `setvbuf()` can easily correct this to make text display instantly. + */ + setvbuf(stderr, NULL, _IONBF, 0); + + /* Call our real main function */ + ret = XSUM_main(argc, utf8_argv); + + /* Cleanup */ + XSUM_freeArgv(argc, utf8_argv); + return ret; + } +} + +#if XSUM_WIN32_USE_WMAIN + +/* + * The preferred method of obtaining the real UTF-16 arguments. Always works + * on MSVC, sometimes works on MinGW-w64 depending on the compiler flags. + */ +#ifdef __cplusplus +extern "C" +#endif +int __cdecl wmain(int argc, wchar_t* utf16_argv[]) +{ + return XSUM_wmain(argc, utf16_argv); +} +#else /* !XSUM_WIN32_USE_WMAIN */ + +/* + * Wrap `XSUM_wmain()` using `main()` and `__wgetmainargs()` on MinGW without + * Unicode support. + * + * `__wgetmainargs()` is used in the CRT startup to retrieve the arguments for + * `wmain()`, so we use it on MinGW to emulate `wmain()`. + * + * It is an internal function and not declared in any public headers, so we + * have to declare it manually. + * + * An alternative that doesn't mess with internal APIs is `GetCommandLineW()` + * with `CommandLineToArgvW()`, but the former doesn't expand wildcards and the + * latter requires linking to Shell32.dll and its numerous dependencies. + * + * This method keeps our dependencies to kernel32.dll and the CRT. + * + * https://docs.microsoft.com/en-us/cpp/c-runtime-library/getmainargs-wgetmainargs?view=vs-2019 + */ +typedef struct { + int newmode; +} _startupinfo; + +#ifdef __cplusplus +extern "C" +#endif +int __cdecl __wgetmainargs( + int* Argc, + wchar_t*** Argv, + wchar_t*** Env, + int DoWildCard, + _startupinfo* StartInfo +); + +int main(int ansi_argc, char* ansi_argv[]) +{ + int utf16_argc; + wchar_t** utf16_argv; + wchar_t** utf16_envp; /* Unused but required */ + _startupinfo startinfo = {0}; /* 0 == don't change new mode */ + + /* Get wmain's UTF-16 arguments. Make sure we expand wildcards. */ + if (__wgetmainargs(&utf16_argc, &utf16_argv, &utf16_envp, 1, &startinfo) < 0) + /* In the very unlikely case of an error, use the ANSI arguments. */ + return XSUM_main(ansi_argc, ansi_argv); + + /* Call XSUM_wmain with our UTF-16 arguments */ + return XSUM_wmain(utf16_argc, utf16_argv); +} + +#endif /* !XSUM_WIN32_USE_WMAIN */ +#endif /* !XSUM_NO_MAIN */ +#endif /* XSUM_WIN32_USE_WCHAR */ diff --git a/programs/xxhsum/xsum_os_specific.h b/programs/xxhsum/xsum_os_specific.h new file mode 100644 index 00000000..695e6463 --- /dev/null +++ b/programs/xxhsum/xsum_os_specific.h @@ -0,0 +1,84 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#ifndef XSUM_OS_SPECIFIC_H +#define XSUM_OS_SPECIFIC_H + +#include "xsum_config.h" +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * Declared here to be implemented in user code. + * + * Functions like main(), but is passed UTF-8 arguments even on Windows. + */ +int XSUM_main(int argc, char* argv[]); + +/* + * Returns whether stream is a console. + * + * Functionally equivalent to isatty(fileno(stream)). + */ +int XSUM_isConsole(FILE* stream); + +/* + * Sets stream to pure binary mode (a.k.a. no CRLF conversions). + */ +void XSUM_setBinaryMode(FILE* stream); + +/* + * Returns whether the file at filename is a directory. + */ +int XSUM_isDirectory(const char* filename); + +/* + * UTF-8 stdio wrappers primarily for Windows + */ + +/* + * fopen() wrapper. Accepts UTF-8 filenames on Windows. + * + * Specifically, on Windows, the arguments will be converted to UTF-16 + * and passed to _wfopen(). + */ +FILE* XSUM_fopen(const char* filename, const char* mode); + +/* + * vfprintf() wrapper which prints UTF-8 strings to Windows consoles + * if applicable. + */ +XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) +int XSUM_vfprintf(FILE* stream, const char* format, va_list ap); + +#ifdef __cplusplus +} +#endif + +#endif /* XSUM_OS_SPECIFIC_H */ diff --git a/programs/xxhsum/xsum_output.c b/programs/xxhsum/xsum_output.c new file mode 100644 index 00000000..83e6e7bd --- /dev/null +++ b/programs/xxhsum/xsum_output.c @@ -0,0 +1,67 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#include "xsum_output.h" +#include "xsum_os_specific.h" +#include + +int XSUM_logLevel = 2; + +XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) +int XSUM_log(const char* format, ...) +{ + int ret; + va_list ap; + va_start(ap, format); + ret = XSUM_vfprintf(stderr, format, ap); + va_end(ap); + return ret; +} + + +XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) +int XSUM_output(const char* format, ...) +{ + int ret; + va_list ap; + va_start(ap, format); + ret = XSUM_vfprintf(stdout, format, ap); + va_end(ap); + return ret; +} + +XSUM_ATTRIBUTE((__format__(__printf__, 2, 3))) +int XSUM_logVerbose(int minLevel, const char* format, ...) +{ + if (XSUM_logLevel >= minLevel) { + int ret; + va_list ap; + va_start(ap, format); + ret = XSUM_vfprintf(stderr, format, ap); + va_end(ap); + return ret; + } + return 0; +} diff --git a/programs/xxhsum/xsum_output.h b/programs/xxhsum/xsum_output.h new file mode 100644 index 00000000..80fec2f0 --- /dev/null +++ b/programs/xxhsum/xsum_output.h @@ -0,0 +1,62 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#ifndef XSUM_OUTPUT_H +#define XSUM_OUTPUT_H + +#include "xsum_config.h" + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * How verbose the output is. + */ +extern int XSUM_logLevel; + +/* + * Same as fprintf(stderr, format, ...) + */ +XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) +int XSUM_log(const char *format, ...); + +/* + * Like XSUM_log, but only outputs if XSUM_logLevel >= minLevel. + */ +XSUM_ATTRIBUTE((__format__(__printf__, 2, 3))) +int XSUM_logVerbose(int minLevel, const char *format, ...); + +/* + * Same as printf(format, ...) + */ +XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) +int XSUM_output(const char *format, ...); + +#ifdef __cplusplus +} +#endif + +#endif /* XSUM_OUTPUT_H */ diff --git a/xxhsum.c b/xxhsum.c index 0f906ae1..64bc6c79 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -29,8 +29,11 @@ * Display convention is Big Endian, for both 32 and 64 bits algorithms */ +/* Transitional headers */ #include "programs/xxhsum/xsum_config.h" #include "programs/xxhsum/xsum_arch.h" +#include "programs/xxhsum/xsum_os_specific.h" +#include "programs/xxhsum/xsum_output.h" /* ************************************ * Includes @@ -52,201 +55,6 @@ # include "xxh_x86dispatch.h" #endif -#if (defined(__linux__) && (XSUM_PLATFORM_POSIX_VERSION >= 1)) \ - || (XSUM_PLATFORM_POSIX_VERSION >= 200112L) \ - || defined(__DJGPP__) \ - || defined(__MSYS__) -# include /* isatty */ -# define XSUM_isConsole(stdStream) isatty(fileno(stdStream)) -#elif defined(MSDOS) || defined(OS2) -# include /* _isatty */ -# define XSUM_isConsole(stdStream) _isatty(_fileno(stdStream)) -#elif defined(WIN32) || defined(_WIN32) -# include /* _isatty */ -# include /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */ -# include /* FILE */ -static __inline int XSUM_isConsole(FILE* stdStream) { - DWORD dummy; - return _isatty(_fileno(stdStream)) && GetConsoleMode((HANDLE)_get_osfhandle(_fileno(stdStream)), &dummy); -} -#else -# define XSUM_isConsole(stdStream) 0 -#endif - -#if defined(MSDOS) || defined(OS2) || defined(WIN32) || defined(_WIN32) -# include /* _O_BINARY */ -# include /* _setmode, _fileno, _get_osfhandle */ -# if !defined(__DJGPP__) -# include /* DeviceIoControl, HANDLE, FSCTL_SET_SPARSE */ -# include /* FSCTL_SET_SPARSE */ -# define SET_BINARY_MODE(file) { int const unused=_setmode(_fileno(file), _O_BINARY); (void)unused; } -# else -# define SET_BINARY_MODE(file) setmode(fileno(file), O_BINARY) -# endif -#else -# define SET_BINARY_MODE(file) -#endif - -/* Unicode helpers for Windows to make UTF-8 act as it should. */ -#ifdef _WIN32 -/* - * Converts a UTF-8 string to UTF-16. Acts like strdup. The string must be freed afterwards. - * This version allows keeping the output length. - */ -static wchar_t* XSUM_widenString(const char* str, int* lenOut) -{ - int const len = MultiByteToWideChar(CP_UTF8, 0, str, -1, NULL, 0); - if (lenOut != NULL) *lenOut = len; - if (len == 0) return NULL; - { wchar_t* buf = (wchar_t*)malloc((size_t)len * sizeof(wchar_t)); - if (buf != NULL) { - if (MultiByteToWideChar(CP_UTF8, 0, str, -1, buf, len) == 0) { - free(buf); - return NULL; - } } - return buf; - } -} - -/* - * Converts a UTF-16 string to UTF-8. Acts like strdup. The string must be freed afterwards. - * This version allows keeping the output length. - */ -static char* XSUM_narrowString(const wchar_t *str, int *lenOut) -{ - int len = WideCharToMultiByte(CP_UTF8, 0, str, -1, NULL, 0, NULL, NULL); - if (lenOut != NULL) *lenOut = len; - if (len == 0) return NULL; - { char* const buf = (char*)malloc((size_t)len * sizeof(char)); - if (buf != NULL) { - if (WideCharToMultiByte(CP_UTF8, 0, str, -1, buf, len, NULL, NULL) == 0) { - free(buf); - return NULL; - } } - return buf; - } -} - -/* - * fopen wrapper that supports UTF-8 - * - * fopen will only accept ANSI filenames, which means that we can't open Unicode filenames. - * - * In order to open a Unicode filename, we need to convert filenames to UTF-16 and use _wfopen. - */ -static FILE* XSUM_fopen_wrapped(const char *filename, const wchar_t *mode) -{ - wchar_t* const wide_filename = XSUM_widenString(filename, NULL); - if (wide_filename == NULL) return NULL; - { FILE* const f = _wfopen(wide_filename, mode); - free(wide_filename); - return f; - } -} - -/* - * In case it isn't available, this is what MSVC 2019 defines in stdarg.h. - */ -#if defined(_MSC_VER) && !defined(__clang__) && !defined(va_copy) -# define va_copy(destination, source) ((destination) = (source)) -#endif - -/* - * fprintf wrapper that supports UTF-8. - * - * fprintf doesn't properly handle Unicode on Windows. - * - * Additionally, it is codepage sensitive on console and may crash the program. - * - * Instead, we use vsnprintf, and either print with fwrite or convert to UTF-16 - * for console output and use the codepage-independent WriteConsoleW. - * - * Credit to t-mat: https://github.com/t-mat/xxHash/commit/5691423 - */ -static int XSUM_fprintf_utf8(FILE *stream, const char *format, ...) -{ - int result; - va_list args; - va_list copy; - - va_start(args, format); - - /* - * To be safe, make a va_copy. - * - * Note that Microsoft doesn't use va_copy in its sample code: - * https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/vsprintf-vsprintf-l-vswprintf-vswprintf-l-vswprintf-l?view=vs-2019 - */ - va_copy(copy, args); - /* Counts the number of characters needed for vsnprintf. */ - result = _vscprintf(format, copy); - va_end(copy); - - if (result > 0) { - /* Create a buffer for vsnprintf */ - const size_t nchar = (size_t)result + 1; - char* u8_str = (char*)malloc(nchar * sizeof(u8_str[0])); - - if (u8_str == NULL) { - result = -1; - } else { - /* Generate the UTF-8 string with vsnprintf. */ - result = _vsnprintf(u8_str, nchar - 1, format, args); - u8_str[nchar - 1] = '\0'; - if (result > 0) { - /* - * Check if we are outputting to a console. Don't use XSUM_isConsole - * directly -- we don't need to call _get_osfhandle twice. - */ - int fileNb = _fileno(stream); - intptr_t handle_raw = _get_osfhandle(fileNb); - HANDLE handle = (HANDLE)handle_raw; - DWORD dwTemp; - - if (handle_raw < 0) { - result = -1; - } else if (_isatty(fileNb) && GetConsoleMode(handle, &dwTemp)) { - /* - * Convert to UTF-16 and output with WriteConsoleW. - * - * This is codepage independent and works on Windows XP's - * default msvcrt.dll. - */ - int len; - wchar_t *const u16_buf = XSUM_widenString(u8_str, &len); - if (u16_buf == NULL) { - result = -1; - } else { - if (WriteConsoleW(handle, u16_buf, (DWORD)len - 1, &dwTemp, NULL)) { - result = (int)dwTemp; - } else { - result = -1; - } - free(u16_buf); - } - } else { - /* fwrite the UTF-8 string if we are printing to a file */ - result = (int)fwrite(u8_str, 1, nchar - 1, stream); - if (result == 0) { - result = -1; - } - } - } - free(u8_str); - } - } - va_end(args); - return result; -} -/* - * Since we always use literals in the "mode" argument, it is just easier to append "L" to - * the string to make it UTF-16 and avoid the hassle of a second manual conversion. - */ -# define XSUM_fopen(filename, mode) XSUM_fopen_wrapped(filename, L##mode) -#else -# define XSUM_fopen(filename, mode) fopen(filename, mode) -#endif - /* ************************************ * Basic Types **************************************/ @@ -273,8 +81,6 @@ static unsigned XSUM_isLittleEndian(void) return one.c[0]; } - - static const int g_nbBits = (int)(sizeof(void*)*8); static const char g_lename[] = "little endian"; static const char g_bename[] = "big endian"; @@ -315,16 +121,6 @@ static AlgoSelected g_defaultAlgo = algo_xxh64; /* required within main() & X /* ************************************ * Display macros **************************************/ -#ifdef _WIN32 -#define XSUM_log(...) XSUM_fprintf_utf8(stderr, __VA_ARGS__) -#define XSUM_output(...) XSUM_fprintf_utf8(stdout, __VA_ARGS__) -#else -#define XSUM_log(...) fprintf(stderr, __VA_ARGS__) -#define XSUM_output(...) fprintf(stdout, __VA_ARGS__) -#endif - -#define XSUM_logVerbose(l, ...) do { if (g_displayLevel>=l) XSUM_log(__VA_ARGS__); } while (0) -static int g_displayLevel = 2; /* ************************************ @@ -341,7 +137,6 @@ static clock_t XSUM_clockSpan( clock_t start ) return clock() - start; /* works even if overflow; Typical max span ~ 30 mn */ } - static size_t XSUM_findMaxMem(U64 requiredMem) { size_t const step = 64 MB; @@ -628,7 +423,7 @@ static void XSUM_benchHash(hashFunction h, const char* hName, int testID, (unsigned)bufferSize, (double)1 / fastestH, ((double)bufferSize / (1 MB)) / fastestH); - if (g_displayLevel<1) + if (XSUM_logLevel<1) XSUM_logVerbose(0, "%u, ", (unsigned)((double)1 / fastestH)); } @@ -675,7 +470,7 @@ static size_t XSUM_selectBenchedSize(const char* fileName) } -static int XSUM_benchFiles(const char*const* fileNamesTable, int nbFiles) +static int XSUM_benchFiles(char*const* fileNamesTable, int nbFiles) { int fileIdx; for (fileIdx=0; fileIdx /* struct stat / __start64 */ -#include /* stat() / _stat64() */ - -int XSUM_isDirectory(const char* infilename) -{ - stat_t statbuf; -#if defined(_MSC_VER) - int const r = _stat64(infilename, &statbuf); - if (!r && (statbuf.st_mode & _S_IFDIR)) return 1; -#else - int const r = stat(infilename, &statbuf); - if (!r && S_ISDIR(statbuf.st_mode)) return 1; -#endif - return 0; -} /* for support of --little-endian display mode */ static void XSUM_display_LittleEndian(const void* ptr, size_t length) @@ -1433,7 +1206,7 @@ static int XSUM_hashFile(const char* fileName, if (fileName == stdinName) { inFile = stdin; fileName = "stdin"; - SET_BINARY_MODE(stdin); + XSUM_setBinaryMode(stdin); } else { if (XSUM_isDirectory(fileName)) { XSUM_log("xxhsum: %s: Is a directory \n", fileName); @@ -1493,7 +1266,7 @@ static int XSUM_hashFile(const char* fileName, * XSUM_hashFiles: * If fnTotal==0, read from stdin instead. */ -static int XSUM_hashFiles(const char*const * fnList, int fnTotal, +static int XSUM_hashFiles(char*const * fnList, int fnTotal, AlgoSelected hashType, Display_endianess displayEndianess, Display_convention convention) @@ -2001,7 +1774,7 @@ static int XSUM_checkFile(const char* inFileName, } -static int XSUM_checkFiles(const char*const* fnList, int fnTotal, +static int XSUM_checkFiles(char*const* fnList, int fnTotal, const Display_endianess displayEndianess, U32 strictMode, U32 statusOnly, @@ -2131,7 +1904,7 @@ static U32 XSUM_readU32FromChar(const char** stringPtr) { return result; } -static int XSUM_main(int argc, const char* const* argv) +int XSUM_main(int argc, char* argv[]) { int i, filenamesStart = 0; const char* const exename = XSUM_lastNameFromPath(argv[0]); @@ -2160,7 +1933,7 @@ static int XSUM_main(int argc, const char* const* argv) if (!strcmp(argument, "--check")) { fileCheckMode = 1; continue; } if (!strcmp(argument, "--benchmark-all")) { benchmarkMode = 1; selectBenchIDs = kBenchAll; continue; } if (!strcmp(argument, "--bench-all")) { benchmarkMode = 1; selectBenchIDs = kBenchAll; continue; } - if (!strcmp(argument, "--quiet")) { g_displayLevel--; continue; } + if (!strcmp(argument, "--quiet")) { XSUM_logLevel--; continue; } if (!strcmp(argument, "--little-endian")) { displayEndianess = little_endian; continue; } if (!strcmp(argument, "--strict")) { strictMode = 1; continue; } if (!strcmp(argument, "--status")) { statusOnly = 1; continue; } @@ -2248,7 +2021,7 @@ static int XSUM_main(int argc, const char* const* argv) /* Modify verbosity of benchmark output (hidden option) */ case 'q': argument++; - g_displayLevel--; + XSUM_logLevel--; break; default: @@ -2274,154 +2047,8 @@ static int XSUM_main(int argc, const char* const* argv) if (filenamesStart==0) filenamesStart = argc; if (fileCheckMode) { return XSUM_checkFiles(argv+filenamesStart, argc-filenamesStart, - displayEndianess, strictMode, statusOnly, warn, (g_displayLevel < 2) /*quiet*/); + displayEndianess, strictMode, statusOnly, warn, (XSUM_logLevel < 2) /*quiet*/); } else { return XSUM_hashFiles(argv+filenamesStart, argc-filenamesStart, algo, displayEndianess, convention); } } - -/* Windows main wrapper which properly handles UTF-8 command line arguments. */ -#ifdef _WIN32 -/* Converts a UTF-16 argv to UTF-8. */ -static char** XSUM_convertArgv(int argc, const wchar_t* const utf16_argv[]) -{ - char** const utf8_argv = (char**)malloc((size_t)(argc + 1) * sizeof(char*)); - if (utf8_argv != NULL) { - int i; - for (i = 0; i < argc; i++) { - utf8_argv[i] = XSUM_narrowString(utf16_argv[i], NULL); - } - utf8_argv[argc] = NULL; - } - return utf8_argv; -} -/* Frees arguments returned by XSUM_convertArgv */ -static void XSUM_freeArgv(int argc, char** argv) -{ - int i; - if (argv == NULL) { - return; - } - for (i = 0; i < argc; i++) { - free(argv[i]); - } - free(argv); -} - - -/* - * On Windows, main's argv parameter is useless. Instead of UTF-8, you get ANSI - * encoding, and any unknown characters will show up as mojibake. - * - * While this doesn't affect most programs, what does happen is that we can't - * open any files with Unicode filenames. - * - * We instead convert wmain's arguments to UTF-8, preserving Unicode arguments. - * - * This function is wrapped by `__wgetmainargs()` and `main()` below on MinGW - * with Unicode disabled, but if possible, we try to use `wmain()`. - */ -static int XSUM_wmain(int argc, const wchar_t* const utf16_argv[]) -{ - /* Convert the UTF-16 arguments to UTF-8. */ - char** utf8_argv = XSUM_convertArgv(argc, utf16_argv); - - if (utf8_argv == NULL) { - /* An unfortunate but incredibly unlikely error, */ - fprintf(stderr, "Error converting command line arguments!\n"); - return 1; - } else { - int ret; - - /* - * MinGW's terminal uses full block buffering for stderr. - * - * This is nonstandard behavior and causes text to not display until - * the buffer fills. - * - * `setvbuf()` can easily correct this to make text display instantly. - */ - setvbuf(stderr, NULL, _IONBF, 0); - - /* Call our real main function */ - ret = XSUM_main(argc, (const char* const *) utf8_argv); - - /* Cleanup */ - XSUM_freeArgv(argc, utf8_argv); - return ret; - } -} - -#if defined(_MSC_VER) /* MSVC always accepts wmain */ \ - || defined(_UNICODE) || defined(UNICODE) /* defined with -municode on MinGW-w64 */ - -/* Preferred: Use the real `wmain()`. */ -#if defined(__cplusplus) -extern "C" -#endif -int wmain(int argc, const wchar_t* utf16_argv[]) -{ - return XSUM_wmain(argc, utf16_argv); -} - -#else /* Non-Unicode MinGW */ - -/* - * Wrap `XSUM_wmain()` using `main()` and `__wgetmainargs()` on MinGW without - * Unicode support. - * - * `__wgetmainargs()` is used in the CRT startup to retrieve the arguments for - * `wmain()`, so we use it on MinGW to emulate `wmain()`. - * - * It is an internal function and not declared in any public headers, so we - * have to declare it manually. - * - * An alternative that doesn't mess with internal APIs is `GetCommandLineW()` - * with `CommandLineToArgvW()`, but the former doesn't expand wildcards and the - * latter requires linking to Shell32.dll and its numerous dependencies. - * - * This method keeps our dependencies to kernel32.dll and the CRT. - * - * https://docs.microsoft.com/en-us/cpp/c-runtime-library/getmainargs-wgetmainargs?view=vs-2019 - */ -typedef struct { - int newmode; -} _startupinfo; - -#ifdef __cplusplus -extern "C" -#endif -int __cdecl __wgetmainargs( - int* Argc, - wchar_t*** Argv, - wchar_t*** Env, - int DoWildCard, - _startupinfo* StartInfo -); - -int main(int ansi_argc, const char* ansi_argv[]) -{ - int utf16_argc; - wchar_t** utf16_argv; - wchar_t** utf16_envp; /* Unused but required */ - _startupinfo startinfo = {0}; /* 0 == don't change new mode */ - - /* Get wmain's UTF-16 arguments. Make sure we expand wildcards. */ - if (__wgetmainargs(&utf16_argc, &utf16_argv, &utf16_envp, 1, &startinfo) < 0) - /* In the very unlikely case of an error, use the ANSI arguments. */ - return XSUM_main(ansi_argc, ansi_argv); - - /* Call XSUM_wmain with our UTF-16 arguments */ - return XSUM_wmain(utf16_argc, (const wchar_t* const *)utf16_argv); -} - -#endif /* Non-Unicode MinGW */ - -#else /* Not Windows */ - -/* Wrap main normally on non-Windows platforms. */ -int main(int argc, const char* argv[]) -{ - return XSUM_main(argc, argv); -} -#endif /* !Windows */ From aaf6716f666665b13cfbe5f2a96a57c95d887215 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 21:38:59 -0400 Subject: [PATCH 014/187] Fix #451 Define _FILE_OFFSET_BITS to 64 --- programs/xxhsum/xsum_config.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/programs/xxhsum/xsum_config.h b/programs/xxhsum/xsum_config.h index f49bf394..dd797069 100644 --- a/programs/xxhsum/xsum_config.h +++ b/programs/xxhsum/xsum_config.h @@ -54,6 +54,9 @@ #ifndef _LARGEFILE64_SOURCE # define _LARGEFILE64_SOURCE #endif +#ifndef _FILE_OFFSET_BITS +# define _FILE_OFFSET_BITS 64 +#endif /* * So we can use __attribute__((__format__)) From 5634a1f3fe70145a5ed25537fc544430333c42b7 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:06:24 -0400 Subject: [PATCH 015/187] Fix dependency issues, XXH_INLINE_ALL affects xxhsum files --- Makefile | 19 ++++++++++--------- programs/xxhsum/xsum_config.h | 5 +++++ programs/xxhsum/xsum_os_specific.c | 14 +++++++------- programs/xxhsum/xsum_os_specific.h | 12 ++++++------ programs/xxhsum/xsum_output.c | 6 +++--- programs/xxhsum/xsum_output.h | 6 +++--- xxhsum.c | 6 +++++- 7 files changed, 39 insertions(+), 29 deletions(-) diff --git a/Makefile b/Makefile index 63349695..15a6c23d 100644 --- a/Makefile +++ b/Makefile @@ -70,9 +70,10 @@ else endif LIBXXH = libxxhash.$(SHARED_EXT_VER) -XXHSUM_OBJS = xxhsum.o \ - programs/xxhsum/xsum_os_specific.o \ - programs/xxhsum/xsum_output.o + +XXHSUM_SPLIT_SRCS = programs/xxhsum/xsum_os_specific.c \ + programs/xxhsum/xsum_output.c +XXHSUM_SPLIT_OBJS = $(XXHSUM_SPLIT_SRCS:.c=.o) XXHSUM_HEADERS = programs/xxhsum/xsum_config.h \ programs/xxhsum/xsum_arch.h \ programs/xxhsum/xsum_os_specific.h \ @@ -91,16 +92,16 @@ ifeq ($(DISPATCH),1) xxhsum: CPPFLAGS += -DXXHSUM_DISPATCH=1 xxhsum: xxh_x86dispatch.o endif -xxhsum: xxhash.o $(XXHSUM_OBJS) +xxhsum: xxhash.o xxhsum.o $(XXHSUM_SPLIT_OBJS) $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) xxhsum32: CFLAGS += -m32 ## generate CLI in 32-bits mode -xxhsum32: xxhash.c xxhsum.c ## do not generate object (avoid mixing different ABI) +xxhsum32: xxhash.c xxhsum.c $(XXHSUM_SPLIT_SRCS) ## do not generate object (avoid mixing different ABI) $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) ## dispatch only works for x86/x64 systems dispatch: CPPFLAGS += -DXXHSUM_DISPATCH=1 -dispatch: xxhash.o xxh_x86dispatch.o xxhsum.c +dispatch: xxhash.o xxh_x86dispatch.o xxhsum.c $(XXHSUM_SPLIT_SRCS) $(CC) $(FLAGS) $^ $(LDFLAGS) -o $@$(EXT) xxhash.o: xxhash.c xxhash.h @@ -115,8 +116,8 @@ xxh32sum xxh64sum xxh128sum: xxhsum ln -sf $<$(EXT) $@$(EXT) xxhsum_inlinedXXH: CPPFLAGS += -DXXH_INLINE_ALL -xxhsum_inlinedXXH: xxhsum.c - $(CC) $(FLAGS) $^ -o $@$(EXT) +xxhsum_inlinedXXH: xxhsum.c $(XXHSUM_SPLIT_SRCS) + $(CC) $(FLAGS) $< -o $@$(EXT) # library @@ -329,7 +330,7 @@ cppcheck: ## check C source files using $(CPPCHECK) static analyzer namespaceTest: ## ensure XXH_NAMESPACE redefines all public symbols $(CC) -c xxhash.c $(CC) -DXXH_NAMESPACE=TEST_ -c xxhash.c -o xxhash2.o - $(CC) xxhash.o xxhash2.o xxhsum.c -o xxhsum2 # will fail if one namespace missing (symbol collision) + $(CC) xxhash.o xxhash2.o xxhsum.c $(XXHSUM_SPLIT_SRCS) -o xxhsum2 # will fail if one namespace missing (symbol collision) $(RM) *.o xxhsum2 # clean MD2ROFF ?= ronn diff --git a/programs/xxhsum/xsum_config.h b/programs/xxhsum/xsum_config.h index dd797069..ef21a5ae 100644 --- a/programs/xxhsum/xsum_config.h +++ b/programs/xxhsum/xsum_config.h @@ -169,4 +169,9 @@ # endif #endif /* XSUM_WIN32_USE_WCHAR */ +#ifdef XXH_INLINE_ALL +# define XSUM_API static +#else +# define XSUM_API +#endif #endif /* XSUM_CONFIG_H */ diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index 45a5896f..684f8473 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -78,24 +78,24 @@ static __inline int XSUM_IS_CONSOLE(FILE* stdStream) # define XSUM_SET_BINARY_MODE(file) ((void)file) #endif -int XSUM_isConsole(FILE* stream) +XSUM_API int XSUM_isConsole(FILE* stream) { return XSUM_IS_CONSOLE(stream); } -void XSUM_setBinaryMode(FILE* stream) +XSUM_API void XSUM_setBinaryMode(FILE* stream) { XSUM_SET_BINARY_MODE(stream); } #if !XSUM_WIN32_USE_WCHAR -FILE* XSUM_fopen(const char* filename, const char* mode) +XSUM_API FILE* XSUM_fopen(const char* filename, const char* mode) { return fopen(filename, mode); } XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) -int XSUM_vfprintf(FILE* stream, const char* format, va_list ap) +XSUM_API int XSUM_vfprintf(FILE* stream, const char* format, va_list ap) { return vfprintf(stream, format, ap); } @@ -179,7 +179,7 @@ static char* XSUM_narrowString(const wchar_t *str, int *lenOut) * * In order to open a Unicode filename, we need to convert filenames to UTF-16 and use _wfopen. */ -FILE* XSUM_fopen(const char* filename, const char* mode) +XSUM_API FILE* XSUM_fopen(const char* filename, const char* mode) { FILE* f = NULL; wchar_t* const wide_filename = XSUM_widenString(filename, NULL); @@ -199,7 +199,7 @@ FILE* XSUM_fopen(const char* filename, const char* mode) * * Accepts UTF-8 filenames, unlike _stat64. */ -int XSUM_isDirectory(const char* filename) +XSUM_API int XSUM_isDirectory(const char* filename) { struct __stat64 statbuf; int result = 0; @@ -273,7 +273,7 @@ static int XSUM_vasprintf(char** strp, const char* format, va_list ap) * Credit to t-mat: https://github.com/t-mat/xxHash/commit/5691423 */ XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) -int XSUM_vfprintf(FILE *stream, const char *format, va_list ap) +XSUM_API int XSUM_vfprintf(FILE *stream, const char *format, va_list ap) { int result; char* u8_str = NULL; diff --git a/programs/xxhsum/xsum_os_specific.h b/programs/xxhsum/xsum_os_specific.h index 695e6463..58fb63c1 100644 --- a/programs/xxhsum/xsum_os_specific.h +++ b/programs/xxhsum/xsum_os_specific.h @@ -39,24 +39,24 @@ extern "C" { * * Functions like main(), but is passed UTF-8 arguments even on Windows. */ -int XSUM_main(int argc, char* argv[]); +XSUM_API int XSUM_main(int argc, char* argv[]); /* * Returns whether stream is a console. * * Functionally equivalent to isatty(fileno(stream)). */ -int XSUM_isConsole(FILE* stream); +XSUM_API int XSUM_isConsole(FILE* stream); /* * Sets stream to pure binary mode (a.k.a. no CRLF conversions). */ -void XSUM_setBinaryMode(FILE* stream); +XSUM_API void XSUM_setBinaryMode(FILE* stream); /* * Returns whether the file at filename is a directory. */ -int XSUM_isDirectory(const char* filename); +XSUM_API int XSUM_isDirectory(const char* filename); /* * UTF-8 stdio wrappers primarily for Windows @@ -68,14 +68,14 @@ int XSUM_isDirectory(const char* filename); * Specifically, on Windows, the arguments will be converted to UTF-16 * and passed to _wfopen(). */ -FILE* XSUM_fopen(const char* filename, const char* mode); +XSUM_API FILE* XSUM_fopen(const char* filename, const char* mode); /* * vfprintf() wrapper which prints UTF-8 strings to Windows consoles * if applicable. */ XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) -int XSUM_vfprintf(FILE* stream, const char* format, va_list ap); +XSUM_API int XSUM_vfprintf(FILE* stream, const char* format, va_list ap); #ifdef __cplusplus } diff --git a/programs/xxhsum/xsum_output.c b/programs/xxhsum/xsum_output.c index 83e6e7bd..a4d74115 100644 --- a/programs/xxhsum/xsum_output.c +++ b/programs/xxhsum/xsum_output.c @@ -30,7 +30,7 @@ int XSUM_logLevel = 2; XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) -int XSUM_log(const char* format, ...) +XSUM_API int XSUM_log(const char* format, ...) { int ret; va_list ap; @@ -42,7 +42,7 @@ int XSUM_log(const char* format, ...) XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) -int XSUM_output(const char* format, ...) +XSUM_API int XSUM_output(const char* format, ...) { int ret; va_list ap; @@ -53,7 +53,7 @@ int XSUM_output(const char* format, ...) } XSUM_ATTRIBUTE((__format__(__printf__, 2, 3))) -int XSUM_logVerbose(int minLevel, const char* format, ...) +XSUM_API int XSUM_logVerbose(int minLevel, const char* format, ...) { if (XSUM_logLevel >= minLevel) { int ret; diff --git a/programs/xxhsum/xsum_output.h b/programs/xxhsum/xsum_output.h index 80fec2f0..8a02c1b7 100644 --- a/programs/xxhsum/xsum_output.h +++ b/programs/xxhsum/xsum_output.h @@ -41,19 +41,19 @@ extern int XSUM_logLevel; * Same as fprintf(stderr, format, ...) */ XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) -int XSUM_log(const char *format, ...); +XSUM_API int XSUM_log(const char *format, ...); /* * Like XSUM_log, but only outputs if XSUM_logLevel >= minLevel. */ XSUM_ATTRIBUTE((__format__(__printf__, 2, 3))) -int XSUM_logVerbose(int minLevel, const char *format, ...); +XSUM_API int XSUM_logVerbose(int minLevel, const char *format, ...); /* * Same as printf(format, ...) */ XSUM_ATTRIBUTE((__format__(__printf__, 1, 2))) -int XSUM_output(const char *format, ...); +XSUM_API int XSUM_output(const char *format, ...); #ifdef __cplusplus } diff --git a/xxhsum.c b/xxhsum.c index 64bc6c79..0e4a87c6 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -34,6 +34,10 @@ #include "programs/xxhsum/xsum_arch.h" #include "programs/xxhsum/xsum_os_specific.h" #include "programs/xxhsum/xsum_output.h" +#ifdef XXH_INLINE_ALL +# include "programs/xxhsum/xsum_os_specific.c" +# include "programs/xxhsum/xsum_output.c" +#endif /* ************************************ * Includes @@ -1904,7 +1908,7 @@ static U32 XSUM_readU32FromChar(const char** stringPtr) { return result; } -int XSUM_main(int argc, char* argv[]) +XSUM_API int XSUM_main(int argc, char* argv[]) { int i, filenamesStart = 0; const char* const exename = XSUM_lastNameFromPath(argv[0]); From 7cd2f455e7370757678f293e7a1c3af3ed7b27f4 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:30:18 -0400 Subject: [PATCH 016/187] xxhsum: Move XSUM_getFileSize --- programs/xxhsum/xsum_config.h | 32 ++++- programs/xxhsum/xsum_os_specific.c | 57 ++++++--- programs/xxhsum/xsum_os_specific.h | 5 + xxhsum.c | 195 ++++++++++++----------------- 4 files changed, 150 insertions(+), 139 deletions(-) diff --git a/programs/xxhsum/xsum_config.h b/programs/xxhsum/xsum_config.h index ef21a5ae..9895744a 100644 --- a/programs/xxhsum/xsum_config.h +++ b/programs/xxhsum/xsum_config.h @@ -169,9 +169,33 @@ # endif #endif /* XSUM_WIN32_USE_WCHAR */ -#ifdef XXH_INLINE_ALL -# define XSUM_API static -#else -# define XSUM_API +#ifndef XSUM_API +# ifdef XXH_INLINE_ALL +# define XSUM_API static +# else +# define XSUM_API +# endif #endif + +/* *************************** + * Basic types + * ***************************/ + +#if defined(__cplusplus) /* C++ */ \ + || (defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L) /* C99 */ +# include + typedef uint8_t XSUM_U8; + typedef uint32_t XSUM_U32; + typedef uint64_t XSUM_U64; +# else +# include + typedef unsigned char XSUM_U8; +# if UINT_MAX == 0xFFFFFFFFUL + typedef unsigned int XSUM_U32; +# else + typedef unsigned long XSUM_U32; +# endif + typedef unsigned long long XSUM_U64; +#endif /* not C++/C99 */ + #endif /* XSUM_CONFIG_H */ diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index 684f8473..5dde3668 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -36,10 +36,10 @@ * platforms. */ #if defined(_MSC_VER) - typedef struct __stat64 stat_t; + typedef struct __stat64 XSUM_stat_t; typedef int mode_t; #else - typedef struct stat stat_t; + typedef struct stat XSUM_stat_t; #endif #if (defined(__linux__) && (XSUM_PLATFORM_POSIX_VERSION >= 1)) \ @@ -100,17 +100,13 @@ XSUM_API int XSUM_vfprintf(FILE* stream, const char* format, va_list ap) return vfprintf(stream, format, ap); } -int XSUM_isDirectory(const char* infilename) +static int XSUM_stat(const char* infilename, XSUM_stat_t* statbuf) { - stat_t statbuf; #if defined(_MSC_VER) - int const r = _stat64(infilename, &statbuf); - if (!r && (statbuf.st_mode & _S_IFDIR)) return 1; + return _stat64(infilename, statbuf); #else - int const r = stat(infilename, &statbuf); - if (!r && S_ISDIR(statbuf.st_mode)) return 1; + return stat(infilename, statbuf); #endif - return 0; } #ifndef XSUM_NO_MAIN @@ -195,23 +191,17 @@ XSUM_API FILE* XSUM_fopen(const char* filename, const char* mode) } /* - * Determines whether the file at path is a directory. - * - * Accepts UTF-8 filenames, unlike _stat64. + * stat() wrapper which supports UTF-8 filenames. */ -XSUM_API int XSUM_isDirectory(const char* filename) +static int XSUM_stat(const char* infilename, XSUM_stat_t* statbuf) { - struct __stat64 statbuf; - int result = 0; + int r = -1; wchar_t* const wide_filename = XSUM_widenString(filename, NULL); if (wide_filename != NULL) { - if (_wstat64(wide_filename, &statbuf) == 0 /* stat fail is ok */ - && (statbuf.st_mode & _S_IFDIR)) { - result = 1; - } + r = _wstat64(wide_filename, statbuf); free(wide_filename); } - return result; + return r; } /* @@ -465,3 +455,30 @@ int main(int ansi_argc, char* ansi_argv[]) #endif /* !XSUM_WIN32_USE_WMAIN */ #endif /* !XSUM_NO_MAIN */ #endif /* XSUM_WIN32_USE_WCHAR */ + + +/* + * Determines whether the file at filename is a directory. + */ +XSUM_API int XSUM_isDirectory(const char* filename) +{ + XSUM_stat_t statbuf; + int r = XSUM_stat(filename, &statbuf); +#ifdef _MSC_VER + if (!r && (statbuf.st_mode & _S_IFDIR)) return 1; +#else + if (!r && S_ISDIR(statbuf.st_mode)) return 1; +#endif + return 0; +} + +/* + * Returns the filesize of the file at filename. + */ +XSUM_API XSUM_U64 XSUM_getFileSize(const char* filename) +{ + XSUM_stat_t statbuf; + int r = XSUM_stat(filename, &statbuf); + if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ + return (XSUM_U64)statbuf.st_size; +} diff --git a/programs/xxhsum/xsum_os_specific.h b/programs/xxhsum/xsum_os_specific.h index 58fb63c1..b3562b26 100644 --- a/programs/xxhsum/xsum_os_specific.h +++ b/programs/xxhsum/xsum_os_specific.h @@ -58,6 +58,11 @@ XSUM_API void XSUM_setBinaryMode(FILE* stream); */ XSUM_API int XSUM_isDirectory(const char* filename); +/* + * Returns the file size of the file at filename. + */ +XSUM_API XSUM_U64 XSUM_getFileSize(const char* filename); + /* * UTF-8 stdio wrappers primarily for Windows */ diff --git a/xxhsum.c b/xxhsum.c index 0e4a87c6..96131f62 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -59,29 +59,9 @@ # include "xxh_x86dispatch.h" #endif -/* ************************************ -* Basic Types -**************************************/ -#if defined(__cplusplus) /* C++ */ \ - || (defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L) /* C99 */ -# include - typedef uint8_t U8; - typedef uint32_t U32; - typedef uint64_t U64; -# else -# include - typedef unsigned char U8; -# if UINT_MAX == 0xFFFFFFFFUL - typedef unsigned int U32; -# else - typedef unsigned long U32; -# endif - typedef unsigned long long U64; -#endif /* not C++/C99 */ - static unsigned XSUM_isLittleEndian(void) { - const union { U32 u; U8 c[4]; } one = { 1 }; /* don't use static: performance detrimental */ + const union { XSUM_U32 u; XSUM_U8 c[4]; } one = { 1 }; /* don't use static: performance detrimental */ return one.c[0]; } @@ -130,7 +110,7 @@ static AlgoSelected g_defaultAlgo = algo_xxh64; /* required within main() & X /* ************************************ * Local variables **************************************/ -static U32 g_nbIterations = NBLOOPS; +static XSUM_U32 g_nbIterations = NBLOOPS; /* ************************************ @@ -141,7 +121,7 @@ static clock_t XSUM_clockSpan( clock_t start ) return clock() - start; /* works even if overflow; Typical max span ~ 30 mn */ } -static size_t XSUM_findMaxMem(U64 requiredMem) +static size_t XSUM_findMaxMem(XSUM_U64 requiredMem) { size_t const step = 64 MB; void* testmem = NULL; @@ -164,21 +144,6 @@ static size_t XSUM_findMaxMem(U64 requiredMem) return (size_t)requiredMem; } - -static U64 XSUM_GetFileSize(const char* infilename) -{ - int r; -#if defined(_MSC_VER) - struct _stat64 statbuf; - r = _stat64(infilename, &statbuf); -#else - struct stat statbuf; - r = stat(infilename, &statbuf); -#endif - if (r || !S_ISREG(statbuf.st_mode)) return 0; /* No good... */ - return (U64)statbuf.st_size; -} - /* * Allocates a string containing s1 and s2 concatenated. Acts like strdup. * The result must be freed. @@ -210,15 +175,15 @@ static char* XSUM_strcatDup(const char* s1, const char* s2) * * This is used in the sanity check - its values must not be changed. */ -static void XSUM_fillTestBuffer(U8* buffer, size_t len) +static void XSUM_fillTestBuffer(XSUM_U8* buffer, size_t len) { - U64 byteGen = PRIME32; + XSUM_U64 byteGen = PRIME32; size_t i; assert(buffer != NULL); for (i=0; i>56); + buffer[i] = (XSUM_U8)(byteGen>>56); byteGen *= PRIME64; } } @@ -231,7 +196,7 @@ static void XSUM_fillTestBuffer(U8* buffer, size_t len) * * Adding a pointer to the parameter list would be messy. */ -static U8 g_benchSecretBuf[XXH3_SECRET_SIZE_MIN]; +static XSUM_U8 g_benchSecretBuf[XXH3_SECRET_SIZE_MIN]; /* * Wrappers for the benchmark. @@ -239,75 +204,75 @@ static U8 g_benchSecretBuf[XXH3_SECRET_SIZE_MIN]; * If you would like to add other hashes to the bench, create a wrapper and add * it to the g_hashesToBench table. It will automatically be added. */ -typedef U32 (*hashFunction)(const void* buffer, size_t bufferSize, U32 seed); +typedef XSUM_U32 (*hashFunction)(const void* buffer, size_t bufferSize, XSUM_U32 seed); -static U32 localXXH32(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH32(const void* buffer, size_t bufferSize, XSUM_U32 seed) { return XXH32(buffer, bufferSize, seed); } -static U32 localXXH64(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH64(const void* buffer, size_t bufferSize, XSUM_U32 seed) { - return (U32)XXH64(buffer, bufferSize, seed); + return (XSUM_U32)XXH64(buffer, bufferSize, seed); } -static U32 localXXH3_64b(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_64b(const void* buffer, size_t bufferSize, XSUM_U32 seed) { (void)seed; - return (U32)XXH3_64bits(buffer, bufferSize); + return (XSUM_U32)XXH3_64bits(buffer, bufferSize); } -static U32 localXXH3_64b_seeded(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_64b_seeded(const void* buffer, size_t bufferSize, XSUM_U32 seed) { - return (U32)XXH3_64bits_withSeed(buffer, bufferSize, seed); + return (XSUM_U32)XXH3_64bits_withSeed(buffer, bufferSize, seed); } -static U32 localXXH3_64b_secret(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_64b_secret(const void* buffer, size_t bufferSize, XSUM_U32 seed) { (void)seed; - return (U32)XXH3_64bits_withSecret(buffer, bufferSize, g_benchSecretBuf, sizeof(g_benchSecretBuf)); + return (XSUM_U32)XXH3_64bits_withSecret(buffer, bufferSize, g_benchSecretBuf, sizeof(g_benchSecretBuf)); } -static U32 localXXH3_128b(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_128b(const void* buffer, size_t bufferSize, XSUM_U32 seed) { (void)seed; - return (U32)(XXH3_128bits(buffer, bufferSize).low64); + return (XSUM_U32)(XXH3_128bits(buffer, bufferSize).low64); } -static U32 localXXH3_128b_seeded(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_128b_seeded(const void* buffer, size_t bufferSize, XSUM_U32 seed) { - return (U32)(XXH3_128bits_withSeed(buffer, bufferSize, seed).low64); + return (XSUM_U32)(XXH3_128bits_withSeed(buffer, bufferSize, seed).low64); } -static U32 localXXH3_128b_secret(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_128b_secret(const void* buffer, size_t bufferSize, XSUM_U32 seed) { (void)seed; - return (U32)(XXH3_128bits_withSecret(buffer, bufferSize, g_benchSecretBuf, sizeof(g_benchSecretBuf)).low64); + return (XSUM_U32)(XXH3_128bits_withSecret(buffer, bufferSize, g_benchSecretBuf, sizeof(g_benchSecretBuf)).low64); } -static U32 localXXH3_stream(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_stream(const void* buffer, size_t bufferSize, XSUM_U32 seed) { XXH3_state_t state; (void)seed; XXH3_64bits_reset(&state); XXH3_64bits_update(&state, buffer, bufferSize); - return (U32)XXH3_64bits_digest(&state); + return (XSUM_U32)XXH3_64bits_digest(&state); } -static U32 localXXH3_stream_seeded(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH3_stream_seeded(const void* buffer, size_t bufferSize, XSUM_U32 seed) { XXH3_state_t state; XXH3_INITSTATE(&state); XXH3_64bits_reset_withSeed(&state, (XXH64_hash_t)seed); XXH3_64bits_update(&state, buffer, bufferSize); - return (U32)XXH3_64bits_digest(&state); + return (XSUM_U32)XXH3_64bits_digest(&state); } -static U32 localXXH128_stream(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH128_stream(const void* buffer, size_t bufferSize, XSUM_U32 seed) { XXH3_state_t state; (void)seed; XXH3_128bits_reset(&state); XXH3_128bits_update(&state, buffer, bufferSize); - return (U32)(XXH3_128bits_digest(&state).low64); + return (XSUM_U32)(XXH3_128bits_digest(&state).low64); } -static U32 localXXH128_stream_seeded(const void* buffer, size_t bufferSize, U32 seed) +static XSUM_U32 localXXH128_stream_seeded(const void* buffer, size_t bufferSize, XSUM_U32 seed) { XXH3_state_t state; XXH3_INITSTATE(&state); XXH3_128bits_reset_withSeed(&state, (XXH64_hash_t)seed); XXH3_128bits_update(&state, buffer, bufferSize); - return (U32)(XXH3_128bits_digest(&state).low64); + return (XSUM_U32)(XXH3_128bits_digest(&state).low64); } @@ -344,14 +309,14 @@ static const char k_testIDs_default[NB_TESTFUNC] = { 0, static void XSUM_benchHash(hashFunction h, const char* hName, int testID, const void* buffer, size_t bufferSize) { - U32 nbh_perIteration = (U32)((300 MB) / (bufferSize+1)) + 1; /* first iteration conservatively aims for 300 MB/s */ + XSUM_U32 nbh_perIteration = (XSUM_U32)((300 MB) / (bufferSize+1)) + 1; /* first iteration conservatively aims for 300 MB/s */ unsigned iterationNb, nbIterations = g_nbIterations + !g_nbIterations /* min 1 */; double fastestH = 100000000.; assert(HASHNAME_MAX > 2); XSUM_logVerbose(2, "\r%80s\r", ""); /* Clean display line */ for (iterationNb = 1; iterationNb <= nbIterations; iterationNb++) { - U32 r=0; + XSUM_U32 r=0; clock_t cStart; XSUM_logVerbose(2, "%2u-%-*.*s : %10u ->\r", @@ -362,7 +327,7 @@ static void XSUM_benchHash(hashFunction h, const char* hName, int testID, while (clock() == cStart); /* starts clock() at its exact beginning */ cStart = clock(); - { U32 u; + { XSUM_U32 u; for (u=0; u (double)(4000U<<20)) nbh_perSecond = (double)(4000U<<20); /* avoid overflow */ - nbh_perIteration = (U32)nbh_perSecond; + nbh_perIteration = (XSUM_U32)nbh_perSecond; } /* g_nbIterations==0 => quick evaluation, no claim of accuracy */ if (g_nbIterations>0) { @@ -418,7 +383,7 @@ static void XSUM_benchHash(hashFunction h, const char* hName, int testID, } } { double nbh_perSecond = (1 / fastestH) + 1; if (nbh_perSecond > (double)(4000U<<20)) nbh_perSecond = (double)(4000U<<20); /* avoid overflow */ - nbh_perIteration = (U32)nbh_perSecond; + nbh_perIteration = (XSUM_U32)nbh_perSecond; } } XSUM_logVerbose(1, "%2i#%-*.*s : %10u -> %8.0f it/s (%7.1f MB/s) \n", @@ -464,9 +429,9 @@ static void XSUM_benchMem(const void* buffer, size_t bufferSize) static size_t XSUM_selectBenchedSize(const char* fileName) { - U64 const inFileSize = XSUM_GetFileSize(fileName); + XSUM_U64 const inFileSize = XSUM_getFileSize(fileName); size_t benchedSize = (size_t) XSUM_findMaxMem(inFileSize); - if ((U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; + if ((XSUM_U64)benchedSize > inFileSize) benchedSize = (size_t)inFileSize; if (benchedSize < inFileSize) { XSUM_log("Not enough memory for '%s' full size; testing %i MB only...\n", fileName, (int)(benchedSize>>20)); } @@ -590,7 +555,7 @@ static void XSUM_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) } -static void XSUM_testXXH32(const void* data, size_t len, U32 seed, U32 Nresult) +static void XSUM_testXXH32(const void* data, size_t len, XSUM_U32 seed, XSUM_U32 Nresult) { XXH32_state_t *state = XXH32_createState(); size_t pos; @@ -611,7 +576,7 @@ static void XSUM_testXXH32(const void* data, size_t len, U32 seed, U32 Nresult) XXH32_freeState(state); } -static void XSUM_testXXH64(const void* data, size_t len, U64 seed, U64 Nresult) +static void XSUM_testXXH64(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) { XXH64_state_t *state = XXH64_createState(); size_t pos; @@ -632,25 +597,25 @@ static void XSUM_testXXH64(const void* data, size_t len, U64 seed, U64 Nresult) XXH64_freeState(state); } -static U32 XSUM_rand(void) +static XSUM_U32 XSUM_rand(void) { - static U64 seed = PRIME32; + static XSUM_U64 seed = PRIME32; seed *= PRIME64; - return (U32)(seed >> 40); + return (XSUM_U32)(seed >> 40); } -void XSUM_testXXH3(const void* data, size_t len, U64 seed, U64 Nresult) +void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) { if (len>0) assert(data != NULL); - { U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); + { XSUM_U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); XSUM_checkResult64(Dresult, Nresult); } /* check that the no-seed variant produces same result as seed==0 */ if (seed == 0) { - U64 const Dresult = XXH3_64bits(data, len); + XSUM_U64 const Dresult = XXH3_64bits(data, len); XSUM_checkResult64(Dresult, Nresult); } @@ -686,11 +651,11 @@ void XSUM_testXXH3(const void* data, size_t len, U64 seed, U64 Nresult) } } -void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* secret, size_t secretSize, U64 Nresult) +void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* secret, size_t secretSize, XSUM_U64 Nresult) { if (len>0) assert(data != NULL); - { U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); + { XSUM_U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); XSUM_checkResult64(Dresult, Nresult); } @@ -725,7 +690,7 @@ void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* secret, } } -void XSUM_testXXH128(const void* data, size_t len, U64 seed, XXH128_hash_t Nresult) +void XSUM_testXXH128(const void* data, size_t len, XSUM_U64 seed, XXH128_hash_t Nresult) { { XXH128_hash_t const Dresult = XXH3_128bits_withSeed(data, len, seed); XSUM_checkResult128(Dresult, Nresult); @@ -815,13 +780,13 @@ void XSUM_testXXH128_withSecret(const void* data, size_t len, const void* secret } #define SECRET_SAMPLE_NBBYTES 4 -typedef struct { U8 byte[SECRET_SAMPLE_NBBYTES]; } verifSample_t; +typedef struct { XSUM_U8 byte[SECRET_SAMPLE_NBBYTES]; } verifSample_t; void XSUM_testSecretGenerator(const void* customSeed, size_t len, verifSample_t result) { static int nbTests = 1; const int sampleIndex[SECRET_SAMPLE_NBBYTES] = { 0, 62, 131, 191}; - U8 secretBuffer[XXH3_SECRET_DEFAULT_SIZE] = {0}; + XSUM_U8 secretBuffer[XXH3_SECRET_DEFAULT_SIZE] = {0}; verifSample_t samples; int i; @@ -849,7 +814,7 @@ void XSUM_testSecretGenerator(const void* customSeed, size_t len, verifSample_t static void XSUM_sanityCheck(void) { #define SANITY_BUFFER_SIZE 2367 - U8 sanityBuffer[SANITY_BUFFER_SIZE]; + XSUM_U8 sanityBuffer[SANITY_BUFFER_SIZE]; XSUM_fillTestBuffer(sanityBuffer, sizeof(sanityBuffer)); XSUM_testXXH32(NULL, 0, 0, 0x02CC5D05); @@ -1048,7 +1013,7 @@ static void XSUM_sanityCheck(void) /* for support of --little-endian display mode */ static void XSUM_display_LittleEndian(const void* ptr, size_t length) { - const U8* const p = (const U8*)ptr; + const XSUM_U8* const p = (const XSUM_U8*)ptr; size_t idx; for (idx=length-1; idx='0') && (**stringPtr <='9')) { if (result > max) return 1; /* overflow error */ result *= 10; - result += (U32)(**stringPtr - '0'); + result += (XSUM_U32)(**stringPtr - '0'); (*stringPtr)++ ; } if ((**stringPtr=='K') || (**stringPtr=='M')) { - U32 const maxK = ((U32)(-1)) >> 10; + XSUM_U32 const maxK = ((XSUM_U32)(-1)) >> 10; if (result > maxK) return 1; /* overflow error */ result <<= 10; if (**stringPtr=='M') { @@ -1899,8 +1864,8 @@ static int XSUM_readU32FromCharChecked(const char** stringPtr, U32* value) * Will also modify `*stringPtr`, advancing it to position where it stopped reading. * Note: function will exit() program if digit sequence overflows */ -static U32 XSUM_readU32FromChar(const char** stringPtr) { - U32 result; +static XSUM_U32 XSUM_readU32FromChar(const char** stringPtr) { + XSUM_U32 result; if (XSUM_readU32FromCharChecked(stringPtr, &result)) { static const char errorMsg[] = "Error: numeric value too large"; errorOut(errorMsg); @@ -1912,14 +1877,14 @@ XSUM_API int XSUM_main(int argc, char* argv[]) { int i, filenamesStart = 0; const char* const exename = XSUM_lastNameFromPath(argv[0]); - U32 benchmarkMode = 0; - U32 fileCheckMode = 0; - U32 strictMode = 0; - U32 statusOnly = 0; - U32 warn = 0; + XSUM_U32 benchmarkMode = 0; + XSUM_U32 fileCheckMode = 0; + XSUM_U32 strictMode = 0; + XSUM_U32 statusOnly = 0; + XSUM_U32 warn = 0; int explicitStdin = 0; - U32 selectBenchIDs= 0; /* 0 == use default k_testIDs_default, kBenchAll == bench all */ - static const U32 kBenchAll = 99; + XSUM_U32 selectBenchIDs= 0; /* 0 == use default k_testIDs_default, kBenchAll == bench all */ + static const XSUM_U32 kBenchAll = 99; size_t keySize = XSUM_DEFAULT_SAMPLE_SIZE; AlgoSelected algo = g_defaultAlgo; Display_endianess displayEndianess = big_endian; From 276bb6a8e6b11d1f00386f602a0a3a8e387d59ee Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:33:25 -0400 Subject: [PATCH 017/187] Add missing printf attribute --- programs/xxhsum/xsum_os_specific.c | 1 + 1 file changed, 1 insertion(+) diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index 5dde3668..e5079c9e 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -216,6 +216,7 @@ static int XSUM_stat(const char* infilename, XSUM_stat_t* statbuf) /* * vasprintf for Windows. */ +XSUM_ATTRIBUTE((__format__(__printf__, 2, 0))) static int XSUM_vasprintf(char** strp, const char* format, va_list ap) { int ret; From fdf2e840cbc205921d0caf8853972d0b9d3b66f3 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:41:36 -0400 Subject: [PATCH 018/187] Fix Unicode test Now uses XXH_INLINE_ALL. --- tests/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/Makefile b/tests/Makefile index 361032fd..092711ad 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -53,7 +53,7 @@ test_ppc_redefine: ppc_define.c $(CC) $(CPPFLAGS) $(CFLAGS) -c $^ xxhsum$(EXT): ../xxhash.c ../xxhash.h ../xxhsum.c - $(CC) $(CFLAGS) $(LDFLAGS) ../xxhash.c ../xxhsum.c -o $@ + $(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) -DXXH_INLINE_ALL ../xxhsum.c -o $@ # Make sure that Unicode filenames work. # https://github.com/Cyan4973/xxHash/issues/293 From 598e32c4902f681fe6195296eaa2280c4cc050c2 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:43:23 -0400 Subject: [PATCH 019/187] Fix typo --- programs/xxhsum/xsum_os_specific.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index e5079c9e..f6e873a0 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -196,7 +196,7 @@ XSUM_API FILE* XSUM_fopen(const char* filename, const char* mode) static int XSUM_stat(const char* infilename, XSUM_stat_t* statbuf) { int r = -1; - wchar_t* const wide_filename = XSUM_widenString(filename, NULL); + wchar_t* const wide_filename = XSUM_widenString(infilename, NULL); if (wide_filename != NULL) { r = _wstat64(wide_filename, statbuf); free(wide_filename); From 2f9c567857d572d5ecea0d8c17f72901a70de330 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Wed, 16 Sep 2020 22:55:19 -0400 Subject: [PATCH 020/187] Use __stat64 on MinGW in wchar mode --- programs/xxhsum/xsum_os_specific.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index f6e873a0..cdf44d6c 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -35,7 +35,7 @@ * This file contains all of the ugly boilerplate to make xxhsum work across * platforms. */ -#if defined(_MSC_VER) +#if defined(_MSC_VER) || XSUM_WIN32_USE_WCHAR typedef struct __stat64 XSUM_stat_t; typedef int mode_t; #else From a217d5fa464c8945261c1833863779035ff399c7 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Thu, 17 Sep 2020 10:05:37 -0400 Subject: [PATCH 021/187] xxhsum: Don't redefine mode_t on MinGW --- programs/xxhsum/xsum_os_specific.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/programs/xxhsum/xsum_os_specific.c b/programs/xxhsum/xsum_os_specific.c index cdf44d6c..8f48ce07 100644 --- a/programs/xxhsum/xsum_os_specific.c +++ b/programs/xxhsum/xsum_os_specific.c @@ -37,7 +37,9 @@ */ #if defined(_MSC_VER) || XSUM_WIN32_USE_WCHAR typedef struct __stat64 XSUM_stat_t; +# if defined(_MSC_VER) typedef int mode_t; +# endif #else typedef struct stat XSUM_stat_t; #endif From 4b0d1731f1a082942c62569e9a8368e184fc8c73 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Fri, 18 Sep 2020 17:44:15 -0400 Subject: [PATCH 022/187] xxhsum: move programs/xxhsum -> cli by request also use XXHSUM_SRC_DIR in Makefile --- .gitignore | 1 - Makefile | 15 ++++++++------- {programs/xxhsum => cli}/xsum_arch.h | 0 {programs/xxhsum => cli}/xsum_config.h | 0 {programs/xxhsum => cli}/xsum_os_specific.c | 0 {programs/xxhsum => cli}/xsum_os_specific.h | 0 {programs/xxhsum => cli}/xsum_output.c | 0 {programs/xxhsum => cli}/xsum_output.h | 0 cmake_unofficial/CMakeLists.txt | 5 +++-- xxhsum.c | 12 ++++++------ 10 files changed, 17 insertions(+), 16 deletions(-) rename {programs/xxhsum => cli}/xsum_arch.h (100%) rename {programs/xxhsum => cli}/xsum_config.h (100%) rename {programs/xxhsum => cli}/xsum_os_specific.c (100%) rename {programs/xxhsum => cli}/xsum_os_specific.h (100%) rename {programs/xxhsum => cli}/xsum_output.c (100%) rename {programs/xxhsum => cli}/xsum_output.h (100%) diff --git a/.gitignore b/.gitignore index a2209534..d0ce9aac 100644 --- a/.gitignore +++ b/.gitignore @@ -13,7 +13,6 @@ xxh32sum xxh64sum xxh128sum xxhsum -!programs/xxhsum xxhsum32 xxhsum_privateXXH xxhsum_inlinedXXH diff --git a/Makefile b/Makefile index 15a6c23d..4ed334d1 100644 --- a/Makefile +++ b/Makefile @@ -71,13 +71,14 @@ endif LIBXXH = libxxhash.$(SHARED_EXT_VER) -XXHSUM_SPLIT_SRCS = programs/xxhsum/xsum_os_specific.c \ - programs/xxhsum/xsum_output.c +XXHSUM_SRC_DIR = cli +XXHSUM_SPLIT_SRCS = $(XXHSUM_SRC_DIR)/xsum_os_specific.c \ + $(XXHSUM_SRC_DIR)/xsum_output.c XXHSUM_SPLIT_OBJS = $(XXHSUM_SPLIT_SRCS:.c=.o) -XXHSUM_HEADERS = programs/xxhsum/xsum_config.h \ - programs/xxhsum/xsum_arch.h \ - programs/xxhsum/xsum_os_specific.h \ - programs/xxhsum/xsum_output.h +XXHSUM_HEADERS = $(XXHSUM_SRC_DIR)/xsum_config.h \ + $(XXHSUM_SRC_DIR)/xsum_arch.h \ + $(XXHSUM_SRC_DIR)/xsum_os_specific.h \ + $(XXHSUM_SRC_DIR)/xsum_output.h ## generate CLI and libraries in release mode (default for `make`) .PHONY: default @@ -168,7 +169,7 @@ clean: ## remove all build artifacts $(Q)$(RM) core *.o *.obj *.$(SHARED_EXT) *.$(SHARED_EXT).* *.a libxxhash.pc $(Q)$(RM) xxhsum$(EXT) xxhsum32$(EXT) xxhsum_inlinedXXH$(EXT) dispatch$(EXT) $(Q)$(RM) xxh32sum$(EXT) xxh64sum$(EXT) xxh128sum$(EXT) - $(Q)$(RM) programs/xxhsum/*.o programs/xxhsum/*.obj + $(Q)$(RM) $(XXHSUM_SRC_DIR)/*.o $(XXHSUM_SRC_DIR)/*.obj @echo cleaning completed diff --git a/programs/xxhsum/xsum_arch.h b/cli/xsum_arch.h similarity index 100% rename from programs/xxhsum/xsum_arch.h rename to cli/xsum_arch.h diff --git a/programs/xxhsum/xsum_config.h b/cli/xsum_config.h similarity index 100% rename from programs/xxhsum/xsum_config.h rename to cli/xsum_config.h diff --git a/programs/xxhsum/xsum_os_specific.c b/cli/xsum_os_specific.c similarity index 100% rename from programs/xxhsum/xsum_os_specific.c rename to cli/xsum_os_specific.c diff --git a/programs/xxhsum/xsum_os_specific.h b/cli/xsum_os_specific.h similarity index 100% rename from programs/xxhsum/xsum_os_specific.h rename to cli/xsum_os_specific.h diff --git a/programs/xxhsum/xsum_output.c b/cli/xsum_output.c similarity index 100% rename from programs/xxhsum/xsum_output.c rename to cli/xsum_output.c diff --git a/programs/xxhsum/xsum_output.h b/cli/xsum_output.h similarity index 100% rename from programs/xxhsum/xsum_output.h rename to cli/xsum_output.h diff --git a/cmake_unofficial/CMakeLists.txt b/cmake_unofficial/CMakeLists.txt index 3a5086c8..5abd0c5f 100644 --- a/cmake_unofficial/CMakeLists.txt +++ b/cmake_unofficial/CMakeLists.txt @@ -82,10 +82,11 @@ set_target_properties(xxhash PROPERTIES VERSION "${XXHASH_VERSION_STRING}") if(XXHASH_BUILD_XXHSUM) + set(XXHSUM_DIR "${XXHASH_DIR}/cli") # xxhsum add_executable(xxhsum "${XXHASH_DIR}/xxhsum.c" - "${XXHASH_DIR}/programs/xxhsum/xsum_os_specific.c" - "${XXHASH_DIR}/programs/xxhsum/xsum_output.c" + "${XXHSUM_DIR}/xsum_os_specific.c" + "${XXHSUM_DIR}/xsum_output.c" ) add_executable(${PROJECT_NAME}::xxhsum ALIAS xxhsum) diff --git a/xxhsum.c b/xxhsum.c index 96131f62..bac55b20 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -30,13 +30,13 @@ */ /* Transitional headers */ -#include "programs/xxhsum/xsum_config.h" -#include "programs/xxhsum/xsum_arch.h" -#include "programs/xxhsum/xsum_os_specific.h" -#include "programs/xxhsum/xsum_output.h" +#include "cli/xsum_config.h" +#include "cli/xsum_arch.h" +#include "cli/xsum_os_specific.h" +#include "cli/xsum_output.h" #ifdef XXH_INLINE_ALL -# include "programs/xxhsum/xsum_os_specific.c" -# include "programs/xxhsum/xsum_output.c" +# include "cli/xsum_os_specific.c" +# include "cli/xsum_output.c" #endif /* ************************************ From 1d00c512374055d81848a7a427f4a1bfb297c96f Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Sat, 19 Sep 2020 19:27:05 -0400 Subject: [PATCH 023/187] xxhsum: split sanity check It now exists in xsum_sanity_check.c Also add `XSUM_NO_TESTS` option, which instead prints "This version of xxhsum is not verified." to stderr whenever XSUM_sanityCheck() is called. --- Makefile | 6 +- cli/xsum_config.h | 4 + cli/xsum_sanity_check.c | 564 ++++++++++++++++++++++++++++++++ cli/xsum_sanity_check.h | 57 ++++ cmake_unofficial/CMakeLists.txt | 1 + xxhsum.c | 524 +---------------------------- 6 files changed, 632 insertions(+), 524 deletions(-) create mode 100644 cli/xsum_sanity_check.c create mode 100644 cli/xsum_sanity_check.h diff --git a/Makefile b/Makefile index 4ed334d1..83092358 100644 --- a/Makefile +++ b/Makefile @@ -73,12 +73,14 @@ LIBXXH = libxxhash.$(SHARED_EXT_VER) XXHSUM_SRC_DIR = cli XXHSUM_SPLIT_SRCS = $(XXHSUM_SRC_DIR)/xsum_os_specific.c \ - $(XXHSUM_SRC_DIR)/xsum_output.c + $(XXHSUM_SRC_DIR)/xsum_output.c \ + $(XXHSUM_SRC_DIR)/xsum_sanity_check.c XXHSUM_SPLIT_OBJS = $(XXHSUM_SPLIT_SRCS:.c=.o) XXHSUM_HEADERS = $(XXHSUM_SRC_DIR)/xsum_config.h \ $(XXHSUM_SRC_DIR)/xsum_arch.h \ $(XXHSUM_SRC_DIR)/xsum_os_specific.h \ - $(XXHSUM_SRC_DIR)/xsum_output.h + $(XXHSUM_SRC_DIR)/xsum_output.h \ + $(XXHSUM_SRC_DIR)/xsum_sanity_check.h ## generate CLI and libraries in release mode (default for `make`) .PHONY: default diff --git a/cli/xsum_config.h b/cli/xsum_config.h index 9895744a..9222144d 100644 --- a/cli/xsum_config.h +++ b/cli/xsum_config.h @@ -177,6 +177,10 @@ # endif #endif +#ifndef XSUM_NO_TESTS +# define XSUM_NO_TESTS 0 +#endif + /* *************************** * Basic types * ***************************/ diff --git a/cli/xsum_sanity_check.c b/cli/xsum_sanity_check.c new file mode 100644 index 00000000..6d9c6c13 --- /dev/null +++ b/cli/xsum_sanity_check.c @@ -0,0 +1,564 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#include "xsum_config.h" +#include "xsum_sanity_check.h" +#include "xsum_output.h" +#include +#include +#include +#ifndef XXH_STATIC_LINKING_ONLY +# define XXH_STATIC_LINKING_ONLY +#endif +#include "../xxhash.h" + +/* use #define to make them constant, required for initialization */ +#define PRIME32 2654435761U +#define PRIME64 11400714785074694797ULL + +/* + * Fills a test buffer with pseudorandom data. + * + * This is used in the sanity check - its values must not be changed. + */ +XSUM_API void XSUM_fillTestBuffer(XSUM_U8* buffer, size_t len) +{ + XSUM_U64 byteGen = PRIME32; + size_t i; + + assert(buffer != NULL); + + for (i=0; i>56); + byteGen *= PRIME64; + } +} + + + +/* ************************************************ + * Self-test: + * ensure results consistency accross platforms + *********************************************** */ +#if XSUM_NO_TESTS +XSUM_API void XSUM_sanityCheck(void) +{ + XSUM_log("This version of xxhsum is not verified.\n"); +} +#else +static void XSUM_checkResult32(XXH32_hash_t r1, XXH32_hash_t r2) +{ + static int nbTests = 1; + if (r1!=r2) { + XSUM_log("\rError: 32-bit hash test %i: Internal sanity check failed!\n", nbTests); + XSUM_log("\rGot 0x%08X, expected 0x%08X.\n", (unsigned)r1, (unsigned)r2); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily recompile with XSUM_NO_TESTS=1.\n"); + exit(1); + } + nbTests++; +} + +static void XSUM_checkResult64(XXH64_hash_t r1, XXH64_hash_t r2) +{ + static int nbTests = 1; + if (r1!=r2) { + XSUM_log("\rError: 64-bit hash test %i: Internal sanity check failed!\n", nbTests); + XSUM_log("\rGot 0x%08X%08XULL, expected 0x%08X%08XULL.\n", + (unsigned)(r1>>32), (unsigned)r1, (unsigned)(r2>>32), (unsigned)r2); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily recompile with XSUM_NO_TESTS=1.\n"); + exit(1); + } + nbTests++; +} + +static void XSUM_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) +{ + static int nbTests = 1; + if ((r1.low64 != r2.low64) || (r1.high64 != r2.high64)) { + XSUM_log("\rError: 128-bit hash test %i: Internal sanity check failed.\n", nbTests); + XSUM_log("\rGot { 0x%08X%08XULL, 0x%08X%08XULL }, expected { 0x%08X%08XULL, 0x%08X%08XULL } \n", + (unsigned)(r1.low64>>32), (unsigned)r1.low64, (unsigned)(r1.high64>>32), (unsigned)r1.high64, + (unsigned)(r2.low64>>32), (unsigned)r2.low64, (unsigned)(r2.high64>>32), (unsigned)r2.high64 ); + XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" + "or temporarily recompile with XSUM_NO_TESTS=1.\n"); + exit(1); + } + nbTests++; +} + + +static void XSUM_testXXH32(const void* data, size_t len, XSUM_U32 seed, XSUM_U32 Nresult) +{ + XXH32_state_t *state = XXH32_createState(); + size_t pos; + + assert(state != NULL); + if (len>0) assert(data != NULL); + + XSUM_checkResult32(XXH32(data, len, seed), Nresult); + + (void)XXH32_reset(state, seed); + (void)XXH32_update(state, data, len); + XSUM_checkResult32(XXH32_digest(state), Nresult); + + (void)XXH32_reset(state, seed); + for (pos=0; pos0) assert(data != NULL); + + XSUM_checkResult64(XXH64(data, len, seed), Nresult); + + (void)XXH64_reset(state, seed); + (void)XXH64_update(state, data, len); + XSUM_checkResult64(XXH64_digest(state), Nresult); + + (void)XXH64_reset(state, seed); + for (pos=0; pos> 40); +} + + +static void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) +{ + if (len>0) assert(data != NULL); + + { XSUM_U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); + XSUM_checkResult64(Dresult, Nresult); + } + + /* check that the no-seed variant produces same result as seed==0 */ + if (seed == 0) { + XSUM_U64 const Dresult = XXH3_64bits(data, len); + XSUM_checkResult64(Dresult, Nresult); + } + + /* streaming API test */ + { XXH3_state_t* const state = XXH3_createState(); + assert(state != NULL); + /* single ingestion */ + (void)XXH3_64bits_reset_withSeed(state, seed); + (void)XXH3_64bits_update(state, data, len); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); + + /* random ingestion */ + { size_t p = 0; + (void)XXH3_64bits_reset_withSeed(state, seed); + while (p < len) { + size_t const modulo = len > 2 ? len : 2; + size_t l = (size_t)(XSUM_rand()) % modulo; + if (p + l > len) l = len - p; + (void)XXH3_64bits_update(state, (const char*)data+p, l); + p += l; + } + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); + } + + /* byte by byte ingestion */ + { size_t pos; + (void)XXH3_64bits_reset_withSeed(state, seed); + for (pos=0; pos0) assert(data != NULL); + + { XSUM_U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); + XSUM_checkResult64(Dresult, Nresult); + } + + /* streaming API test */ + { XXH3_state_t *state = XXH3_createState(); + assert(state != NULL); + (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); + (void)XXH3_64bits_update(state, data, len); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); + + /* random ingestion */ + { size_t p = 0; + (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); + while (p < len) { + size_t const modulo = len > 2 ? len : 2; + size_t l = (size_t)(XSUM_rand()) % modulo; + if (p + l > len) l = len - p; + (void)XXH3_64bits_update(state, (const char*)data+p, l); + p += l; + } + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); + } + + /* byte by byte ingestion */ + { size_t pos; + (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); + for (pos=0; pos 2 ? len : 2; + size_t l = (size_t)(XSUM_rand()) % modulo; + if (p + l > len) l = len - p; + (void)XXH3_128bits_update(state, (const char*)data+p, l); + p += l; + } + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); + } + + /* byte by byte ingestion */ + { size_t pos; + (void)XXH3_128bits_reset_withSeed(state, seed); + for (pos=0; pos0) assert(data != NULL); + + { XXH128_hash_t const Dresult = XXH3_128bits_withSecret(data, len, secret, secretSize); + XSUM_checkResult128(Dresult, Nresult); + } + + /* streaming API test */ + { XXH3_state_t* const state = XXH3_createState(); + assert(state != NULL); + (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); + (void)XXH3_128bits_update(state, data, len); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); + + /* random ingestion */ + { size_t p = 0; + (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); + while (p < len) { + size_t const modulo = len > 2 ? len : 2; + size_t l = (size_t)(XSUM_rand()) % modulo; + if (p + l > len) l = len - p; + (void)XXH3_128bits_update(state, (const char*)data+p, l); + p += l; + } + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); + } + + /* byte by byte ingestion */ + { size_t pos; + (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); + for (pos=0; pos= 7 + secretSize); + XSUM_testXXH3_withSecret(NULL, 0, secret, secretSize, 0x3559D64878C5C66CULL); /* empty string */ + XSUM_testXXH3_withSecret(sanityBuffer, 1, secret, secretSize, 0x8A52451418B2DA4DULL); /* 1 - 3 */ + XSUM_testXXH3_withSecret(sanityBuffer, 6, secret, secretSize, 0x82C90AB0519369ADULL); /* 4 - 8 */ + XSUM_testXXH3_withSecret(sanityBuffer, 12, secret, secretSize, 0x14631E773B78EC57ULL); /* 9 - 16 */ + XSUM_testXXH3_withSecret(sanityBuffer, 24, secret, secretSize, 0xCDD5542E4A9D9FE8ULL); /* 17 - 32 */ + XSUM_testXXH3_withSecret(sanityBuffer, 48, secret, secretSize, 0x33ABD54D094B2534ULL); /* 33 - 64 */ + XSUM_testXXH3_withSecret(sanityBuffer, 80, secret, secretSize, 0xE687BA1684965297ULL); /* 65 - 96 */ + XSUM_testXXH3_withSecret(sanityBuffer, 195, secret, secretSize, 0xA057273F5EECFB20ULL); /* 129-240 */ + + XSUM_testXXH3_withSecret(sanityBuffer, 403, secret, secretSize, 0x14546019124D43B8ULL); /* one block, last stripe is overlapping */ + XSUM_testXXH3_withSecret(sanityBuffer, 512, secret, secretSize, 0x7564693DD526E28DULL); /* one block, finishing at stripe boundary */ + XSUM_testXXH3_withSecret(sanityBuffer,2048, secret, secretSize, 0xD32E975821D6519FULL); /* >= 2 blocks, at least one scrambling */ + XSUM_testXXH3_withSecret(sanityBuffer,2367, secret, secretSize, 0x293FA8E5173BB5E7ULL); /* >= 2 blocks, at least one scrambling, last stripe unaligned */ + + XSUM_testXXH3_withSecret(sanityBuffer,64*10*3, secret, secretSize, 0x751D2EC54BC6038BULL); /* exactly 3 full blocks, not a multiple of 256 */ + } + + /* XXH128 */ + { XXH128_hash_t const expected = { 0x6001C324468D497FULL, 0x99AA06D3014798D8ULL }; + XSUM_testXXH128(NULL, 0, 0, expected); /* empty string */ + } + { XXH128_hash_t const expected = { 0x5444F7869C671AB0ULL, 0x92220AE55E14AB50ULL }; + XSUM_testXXH128(NULL, 0, PRIME32, expected); + } + { XXH128_hash_t const expected = { 0xC44BDFF4074EECDBULL, 0xA6CD5E9392000F6AULL }; + XSUM_testXXH128(sanityBuffer, 1, 0, expected); /* 1-3 */ + } + { XXH128_hash_t const expected = { 0xB53D5557E7F76F8DULL, 0x89B99554BA22467CULL }; + XSUM_testXXH128(sanityBuffer, 1, PRIME32, expected); /* 1-3 */ + } + { XXH128_hash_t const expected = { 0x3E7039BDDA43CFC6ULL, 0x082AFE0B8162D12AULL }; + XSUM_testXXH128(sanityBuffer, 6, 0, expected); /* 4-8 */ + } + { XXH128_hash_t const expected = { 0x269D8F70BE98856EULL, 0x5A865B5389ABD2B1ULL }; + XSUM_testXXH128(sanityBuffer, 6, PRIME32, expected); /* 4-8 */ + } + { XXH128_hash_t const expected = { 0x061A192713F69AD9ULL, 0x6E3EFD8FC7802B18ULL }; + XSUM_testXXH128(sanityBuffer, 12, 0, expected); /* 9-16 */ + } + { XXH128_hash_t const expected = { 0x9BE9F9A67F3C7DFBULL, 0xD7E09D518A3405D3ULL }; + XSUM_testXXH128(sanityBuffer, 12, PRIME32, expected); /* 9-16 */ + } + { XXH128_hash_t const expected = { 0x1E7044D28B1B901DULL, 0x0CE966E4678D3761ULL }; + XSUM_testXXH128(sanityBuffer, 24, 0, expected); /* 17-32 */ + } + { XXH128_hash_t const expected = { 0xD7304C54EBAD40A9ULL, 0x3162026714A6A243ULL }; + XSUM_testXXH128(sanityBuffer, 24, PRIME32, expected); /* 17-32 */ + } + { XXH128_hash_t const expected = { 0xF942219AED80F67BULL, 0xA002AC4E5478227EULL }; + XSUM_testXXH128(sanityBuffer, 48, 0, expected); /* 33-64 */ + } + { XXH128_hash_t const expected = { 0x7BA3C3E453A1934EULL, 0x163ADDE36C072295ULL }; + XSUM_testXXH128(sanityBuffer, 48, PRIME32, expected); /* 33-64 */ + } + { XXH128_hash_t const expected = { 0x5E8BAFB9F95FB803ULL, 0x4952F58181AB0042ULL }; + XSUM_testXXH128(sanityBuffer, 81, 0, expected); /* 65-96 */ + } + { XXH128_hash_t const expected = { 0x703FBB3D7A5F755CULL, 0x2724EC7ADC750FB6ULL }; + XSUM_testXXH128(sanityBuffer, 81, PRIME32, expected); /* 65-96 */ + } + { XXH128_hash_t const expected = { 0xF1AEBD597CEC6B3AULL, 0x337E09641B948717ULL }; + XSUM_testXXH128(sanityBuffer, 222, 0, expected); /* 129-240 */ + } + { XXH128_hash_t const expected = { 0xAE995BB8AF917A8DULL, 0x91820016621E97F1ULL }; + XSUM_testXXH128(sanityBuffer, 222, PRIME32, expected); /* 129-240 */ + } + { XXH128_hash_t const expected = { 0xCDEB804D65C6DEA4ULL, 0x1B6DE21E332DD73DULL }; + XSUM_testXXH128(sanityBuffer, 403, 0, expected); /* one block, last stripe is overlapping */ + } + { XXH128_hash_t const expected = { 0x6259F6ECFD6443FDULL, 0xBED311971E0BE8F2ULL }; + XSUM_testXXH128(sanityBuffer, 403, PRIME64, expected); /* one block, last stripe is overlapping */ + } + { XXH128_hash_t const expected = { 0x617E49599013CB6BULL, 0x18D2D110DCC9BCA1ULL }; + XSUM_testXXH128(sanityBuffer, 512, 0, expected); /* one block, finishing at stripe boundary */ + } + { XXH128_hash_t const expected = { 0x3CE457DE14C27708ULL, 0x925D06B8EC5B8040ULL }; + XSUM_testXXH128(sanityBuffer, 512, PRIME64, expected); /* one block, finishing at stripe boundary */ + } + { XXH128_hash_t const expected = { 0xDD59E2C3A5F038E0ULL, 0xF736557FD47073A5ULL }; + XSUM_testXXH128(sanityBuffer,2048, 0, expected); /* two blocks, finishing at block boundary */ + } + { XXH128_hash_t const expected = { 0x230D43F30206260BULL, 0x7FB03F7E7186C3EAULL }; + XSUM_testXXH128(sanityBuffer,2048, PRIME32, expected); /* two blocks, finishing at block boundary */ + } + { XXH128_hash_t const expected = { 0x6E73A90539CF2948ULL, 0xCCB134FBFA7CE49DULL }; + XSUM_testXXH128(sanityBuffer,2240, 0, expected); /* two blocks, ends at stripe boundary */ + } + { XXH128_hash_t const expected = { 0xED385111126FBA6FULL, 0x50A1FE17B338995FULL }; + XSUM_testXXH128(sanityBuffer,2240, PRIME32, expected); /* two blocks, ends at stripe boundary */ + } + { XXH128_hash_t const expected = { 0xCB37AEB9E5D361EDULL, 0xE89C0F6FF369B427ULL }; + XSUM_testXXH128(sanityBuffer,2367, 0, expected); /* two blocks, last stripe is overlapping */ + } + { XXH128_hash_t const expected = { 0x6F5360AE69C2F406ULL, 0xD23AAE4B76C31ECBULL }; + XSUM_testXXH128(sanityBuffer,2367, PRIME32, expected); /* two blocks, last stripe is overlapping */ + } + + /* XXH128 with custom Secret */ + { const void* const secret = sanityBuffer + 7; + const size_t secretSize = XXH3_SECRET_SIZE_MIN + 11; + assert(sizeof(sanityBuffer) >= 7 + secretSize); + + { XXH128_hash_t const expected = { 0x005923CCEECBE8AEULL, 0x5F70F4EA232F1D38ULL }; + XSUM_testXXH128_withSecret(NULL, 0, secret, secretSize, expected); /* empty string */ + } + { XXH128_hash_t const expected = { 0x8A52451418B2DA4DULL, 0x3A66AF5A9819198EULL }; + XSUM_testXXH128_withSecret(sanityBuffer, 1, secret, secretSize, expected); /* 1-3 */ + } + { XXH128_hash_t const expected = { 0x0B61C8ACA7D4778FULL, 0x376BD91B6432F36DULL }; + XSUM_testXXH128_withSecret(sanityBuffer, 6, secret, secretSize, expected); /* 4-8 */ + } + { XXH128_hash_t const expected = { 0xAF82F6EBA263D7D8ULL, 0x90A3C2D839F57D0FULL }; + XSUM_testXXH128_withSecret(sanityBuffer, 12, secret, secretSize, expected); /* 9-16 */ + } + } + + /* secret generator */ + { verifSample_t const expected = { { 0xB8, 0x26, 0x83, 0x7E } }; + XSUM_testSecretGenerator(NULL, 0, expected); + } + + { verifSample_t const expected = { { 0xA6, 0x16, 0x06, 0x7B } }; + XSUM_testSecretGenerator(sanityBuffer, 1, expected); + } + + { verifSample_t const expected = { { 0xDA, 0x2A, 0x12, 0x11 } }; + XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_SIZE_MIN - 1, expected); + } + + { verifSample_t const expected = { { 0x7E, 0x48, 0x0C, 0xA7 } }; + XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_DEFAULT_SIZE + 500, expected); + } + + XSUM_logVerbose(3, "\r%70s\r", ""); /* Clean display line */ + XSUM_logVerbose(3, "Sanity check -- all tests ok\n"); +} + +#endif /* !XSUM_NO_TESTS */ diff --git a/cli/xsum_sanity_check.h b/cli/xsum_sanity_check.h new file mode 100644 index 00000000..a3f57a16 --- /dev/null +++ b/cli/xsum_sanity_check.h @@ -0,0 +1,57 @@ +/* + * xxhsum - Command line interface for xxhash algorithms + * Copyright (C) 2013-2020 Yann Collet + * + * GPL v2 License + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License along + * with this program; if not, write to the Free Software Foundation, Inc., + * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + * + * You can contact the author at: + * - xxHash homepage: https://www.xxhash.com + * - xxHash source repository: https://github.com/Cyan4973/xxHash + */ + +#ifndef XSUM_SANITY_CHECK_H +#define XSUM_SANITY_CHECK_H + +#include "xsum_config.h" + +#ifdef __cplusplus +extern "C" { +#endif + +/* + * Runs a series of self-tests. + * + * Exits if any of these tests fail, printing a message to stderr. + * + * If XSUM_NO_TESTS is defined to non-zero, this will instead print a warning + * if this is called (e.g. via xxhsum -b). + */ +XSUM_API void XSUM_sanityCheck(void); + +/* + * Fills a test buffer with pseudorandom data. + * + * This is used in the sanity check and the benchmarks - its values must not be + * changed. + */ +XSUM_API void XSUM_fillTestBuffer(XSUM_U8* buffer, size_t len); + +#ifdef __cplusplus +} +#endif + +#endif /* XSUM_SANITY_CHECK_H */ diff --git a/cmake_unofficial/CMakeLists.txt b/cmake_unofficial/CMakeLists.txt index 5abd0c5f..cd38be4b 100644 --- a/cmake_unofficial/CMakeLists.txt +++ b/cmake_unofficial/CMakeLists.txt @@ -87,6 +87,7 @@ if(XXHASH_BUILD_XXHSUM) add_executable(xxhsum "${XXHASH_DIR}/xxhsum.c" "${XXHSUM_DIR}/xsum_os_specific.c" "${XXHSUM_DIR}/xsum_output.c" + "${XXHSUM_DIR}/xsum_sanity_check.c" ) add_executable(${PROJECT_NAME}::xxhsum ALIAS xxhsum) diff --git a/xxhsum.c b/xxhsum.c index bac55b20..82324091 100644 --- a/xxhsum.c +++ b/xxhsum.c @@ -34,9 +34,11 @@ #include "cli/xsum_arch.h" #include "cli/xsum_os_specific.h" #include "cli/xsum_output.h" +#include "cli/xsum_sanity_check.h" #ifdef XXH_INLINE_ALL # include "cli/xsum_os_specific.c" # include "cli/xsum_output.c" +# include "cli/xsum_sanity_check.c" #endif /* ************************************ @@ -166,28 +168,6 @@ static char* XSUM_strcatDup(const char* s1, const char* s2) } -/* use #define to make them constant, required for initialization */ -#define PRIME32 2654435761U -#define PRIME64 11400714785074694797ULL - -/* - * Fills a test buffer with pseudorandom data. - * - * This is used in the sanity check - its values must not be changed. - */ -static void XSUM_fillTestBuffer(XSUM_U8* buffer, size_t len) -{ - XSUM_U64 byteGen = PRIME32; - size_t i; - - assert(buffer != NULL); - - for (i=0; i>56); - byteGen *= PRIME64; - } -} - /* * A secret buffer used for benchmarking XXH3's withSecret variants. * @@ -506,506 +486,6 @@ static int XSUM_benchInternal(size_t keySize) return 0; } - -/* ************************************************ - * Self-test: - * ensure results consistency accross platforms - *********************************************** */ - -static void XSUM_checkResult32(XXH32_hash_t r1, XXH32_hash_t r2) -{ - static int nbTests = 1; - if (r1!=r2) { - XSUM_log("\rError: 32-bit hash test %i: Internal sanity check failed!\n", nbTests); - XSUM_log("\rGot 0x%08X, expected 0x%08X.\n", (unsigned)r1, (unsigned)r2); - XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in XSUM_sanityCheck.\n"); - exit(1); - } - nbTests++; -} - -static void XSUM_checkResult64(XXH64_hash_t r1, XXH64_hash_t r2) -{ - static int nbTests = 1; - if (r1!=r2) { - XSUM_log("\rError: 64-bit hash test %i: Internal sanity check failed!\n", nbTests); - XSUM_log("\rGot 0x%08X%08XULL, expected 0x%08X%08XULL.\n", - (unsigned)(r1>>32), (unsigned)r1, (unsigned)(r2>>32), (unsigned)r2); - XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in XSUM_sanityCheck.\n"); - exit(1); - } - nbTests++; -} - -static void XSUM_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) -{ - static int nbTests = 1; - if ((r1.low64 != r2.low64) || (r1.high64 != r2.high64)) { - XSUM_log("\rError: 128-bit hash test %i: Internal sanity check failed.\n", nbTests); - XSUM_log("\rGot { 0x%08X%08XULL, 0x%08X%08XULL }, expected { 0x%08X%08XULL, 0x%08X%08XULL } \n", - (unsigned)(r1.low64>>32), (unsigned)r1.low64, (unsigned)(r1.high64>>32), (unsigned)r1.high64, - (unsigned)(r2.low64>>32), (unsigned)r2.low64, (unsigned)(r2.high64>>32), (unsigned)r2.high64 ); - XSUM_log("\rNote: If you modified the hash functions, make sure to either update the values\n" - "or temporarily comment out the tests in XSUM_sanityCheck.\n"); - exit(1); - } - nbTests++; -} - - -static void XSUM_testXXH32(const void* data, size_t len, XSUM_U32 seed, XSUM_U32 Nresult) -{ - XXH32_state_t *state = XXH32_createState(); - size_t pos; - - assert(state != NULL); - if (len>0) assert(data != NULL); - - XSUM_checkResult32(XXH32(data, len, seed), Nresult); - - (void)XXH32_reset(state, seed); - (void)XXH32_update(state, data, len); - XSUM_checkResult32(XXH32_digest(state), Nresult); - - (void)XXH32_reset(state, seed); - for (pos=0; pos0) assert(data != NULL); - - XSUM_checkResult64(XXH64(data, len, seed), Nresult); - - (void)XXH64_reset(state, seed); - (void)XXH64_update(state, data, len); - XSUM_checkResult64(XXH64_digest(state), Nresult); - - (void)XXH64_reset(state, seed); - for (pos=0; pos> 40); -} - - -void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) -{ - if (len>0) assert(data != NULL); - - { XSUM_U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); - XSUM_checkResult64(Dresult, Nresult); - } - - /* check that the no-seed variant produces same result as seed==0 */ - if (seed == 0) { - XSUM_U64 const Dresult = XXH3_64bits(data, len); - XSUM_checkResult64(Dresult, Nresult); - } - - /* streaming API test */ - { XXH3_state_t* const state = XXH3_createState(); - assert(state != NULL); - /* single ingestion */ - (void)XXH3_64bits_reset_withSeed(state, seed); - (void)XXH3_64bits_update(state, data, len); - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - - /* random ingestion */ - { size_t p = 0; - (void)XXH3_64bits_reset_withSeed(state, seed); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_64bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - } - - /* byte by byte ingestion */ - { size_t pos; - (void)XXH3_64bits_reset_withSeed(state, seed); - for (pos=0; pos0) assert(data != NULL); - - { XSUM_U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); - XSUM_checkResult64(Dresult, Nresult); - } - - /* streaming API test */ - { XXH3_state_t *state = XXH3_createState(); - assert(state != NULL); - (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); - (void)XXH3_64bits_update(state, data, len); - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - - /* random ingestion */ - { size_t p = 0; - (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_64bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - } - - /* byte by byte ingestion */ - { size_t pos; - (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); - for (pos=0; pos 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_128bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); - } - - /* byte by byte ingestion */ - { size_t pos; - (void)XXH3_128bits_reset_withSeed(state, seed); - for (pos=0; pos0) assert(data != NULL); - - { XXH128_hash_t const Dresult = XXH3_128bits_withSecret(data, len, secret, secretSize); - XSUM_checkResult128(Dresult, Nresult); - } - - /* streaming API test */ - { XXH3_state_t* const state = XXH3_createState(); - assert(state != NULL); - (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); - (void)XXH3_128bits_update(state, data, len); - XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); - - /* random ingestion */ - { size_t p = 0; - (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_128bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); - } - - /* byte by byte ingestion */ - { size_t pos; - (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); - for (pos=0; pos= 7 + secretSize); - XSUM_testXXH3_withSecret(NULL, 0, secret, secretSize, 0x3559D64878C5C66CULL); /* empty string */ - XSUM_testXXH3_withSecret(sanityBuffer, 1, secret, secretSize, 0x8A52451418B2DA4DULL); /* 1 - 3 */ - XSUM_testXXH3_withSecret(sanityBuffer, 6, secret, secretSize, 0x82C90AB0519369ADULL); /* 4 - 8 */ - XSUM_testXXH3_withSecret(sanityBuffer, 12, secret, secretSize, 0x14631E773B78EC57ULL); /* 9 - 16 */ - XSUM_testXXH3_withSecret(sanityBuffer, 24, secret, secretSize, 0xCDD5542E4A9D9FE8ULL); /* 17 - 32 */ - XSUM_testXXH3_withSecret(sanityBuffer, 48, secret, secretSize, 0x33ABD54D094B2534ULL); /* 33 - 64 */ - XSUM_testXXH3_withSecret(sanityBuffer, 80, secret, secretSize, 0xE687BA1684965297ULL); /* 65 - 96 */ - XSUM_testXXH3_withSecret(sanityBuffer, 195, secret, secretSize, 0xA057273F5EECFB20ULL); /* 129-240 */ - - XSUM_testXXH3_withSecret(sanityBuffer, 403, secret, secretSize, 0x14546019124D43B8ULL); /* one block, last stripe is overlapping */ - XSUM_testXXH3_withSecret(sanityBuffer, 512, secret, secretSize, 0x7564693DD526E28DULL); /* one block, finishing at stripe boundary */ - XSUM_testXXH3_withSecret(sanityBuffer,2048, secret, secretSize, 0xD32E975821D6519FULL); /* >= 2 blocks, at least one scrambling */ - XSUM_testXXH3_withSecret(sanityBuffer,2367, secret, secretSize, 0x293FA8E5173BB5E7ULL); /* >= 2 blocks, at least one scrambling, last stripe unaligned */ - - XSUM_testXXH3_withSecret(sanityBuffer,64*10*3, secret, secretSize, 0x751D2EC54BC6038BULL); /* exactly 3 full blocks, not a multiple of 256 */ - } - - /* XXH128 */ - { XXH128_hash_t const expected = { 0x6001C324468D497FULL, 0x99AA06D3014798D8ULL }; - XSUM_testXXH128(NULL, 0, 0, expected); /* empty string */ - } - { XXH128_hash_t const expected = { 0x5444F7869C671AB0ULL, 0x92220AE55E14AB50ULL }; - XSUM_testXXH128(NULL, 0, PRIME32, expected); - } - { XXH128_hash_t const expected = { 0xC44BDFF4074EECDBULL, 0xA6CD5E9392000F6AULL }; - XSUM_testXXH128(sanityBuffer, 1, 0, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0xB53D5557E7F76F8DULL, 0x89B99554BA22467CULL }; - XSUM_testXXH128(sanityBuffer, 1, PRIME32, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0x3E7039BDDA43CFC6ULL, 0x082AFE0B8162D12AULL }; - XSUM_testXXH128(sanityBuffer, 6, 0, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0x269D8F70BE98856EULL, 0x5A865B5389ABD2B1ULL }; - XSUM_testXXH128(sanityBuffer, 6, PRIME32, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0x061A192713F69AD9ULL, 0x6E3EFD8FC7802B18ULL }; - XSUM_testXXH128(sanityBuffer, 12, 0, expected); /* 9-16 */ - } - { XXH128_hash_t const expected = { 0x9BE9F9A67F3C7DFBULL, 0xD7E09D518A3405D3ULL }; - XSUM_testXXH128(sanityBuffer, 12, PRIME32, expected); /* 9-16 */ - } - { XXH128_hash_t const expected = { 0x1E7044D28B1B901DULL, 0x0CE966E4678D3761ULL }; - XSUM_testXXH128(sanityBuffer, 24, 0, expected); /* 17-32 */ - } - { XXH128_hash_t const expected = { 0xD7304C54EBAD40A9ULL, 0x3162026714A6A243ULL }; - XSUM_testXXH128(sanityBuffer, 24, PRIME32, expected); /* 17-32 */ - } - { XXH128_hash_t const expected = { 0xF942219AED80F67BULL, 0xA002AC4E5478227EULL }; - XSUM_testXXH128(sanityBuffer, 48, 0, expected); /* 33-64 */ - } - { XXH128_hash_t const expected = { 0x7BA3C3E453A1934EULL, 0x163ADDE36C072295ULL }; - XSUM_testXXH128(sanityBuffer, 48, PRIME32, expected); /* 33-64 */ - } - { XXH128_hash_t const expected = { 0x5E8BAFB9F95FB803ULL, 0x4952F58181AB0042ULL }; - XSUM_testXXH128(sanityBuffer, 81, 0, expected); /* 65-96 */ - } - { XXH128_hash_t const expected = { 0x703FBB3D7A5F755CULL, 0x2724EC7ADC750FB6ULL }; - XSUM_testXXH128(sanityBuffer, 81, PRIME32, expected); /* 65-96 */ - } - { XXH128_hash_t const expected = { 0xF1AEBD597CEC6B3AULL, 0x337E09641B948717ULL }; - XSUM_testXXH128(sanityBuffer, 222, 0, expected); /* 129-240 */ - } - { XXH128_hash_t const expected = { 0xAE995BB8AF917A8DULL, 0x91820016621E97F1ULL }; - XSUM_testXXH128(sanityBuffer, 222, PRIME32, expected); /* 129-240 */ - } - { XXH128_hash_t const expected = { 0xCDEB804D65C6DEA4ULL, 0x1B6DE21E332DD73DULL }; - XSUM_testXXH128(sanityBuffer, 403, 0, expected); /* one block, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x6259F6ECFD6443FDULL, 0xBED311971E0BE8F2ULL }; - XSUM_testXXH128(sanityBuffer, 403, PRIME64, expected); /* one block, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x617E49599013CB6BULL, 0x18D2D110DCC9BCA1ULL }; - XSUM_testXXH128(sanityBuffer, 512, 0, expected); /* one block, finishing at stripe boundary */ - } - { XXH128_hash_t const expected = { 0x3CE457DE14C27708ULL, 0x925D06B8EC5B8040ULL }; - XSUM_testXXH128(sanityBuffer, 512, PRIME64, expected); /* one block, finishing at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xDD59E2C3A5F038E0ULL, 0xF736557FD47073A5ULL }; - XSUM_testXXH128(sanityBuffer,2048, 0, expected); /* two blocks, finishing at block boundary */ - } - { XXH128_hash_t const expected = { 0x230D43F30206260BULL, 0x7FB03F7E7186C3EAULL }; - XSUM_testXXH128(sanityBuffer,2048, PRIME32, expected); /* two blocks, finishing at block boundary */ - } - { XXH128_hash_t const expected = { 0x6E73A90539CF2948ULL, 0xCCB134FBFA7CE49DULL }; - XSUM_testXXH128(sanityBuffer,2240, 0, expected); /* two blocks, ends at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xED385111126FBA6FULL, 0x50A1FE17B338995FULL }; - XSUM_testXXH128(sanityBuffer,2240, PRIME32, expected); /* two blocks, ends at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xCB37AEB9E5D361EDULL, 0xE89C0F6FF369B427ULL }; - XSUM_testXXH128(sanityBuffer,2367, 0, expected); /* two blocks, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x6F5360AE69C2F406ULL, 0xD23AAE4B76C31ECBULL }; - XSUM_testXXH128(sanityBuffer,2367, PRIME32, expected); /* two blocks, last stripe is overlapping */ - } - - /* XXH128 with custom Secret */ - { const void* const secret = sanityBuffer + 7; - const size_t secretSize = XXH3_SECRET_SIZE_MIN + 11; - assert(sizeof(sanityBuffer) >= 7 + secretSize); - - { XXH128_hash_t const expected = { 0x005923CCEECBE8AEULL, 0x5F70F4EA232F1D38ULL }; - XSUM_testXXH128_withSecret(NULL, 0, secret, secretSize, expected); /* empty string */ - } - { XXH128_hash_t const expected = { 0x8A52451418B2DA4DULL, 0x3A66AF5A9819198EULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 1, secret, secretSize, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0x0B61C8ACA7D4778FULL, 0x376BD91B6432F36DULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 6, secret, secretSize, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0xAF82F6EBA263D7D8ULL, 0x90A3C2D839F57D0FULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 12, secret, secretSize, expected); /* 9-16 */ - } - } - - /* secret generator */ - { verifSample_t const expected = { { 0xB8, 0x26, 0x83, 0x7E } }; - XSUM_testSecretGenerator(NULL, 0, expected); - } - - { verifSample_t const expected = { { 0xA6, 0x16, 0x06, 0x7B } }; - XSUM_testSecretGenerator(sanityBuffer, 1, expected); - } - - { verifSample_t const expected = { { 0xDA, 0x2A, 0x12, 0x11 } }; - XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_SIZE_MIN - 1, expected); - } - - { verifSample_t const expected = { { 0x7E, 0x48, 0x0C, 0xA7 } }; - XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_DEFAULT_SIZE + 500, expected); - } - - XSUM_logVerbose(3, "\r%70s\r", ""); /* Clean display line */ - XSUM_logVerbose(3, "Sanity check -- all tests ok\n"); -} - - /* ******************************************************** * File Hashing **********************************************************/ From 684812267dcaf99d8571c00bd5a215eca0c4daaa Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Sun, 20 Sep 2020 14:16:06 -0400 Subject: [PATCH 024/187] xsum_sanity_check.c: Refactor to use tables Refactored xsum_sanity_check.c to use a table instead of direct function calls. This is cleaner and has smaller code size. Additionally, did some cleanup to reduce repetition a bit. There is still a lot of DRY that could be applied to this file. --- cli/xsum_sanity_check.c | 520 +++++++++++++++++++++------------------- 1 file changed, 279 insertions(+), 241 deletions(-) diff --git a/cli/xsum_sanity_check.c b/cli/xsum_sanity_check.c index 6d9c6c13..347d1db5 100644 --- a/cli/xsum_sanity_check.c +++ b/cli/xsum_sanity_check.c @@ -68,6 +68,158 @@ XSUM_API void XSUM_sanityCheck(void) XSUM_log("This version of xxhsum is not verified.\n"); } #else + +/* + * Test data vectors + */ +typedef struct { + XSUM_U32 len; + XSUM_U32 seed; + XSUM_U32 Nresult; +} XSUM_testdata32_t; + +typedef struct { + XSUM_U32 len; + XSUM_U64 seed; + XSUM_U64 Nresult; +} XSUM_testdata64_t; + +typedef struct { + XSUM_U32 len; + XSUM_U64 seed; + XXH128_hash_t Nresult; +} XSUM_testdata128_t; + +#define SECRET_SAMPLE_NBBYTES 4 +typedef struct { + XSUM_U32 len; + XSUM_U8 byte[SECRET_SAMPLE_NBBYTES]; +} XSUM_testdata_sample_t; + +/* XXH32 */ +static const XSUM_testdata32_t XSUM_XXH32_testdata[] = { + { 0, 0, 0x02CC5D05U }, + { 0, PRIME32, 0x36B78AE7U }, + { 1, 0, 0xCF65B03EU }, + { 1, PRIME32, 0xB4545AA4U }, + { 14, 0, 0x1208E7E2U }, + { 14, PRIME32, 0x6AF1D1FEU }, + { 222, 0, 0x5BD11DBDU }, + { 222, PRIME32, 0x58803C5FU } +}; + +/* XXH64 */ +static const XSUM_testdata64_t XSUM_XXH64_testdata[] = { + { 0, 0, 0xEF46DB3751D8E999ULL }, + { 0, PRIME32, 0xAC75FDA2929B17EFULL }, + { 1, 0, 0xE934A84ADB052768ULL }, + { 1, PRIME32, 0x5014607643A9B4C3ULL }, + { 4, 0, 0x9136A0DCA57457EEULL }, + { 14, 0, 0x8282DCC4994E35C8ULL }, + { 14, PRIME32, 0xC3BD6BF63DEB6DF0ULL }, + { 222, 0, 0xB641AE8CB691C174ULL }, + { 222, PRIME32, 0x20CB8AB7AE10C14AULL } +}; +/* + * XXH3: + * Due to being a more complex hash function with specializations for certain + * lengths, a more extensive test is used for XXH3. + */ + +/* XXH3_64bits, seeded */ +static const XSUM_testdata64_t XSUM_XXH3_testdata[] = { + { 0, 0, 0x2D06800538D394C2ULL }, /* empty string */ + { 0, PRIME64, 0xA8A6B918B2F0364AULL }, /* empty string */ + { 1, 0, 0xC44BDFF4074EECDBULL }, /* 1 - 3 */ + { 1, PRIME64, 0x032BE332DD766EF8ULL }, /* 1 - 3 */ + { 6, 0, 0x27B56A84CD2D7325ULL }, /* 4 - 8 */ + { 6, PRIME64, 0x84589C116AB59AB9ULL }, /* 4 - 8 */ + { 12, 0, 0xA713DAF0DFBB77E7ULL }, /* 9 - 16 */ + { 12, PRIME64, 0xE7303E1B2336DE0EULL }, /* 9 - 16 */ + { 24, 0, 0xA3FE70BF9D3510EBULL }, /* 17 - 32 */ + { 24, PRIME64, 0x850E80FC35BDD690ULL }, /* 17 - 32 */ + { 48, 0, 0x397DA259ECBA1F11ULL }, /* 33 - 64 */ + { 48, PRIME64, 0xADC2CBAA44ACC616ULL }, /* 33 - 64 */ + { 80, 0, 0xBCDEFBBB2C47C90AULL }, /* 65 - 96 */ + { 80, PRIME64, 0xC6DD0CB699532E73ULL }, /* 65 - 96 */ + { 195, 0, 0xCD94217EE362EC3AULL }, /* 129-240 */ + { 195, PRIME64, 0xBA68003D370CB3D9ULL }, /* 129-240 */ + + { 403, 0, 0xCDEB804D65C6DEA4ULL }, /* one block, last stripe is overlapping */ + { 403, PRIME64, 0x6259F6ECFD6443FDULL }, /* one block, last stripe is overlapping */ + { 512, 0, 0x617E49599013CB6BULL }, /* one block, finishing at stripe boundary */ + { 512, PRIME64, 0x3CE457DE14C27708ULL }, /* one block, finishing at stripe boundary */ + { 2048, 0, 0xDD59E2C3A5F038E0ULL }, /* 2 blocks, finishing at block boundary */ + { 2048, PRIME64, 0x66F81670669ABABCULL }, /* 2 blocks, finishing at block boundary */ + { 2240, 0, 0x6E73A90539CF2948ULL }, /* 3 blocks, finishing at stripe boundary */ + { 2240, PRIME64, 0x757BA8487D1B5247ULL }, /* 3 blocks, finishing at stripe boundary */ + { 2367, 0, 0xCB37AEB9E5D361EDULL }, /* 3 blocks, last stripe is overlapping */ + { 2367, PRIME64, 0xD2DB3415B942B42AULL } /* 3 blocks, last stripe is overlapping */ +}; +/* XXH3_64bits, custom secret */ +static const XSUM_testdata64_t XSUM_XXH3_withSecret_testdata[] = { + { 0, 0, 0x3559D64878C5C66CULL }, /* empty string */ + { 1, 0, 0x8A52451418B2DA4DULL }, /* 1 - 3 */ + { 6, 0, 0x82C90AB0519369ADULL }, /* 4 - 8 */ + { 12, 0, 0x14631E773B78EC57ULL }, /* 9 - 16 */ + { 24, 0, 0xCDD5542E4A9D9FE8ULL }, /* 17 - 32 */ + { 48, 0, 0x33ABD54D094B2534ULL }, /* 33 - 64 */ + { 80, 0, 0xE687BA1684965297ULL }, /* 65 - 96 */ + { 195, 0, 0xA057273F5EECFB20ULL }, /* 129-240 */ + + { 403, 0, 0x14546019124D43B8ULL }, /* one block, last stripe is overlapping */ + { 512, 0, 0x7564693DD526E28DULL }, /* one block, finishing at stripe boundary */ + { 2048, 0, 0xD32E975821D6519FULL }, /* >= 2 blodcks, at least one scrambling */ + { 2367, 0, 0x293FA8E5173BB5E7ULL }, /* >= 2 blocks, at least one scrambling, last stripe unaligned */ + + { 64*10*3, 0, 0x751D2EC54BC6038BULL } /* exactly 3 full blocks, not a multiple of 256 */ +}; +/* XXH3_128bits, seeded */ +static const XSUM_testdata128_t XSUM_XXH128_testdata[] = { + { 0, 0, { 0x6001C324468D497FULL, 0x99AA06D3014798D8ULL } }, /* empty string */ + { 0, PRIME32, { 0x5444F7869C671AB0ULL, 0x92220AE55E14AB50ULL } }, /* empty string */ + { 1, 0, { 0xC44BDFF4074EECDBULL, 0xA6CD5E9392000F6AULL } }, /* 1 - 3 */ + { 1, PRIME32, { 0xB53D5557E7F76F8DULL, 0x89B99554BA22467CULL } }, /* 1 - 3 */ + { 6, 0, { 0x3E7039BDDA43CFC6ULL, 0x082AFE0B8162D12AULL } }, /* 4 - 8 */ + { 6, PRIME32, { 0x269D8F70BE98856EULL, 0x5A865B5389ABD2B1ULL } }, /* 4 - 8 */ + { 12, 0, { 0x061A192713F69AD9ULL, 0x6E3EFD8FC7802B18ULL } }, /* 9 - 16 */ + { 12, PRIME32, { 0x9BE9F9A67F3C7DFBULL, 0xD7E09D518A3405D3ULL } }, /* 9 - 16 */ + { 24, 0, { 0x1E7044D28B1B901DULL, 0x0CE966E4678D3761ULL } }, /* 17 - 32 */ + { 24, PRIME32, { 0xD7304C54EBAD40A9ULL, 0x3162026714A6A243ULL } }, /* 17 - 32 */ + { 48, 0, { 0xF942219AED80F67BULL, 0xA002AC4E5478227EULL } }, /* 33 - 64 */ + { 48, PRIME32, { 0x7BA3C3E453A1934EULL, 0x163ADDE36C072295ULL } }, /* 33 - 64 */ + { 81, 0, { 0x5E8BAFB9F95FB803ULL, 0x4952F58181AB0042ULL } }, /* 65 - 96 */ + { 81, PRIME32, { 0x703FBB3D7A5F755CULL, 0x2724EC7ADC750FB6ULL } }, /* 65 - 96 */ + { 222, 0, { 0xF1AEBD597CEC6B3AULL, 0x337E09641B948717ULL } }, /* 129-240 */ + { 222, PRIME32, { 0xAE995BB8AF917A8DULL, 0x91820016621E97F1ULL } }, /* 129-240 */ + + { 403, 0, { 0xCDEB804D65C6DEA4ULL, 0x1B6DE21E332DD73DULL } }, /* one block, last stripe is overlapping */ + { 403, PRIME64, { 0x6259F6ECFD6443FDULL, 0xBED311971E0BE8F2ULL } }, /* one block, last stripe is overlapping */ + { 512, 0, { 0x617E49599013CB6BULL, 0x18D2D110DCC9BCA1ULL } }, /* one block, finishing at stripe boundary */ + { 512, PRIME64, { 0x3CE457DE14C27708ULL, 0x925D06B8EC5B8040ULL } }, /* one block, finishing at stripe boundary */ + { 2048, 0, { 0xDD59E2C3A5F038E0ULL, 0xF736557FD47073A5ULL } }, /* 2 blocks, finishing at block boundary */ + { 2048, PRIME32, { 0x230D43F30206260BULL, 0x7FB03F7E7186C3EAULL } }, /* 2 blocks, finishing at block boundary */ + { 2240, 0, { 0x6E73A90539CF2948ULL, 0xCCB134FBFA7CE49DULL } }, /* 3 blocks, finishing at stripe boundary */ + { 2240, PRIME32, { 0xED385111126FBA6FULL, 0x50A1FE17B338995FULL } }, /* 3 blocks, finishing at stripe boundary */ + { 2367, 0, { 0xCB37AEB9E5D361EDULL, 0xE89C0F6FF369B427ULL } }, /* 3 blocks, last stripe is overlapping */ + { 2367, PRIME32, { 0x6F5360AE69C2F406ULL, 0xD23AAE4B76C31ECBULL } } /* 3 blocks, last stripe is overlapping */ +}; + +/* XXH128, custom secret */ +static const XSUM_testdata128_t XSUM_XXH128_withSecret_testdata[] = { + { 0, 0, { 0x005923CCEECBE8AEULL, 0x5F70F4EA232F1D38ULL } }, /* empty string */ + { 1, 0, { 0x8A52451418B2DA4DULL, 0x3A66AF5A9819198EULL } }, /* 1 - 3 */ + { 6, 0, { 0x0B61C8ACA7D4778FULL, 0x376BD91B6432F36DULL } }, /* 4 - 8 */ + { 12, 0, { 0xAF82F6EBA263D7D8ULL, 0x90A3C2D839F57D0FULL } } /* 9 - 16 */ +}; + +static const XSUM_testdata_sample_t XSUM_XXH3_generateSecret_testdata[] = { + { 0, { 0xB8, 0x26, 0x83, 0x7E } }, + { 1, { 0xA6, 0x16, 0x06, 0x7B } }, + { XXH3_SECRET_SIZE_MIN - 1, { 0xDA, 0x2A, 0x12, 0x11 } }, + { XXH3_SECRET_DEFAULT_SIZE + 500, { 0x7E, 0x48, 0x0C, 0xA7 } } +}; + static void XSUM_checkResult32(XXH32_hash_t r1, XXH32_hash_t r2) { static int nbTests = 1; @@ -111,13 +263,22 @@ static void XSUM_checkResult128(XXH128_hash_t r1, XXH128_hash_t r2) } -static void XSUM_testXXH32(const void* data, size_t len, XSUM_U32 seed, XSUM_U32 Nresult) +static void XSUM_testXXH32(const void* data, const XSUM_testdata32_t* testData) { XXH32_state_t *state = XXH32_createState(); size_t pos; + size_t len = testData->len; + XSUM_U32 seed = testData->seed; + XSUM_U32 Nresult = testData->Nresult; + + if (len == 0) { + data = NULL; + } else { + assert(data != NULL); + } + assert(state != NULL); - if (len>0) assert(data != NULL); XSUM_checkResult32(XXH32(data, len, seed), Nresult); @@ -132,13 +293,22 @@ static void XSUM_testXXH32(const void* data, size_t len, XSUM_U32 seed, XSUM_U32 XXH32_freeState(state); } -static void XSUM_testXXH64(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) +static void XSUM_testXXH64(const void* data, const XSUM_testdata64_t* testData) { XXH64_state_t *state = XXH64_createState(); size_t pos; + size_t len = (size_t)testData->len; + XSUM_U64 seed = testData->seed; + XSUM_U64 Nresult = testData->Nresult; + + if (len == 0) { + data = NULL; + } else { + assert(data != NULL); + } + assert(state != NULL); - if (len>0) assert(data != NULL); XSUM_checkResult64(XXH64(data, len, seed), Nresult); @@ -153,6 +323,10 @@ static void XSUM_testXXH64(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 XXH64_freeState(state); } +/* + * Used to get "random" (but actually 100% reproducible) lengths for + * XSUM_XXH3_randomUpdate. + */ static XSUM_U32 XSUM_rand(void) { static XSUM_U64 seed = PRIME32; @@ -160,11 +334,40 @@ static XSUM_U32 XSUM_rand(void) return (XSUM_U32)(seed >> 40); } +/* + * Technically, XXH3_64bits_update is identical to XXH3_128bits_update as of + * v0.8.0, but we treat them as separate. + */ +typedef XXH_errorcode (*XSUM_XXH3_update_t)(XXH3_state_t* state, const void* input, size_t length); -static void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 Nresult) +/* + * Runs the passed XXH3_update variant on random lengths. This is to test the + * more complex logic of the update function, catching bugs like this one: + * https://github.com/Cyan4973/xxHash/issues/378 + */ +static void XSUM_XXH3_randomUpdate(XXH3_state_t* state, const void* data, + size_t len, XSUM_XXH3_update_t update_fn) { - if (len>0) assert(data != NULL); + size_t p = 0; + while (p < len) { + size_t const modulo = len > 2 ? len : 2; + size_t l = (size_t)(XSUM_rand()) % modulo; + if (p + l > len) l = len - p; + (void)update_fn(state, (const char*)data+p, l); + p += l; + } +} +static void XSUM_testXXH3(const void* data, const XSUM_testdata64_t* testData) +{ + size_t len = testData->len; + XSUM_U64 seed = testData->seed; + XSUM_U64 Nresult = testData->Nresult; + if (len == 0) { + data = NULL; + } else { + assert(data != NULL); + } { XSUM_U64 const Dresult = XXH3_64bits_withSeed(data, len, seed); XSUM_checkResult64(Dresult, Nresult); } @@ -184,17 +387,9 @@ static void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* random ingestion */ - { size_t p = 0; - (void)XXH3_64bits_reset_withSeed(state, seed); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_64bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - } + (void)XXH3_64bits_reset_withSeed(state, seed); + XSUM_XXH3_randomUpdate(state, data, len, &XXH3_64bits_update); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* byte by byte ingestion */ { size_t pos; @@ -207,10 +402,17 @@ static void XSUM_testXXH3(const void* data, size_t len, XSUM_U64 seed, XSUM_U64 } } -static void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* secret, size_t secretSize, XSUM_U64 Nresult) +static void XSUM_testXXH3_withSecret(const void* data, const void* secret, + size_t secretSize, const XSUM_testdata64_t* testData) { - if (len>0) assert(data != NULL); + size_t len = (size_t)testData->len; + XSUM_U64 Nresult = testData->Nresult; + if (len == 0) { + data = NULL; + } else { + assert(data != NULL); + } { XSUM_U64 const Dresult = XXH3_64bits_withSecret(data, len, secret, secretSize); XSUM_checkResult64(Dresult, Nresult); } @@ -223,17 +425,9 @@ static void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* s XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* random ingestion */ - { size_t p = 0; - (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_64bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); - } + (void)XXH3_64bits_reset_withSecret(state, secret, secretSize); + XSUM_XXH3_randomUpdate(state, data, len, &XXH3_64bits_update); + XSUM_checkResult64(XXH3_64bits_digest(state), Nresult); /* byte by byte ingestion */ { size_t pos; @@ -246,8 +440,17 @@ static void XSUM_testXXH3_withSecret(const void* data, size_t len, const void* s } } -static void XSUM_testXXH128(const void* data, size_t len, XSUM_U64 seed, XXH128_hash_t Nresult) +static void XSUM_testXXH128(const void* data, const XSUM_testdata128_t* testData) { + size_t len = (size_t)testData->len; + XSUM_U64 seed = testData->seed; + XXH128_hash_t const Nresult = testData->Nresult; + if (len == 0) { + data = NULL; + } else { + assert(data != NULL); + } + { XXH128_hash_t const Dresult = XXH3_128bits_withSeed(data, len, seed); XSUM_checkResult128(Dresult, Nresult); } @@ -273,17 +476,9 @@ static void XSUM_testXXH128(const void* data, size_t len, XSUM_U64 seed, XXH128_ XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); /* random ingestion */ - { size_t p = 0; - (void)XXH3_128bits_reset_withSeed(state, seed); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_128bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); - } + (void)XXH3_128bits_reset_withSeed(state, seed); + XSUM_XXH3_randomUpdate(state, data, len, &XXH3_128bits_update); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); /* byte by byte ingestion */ { size_t pos; @@ -296,10 +491,15 @@ static void XSUM_testXXH128(const void* data, size_t len, XSUM_U64 seed, XXH128_ } } -static void XSUM_testXXH128_withSecret(const void* data, size_t len, const void* secret, size_t secretSize, XXH128_hash_t Nresult) +static void XSUM_testXXH128_withSecret(const void* data, const void* secret, size_t secretSize, const XSUM_testdata128_t* testData) { - if (len>0) assert(data != NULL); - + size_t len = testData->len; + XXH128_hash_t Nresult = testData->Nresult; + if (len == 0) { + data = NULL; + } else if (len>0) { + assert(data != NULL); + } { XXH128_hash_t const Dresult = XXH3_128bits_withSecret(data, len, secret, secretSize); XSUM_checkResult128(Dresult, Nresult); } @@ -312,17 +512,9 @@ static void XSUM_testXXH128_withSecret(const void* data, size_t len, const void* XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); /* random ingestion */ - { size_t p = 0; - (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); - while (p < len) { - size_t const modulo = len > 2 ? len : 2; - size_t l = (size_t)(XSUM_rand()) % modulo; - if (p + l > len) l = len - p; - (void)XXH3_128bits_update(state, (const char*)data+p, l); - p += l; - } - XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); - } + (void)XXH3_128bits_reset_withSecret(state, secret, secretSize); + XSUM_XXH3_randomUpdate(state, data, len, &XXH3_128bits_update); + XSUM_checkResult128(XXH3_128bits_digest(state), Nresult); /* byte by byte ingestion */ { size_t pos; @@ -335,32 +527,28 @@ static void XSUM_testXXH128_withSecret(const void* data, size_t len, const void* } } -#define SECRET_SAMPLE_NBBYTES 4 -typedef struct { XSUM_U8 byte[SECRET_SAMPLE_NBBYTES]; } verifSample_t; - -static void XSUM_testSecretGenerator(const void* customSeed, size_t len, verifSample_t result) +static void XSUM_testSecretGenerator(const void* customSeed, const XSUM_testdata_sample_t* testData) { static int nbTests = 1; const int sampleIndex[SECRET_SAMPLE_NBBYTES] = { 0, 62, 131, 191}; XSUM_U8 secretBuffer[XXH3_SECRET_DEFAULT_SIZE] = {0}; - verifSample_t samples; + XSUM_U8 samples[SECRET_SAMPLE_NBBYTES]; int i; - XXH3_generateSecret(secretBuffer, customSeed, len); + XXH3_generateSecret(secretBuffer, customSeed, testData->len); for (i=0; ibyte, sizeof(testData->byte))) { XSUM_log("\rError: Secret generation test %i: Internal sanity check failed. \n", nbTests); XSUM_log("\rGot { 0x%02X, 0x%02X, 0x%02X, 0x%02X }, expected { 0x%02X, 0x%02X, 0x%02X, 0x%02X } \n", - samples.byte[0], samples.byte[1], samples.byte[2], samples.byte[3], - result.byte[0], result.byte[1], result.byte[2], result.byte[3] ); + samples[0], samples[1], samples[2], samples[3], + testData->byte[0], testData->byte[1], testData->byte[2], testData->byte[3] ); exit(1); } nbTests++; } - /*! * XSUM_sanityCheck(): * Runs a sanity check before the benchmark. @@ -369,192 +557,42 @@ static void XSUM_testSecretGenerator(const void* customSeed, size_t len, verifSa */ XSUM_API void XSUM_sanityCheck(void) { + size_t i; #define SANITY_BUFFER_SIZE 2367 XSUM_U8 sanityBuffer[SANITY_BUFFER_SIZE]; - XSUM_fillTestBuffer(sanityBuffer, sizeof(sanityBuffer)); + const void* const secret = sanityBuffer + 7; + const size_t secretSize = XXH3_SECRET_SIZE_MIN + 11; + assert(sizeof(sanityBuffer) >= 7 + secretSize); - XSUM_testXXH32(NULL, 0, 0, 0x02CC5D05); - XSUM_testXXH32(NULL, 0, PRIME32, 0x36B78AE7); - XSUM_testXXH32(sanityBuffer, 1, 0, 0xCF65B03E); - XSUM_testXXH32(sanityBuffer, 1, PRIME32, 0xB4545AA4); - XSUM_testXXH32(sanityBuffer, 14, 0, 0x1208E7E2); - XSUM_testXXH32(sanityBuffer, 14, PRIME32, 0x6AF1D1FE); - XSUM_testXXH32(sanityBuffer,222, 0, 0x5BD11DBD); - XSUM_testXXH32(sanityBuffer,222, PRIME32, 0x58803C5F); - - XSUM_testXXH64(NULL , 0, 0, 0xEF46DB3751D8E999ULL); - XSUM_testXXH64(NULL , 0, PRIME32, 0xAC75FDA2929B17EFULL); - XSUM_testXXH64(sanityBuffer, 1, 0, 0xE934A84ADB052768ULL); - XSUM_testXXH64(sanityBuffer, 1, PRIME32, 0x5014607643A9B4C3ULL); - XSUM_testXXH64(sanityBuffer, 4, 0, 0x9136A0DCA57457EEULL); - XSUM_testXXH64(sanityBuffer, 14, 0, 0x8282DCC4994E35C8ULL); - XSUM_testXXH64(sanityBuffer, 14, PRIME32, 0xC3BD6BF63DEB6DF0ULL); - XSUM_testXXH64(sanityBuffer,222, 0, 0xB641AE8CB691C174ULL); - XSUM_testXXH64(sanityBuffer,222, PRIME32, 0x20CB8AB7AE10C14AULL); - - XSUM_testXXH3(NULL, 0, 0, 0x2D06800538D394C2ULL); /* empty string */ - XSUM_testXXH3(NULL, 0, PRIME64, 0xA8A6B918B2F0364AULL); - XSUM_testXXH3(sanityBuffer, 1, 0, 0xC44BDFF4074EECDBULL); /* 1 - 3 */ - XSUM_testXXH3(sanityBuffer, 1, PRIME64, 0x032BE332DD766EF8ULL); /* 1 - 3 */ - XSUM_testXXH3(sanityBuffer, 6, 0, 0x27B56A84CD2D7325ULL); /* 4 - 8 */ - XSUM_testXXH3(sanityBuffer, 6, PRIME64, 0x84589C116AB59AB9ULL); /* 4 - 8 */ - XSUM_testXXH3(sanityBuffer, 12, 0, 0xA713DAF0DFBB77E7ULL); /* 9 - 16 */ - XSUM_testXXH3(sanityBuffer, 12, PRIME64, 0xE7303E1B2336DE0EULL); /* 9 - 16 */ - XSUM_testXXH3(sanityBuffer, 24, 0, 0xA3FE70BF9D3510EBULL); /* 17 - 32 */ - XSUM_testXXH3(sanityBuffer, 24, PRIME64, 0x850E80FC35BDD690ULL); /* 17 - 32 */ - XSUM_testXXH3(sanityBuffer, 48, 0, 0x397DA259ECBA1F11ULL); /* 33 - 64 */ - XSUM_testXXH3(sanityBuffer, 48, PRIME64, 0xADC2CBAA44ACC616ULL); /* 33 - 64 */ - XSUM_testXXH3(sanityBuffer, 80, 0, 0xBCDEFBBB2C47C90AULL); /* 65 - 96 */ - XSUM_testXXH3(sanityBuffer, 80, PRIME64, 0xC6DD0CB699532E73ULL); /* 65 - 96 */ - XSUM_testXXH3(sanityBuffer, 195, 0, 0xCD94217EE362EC3AULL); /* 129-240 */ - XSUM_testXXH3(sanityBuffer, 195, PRIME64, 0xBA68003D370CB3D9ULL); /* 129-240 */ - - XSUM_testXXH3(sanityBuffer, 403, 0, 0xCDEB804D65C6DEA4ULL); /* one block, last stripe is overlapping */ - XSUM_testXXH3(sanityBuffer, 403, PRIME64, 0x6259F6ECFD6443FDULL); /* one block, last stripe is overlapping */ - XSUM_testXXH3(sanityBuffer, 512, 0, 0x617E49599013CB6BULL); /* one block, finishing at stripe boundary */ - XSUM_testXXH3(sanityBuffer, 512, PRIME64, 0x3CE457DE14C27708ULL); /* one block, finishing at stripe boundary */ - XSUM_testXXH3(sanityBuffer,2048, 0, 0xDD59E2C3A5F038E0ULL); /* 2 blocks, finishing at block boundary */ - XSUM_testXXH3(sanityBuffer,2048, PRIME64, 0x66F81670669ABABCULL); /* 2 blocks, finishing at block boundary */ - XSUM_testXXH3(sanityBuffer,2240, 0, 0x6E73A90539CF2948ULL); /* 3 blocks, finishing at stripe boundary */ - XSUM_testXXH3(sanityBuffer,2240, PRIME64, 0x757BA8487D1B5247ULL); /* 3 blocks, finishing at stripe boundary */ - XSUM_testXXH3(sanityBuffer,2367, 0, 0xCB37AEB9E5D361EDULL); /* 3 blocks, last stripe is overlapping */ - XSUM_testXXH3(sanityBuffer,2367, PRIME64, 0xD2DB3415B942B42AULL); /* 3 blocks, last stripe is overlapping */ - - /* XXH3 with Custom Secret */ - { const void* const secret = sanityBuffer + 7; - const size_t secretSize = XXH3_SECRET_SIZE_MIN + 11; - assert(sizeof(sanityBuffer) >= 7 + secretSize); - XSUM_testXXH3_withSecret(NULL, 0, secret, secretSize, 0x3559D64878C5C66CULL); /* empty string */ - XSUM_testXXH3_withSecret(sanityBuffer, 1, secret, secretSize, 0x8A52451418B2DA4DULL); /* 1 - 3 */ - XSUM_testXXH3_withSecret(sanityBuffer, 6, secret, secretSize, 0x82C90AB0519369ADULL); /* 4 - 8 */ - XSUM_testXXH3_withSecret(sanityBuffer, 12, secret, secretSize, 0x14631E773B78EC57ULL); /* 9 - 16 */ - XSUM_testXXH3_withSecret(sanityBuffer, 24, secret, secretSize, 0xCDD5542E4A9D9FE8ULL); /* 17 - 32 */ - XSUM_testXXH3_withSecret(sanityBuffer, 48, secret, secretSize, 0x33ABD54D094B2534ULL); /* 33 - 64 */ - XSUM_testXXH3_withSecret(sanityBuffer, 80, secret, secretSize, 0xE687BA1684965297ULL); /* 65 - 96 */ - XSUM_testXXH3_withSecret(sanityBuffer, 195, secret, secretSize, 0xA057273F5EECFB20ULL); /* 129-240 */ - - XSUM_testXXH3_withSecret(sanityBuffer, 403, secret, secretSize, 0x14546019124D43B8ULL); /* one block, last stripe is overlapping */ - XSUM_testXXH3_withSecret(sanityBuffer, 512, secret, secretSize, 0x7564693DD526E28DULL); /* one block, finishing at stripe boundary */ - XSUM_testXXH3_withSecret(sanityBuffer,2048, secret, secretSize, 0xD32E975821D6519FULL); /* >= 2 blocks, at least one scrambling */ - XSUM_testXXH3_withSecret(sanityBuffer,2367, secret, secretSize, 0x293FA8E5173BB5E7ULL); /* >= 2 blocks, at least one scrambling, last stripe unaligned */ - - XSUM_testXXH3_withSecret(sanityBuffer,64*10*3, secret, secretSize, 0x751D2EC54BC6038BULL); /* exactly 3 full blocks, not a multiple of 256 */ - } + XSUM_fillTestBuffer(sanityBuffer, sizeof(sanityBuffer)); - /* XXH128 */ - { XXH128_hash_t const expected = { 0x6001C324468D497FULL, 0x99AA06D3014798D8ULL }; - XSUM_testXXH128(NULL, 0, 0, expected); /* empty string */ - } - { XXH128_hash_t const expected = { 0x5444F7869C671AB0ULL, 0x92220AE55E14AB50ULL }; - XSUM_testXXH128(NULL, 0, PRIME32, expected); - } - { XXH128_hash_t const expected = { 0xC44BDFF4074EECDBULL, 0xA6CD5E9392000F6AULL }; - XSUM_testXXH128(sanityBuffer, 1, 0, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0xB53D5557E7F76F8DULL, 0x89B99554BA22467CULL }; - XSUM_testXXH128(sanityBuffer, 1, PRIME32, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0x3E7039BDDA43CFC6ULL, 0x082AFE0B8162D12AULL }; - XSUM_testXXH128(sanityBuffer, 6, 0, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0x269D8F70BE98856EULL, 0x5A865B5389ABD2B1ULL }; - XSUM_testXXH128(sanityBuffer, 6, PRIME32, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0x061A192713F69AD9ULL, 0x6E3EFD8FC7802B18ULL }; - XSUM_testXXH128(sanityBuffer, 12, 0, expected); /* 9-16 */ - } - { XXH128_hash_t const expected = { 0x9BE9F9A67F3C7DFBULL, 0xD7E09D518A3405D3ULL }; - XSUM_testXXH128(sanityBuffer, 12, PRIME32, expected); /* 9-16 */ - } - { XXH128_hash_t const expected = { 0x1E7044D28B1B901DULL, 0x0CE966E4678D3761ULL }; - XSUM_testXXH128(sanityBuffer, 24, 0, expected); /* 17-32 */ - } - { XXH128_hash_t const expected = { 0xD7304C54EBAD40A9ULL, 0x3162026714A6A243ULL }; - XSUM_testXXH128(sanityBuffer, 24, PRIME32, expected); /* 17-32 */ - } - { XXH128_hash_t const expected = { 0xF942219AED80F67BULL, 0xA002AC4E5478227EULL }; - XSUM_testXXH128(sanityBuffer, 48, 0, expected); /* 33-64 */ - } - { XXH128_hash_t const expected = { 0x7BA3C3E453A1934EULL, 0x163ADDE36C072295ULL }; - XSUM_testXXH128(sanityBuffer, 48, PRIME32, expected); /* 33-64 */ - } - { XXH128_hash_t const expected = { 0x5E8BAFB9F95FB803ULL, 0x4952F58181AB0042ULL }; - XSUM_testXXH128(sanityBuffer, 81, 0, expected); /* 65-96 */ + /* XXH32 */ + for (i = 0; i < (sizeof(XSUM_XXH32_testdata)/sizeof(XSUM_XXH32_testdata[0])); i++) { + XSUM_testXXH32(sanityBuffer, &XSUM_XXH32_testdata[i]); } - { XXH128_hash_t const expected = { 0x703FBB3D7A5F755CULL, 0x2724EC7ADC750FB6ULL }; - XSUM_testXXH128(sanityBuffer, 81, PRIME32, expected); /* 65-96 */ + /* XXH64 */ + for (i = 0; i < (sizeof(XSUM_XXH64_testdata)/sizeof(XSUM_XXH64_testdata[0])); i++) { + XSUM_testXXH64(sanityBuffer, &XSUM_XXH64_testdata[i]); } - { XXH128_hash_t const expected = { 0xF1AEBD597CEC6B3AULL, 0x337E09641B948717ULL }; - XSUM_testXXH128(sanityBuffer, 222, 0, expected); /* 129-240 */ + /* XXH3_64bits, seeded */ + for (i = 0; i < (sizeof(XSUM_XXH3_testdata)/sizeof(XSUM_XXH3_testdata[0])); i++) { + XSUM_testXXH3(sanityBuffer, &XSUM_XXH3_testdata[i]); } - { XXH128_hash_t const expected = { 0xAE995BB8AF917A8DULL, 0x91820016621E97F1ULL }; - XSUM_testXXH128(sanityBuffer, 222, PRIME32, expected); /* 129-240 */ + /* XXH3_64bits, custom secret */ + for (i = 0; i < (sizeof(XSUM_XXH3_withSecret_testdata)/sizeof(XSUM_XXH3_withSecret_testdata[0])); i++) { + XSUM_testXXH3_withSecret(sanityBuffer, secret, secretSize, &XSUM_XXH3_withSecret_testdata[i]); } - { XXH128_hash_t const expected = { 0xCDEB804D65C6DEA4ULL, 0x1B6DE21E332DD73DULL }; - XSUM_testXXH128(sanityBuffer, 403, 0, expected); /* one block, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x6259F6ECFD6443FDULL, 0xBED311971E0BE8F2ULL }; - XSUM_testXXH128(sanityBuffer, 403, PRIME64, expected); /* one block, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x617E49599013CB6BULL, 0x18D2D110DCC9BCA1ULL }; - XSUM_testXXH128(sanityBuffer, 512, 0, expected); /* one block, finishing at stripe boundary */ - } - { XXH128_hash_t const expected = { 0x3CE457DE14C27708ULL, 0x925D06B8EC5B8040ULL }; - XSUM_testXXH128(sanityBuffer, 512, PRIME64, expected); /* one block, finishing at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xDD59E2C3A5F038E0ULL, 0xF736557FD47073A5ULL }; - XSUM_testXXH128(sanityBuffer,2048, 0, expected); /* two blocks, finishing at block boundary */ - } - { XXH128_hash_t const expected = { 0x230D43F30206260BULL, 0x7FB03F7E7186C3EAULL }; - XSUM_testXXH128(sanityBuffer,2048, PRIME32, expected); /* two blocks, finishing at block boundary */ - } - { XXH128_hash_t const expected = { 0x6E73A90539CF2948ULL, 0xCCB134FBFA7CE49DULL }; - XSUM_testXXH128(sanityBuffer,2240, 0, expected); /* two blocks, ends at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xED385111126FBA6FULL, 0x50A1FE17B338995FULL }; - XSUM_testXXH128(sanityBuffer,2240, PRIME32, expected); /* two blocks, ends at stripe boundary */ - } - { XXH128_hash_t const expected = { 0xCB37AEB9E5D361EDULL, 0xE89C0F6FF369B427ULL }; - XSUM_testXXH128(sanityBuffer,2367, 0, expected); /* two blocks, last stripe is overlapping */ - } - { XXH128_hash_t const expected = { 0x6F5360AE69C2F406ULL, 0xD23AAE4B76C31ECBULL }; - XSUM_testXXH128(sanityBuffer,2367, PRIME32, expected); /* two blocks, last stripe is overlapping */ + /* XXH128 */ + for (i = 0; i < (sizeof(XSUM_XXH128_testdata)/sizeof(XSUM_XXH128_testdata[0])); i++) { + XSUM_testXXH128(sanityBuffer, &XSUM_XXH128_testdata[i]); } - /* XXH128 with custom Secret */ - { const void* const secret = sanityBuffer + 7; - const size_t secretSize = XXH3_SECRET_SIZE_MIN + 11; - assert(sizeof(sanityBuffer) >= 7 + secretSize); - - { XXH128_hash_t const expected = { 0x005923CCEECBE8AEULL, 0x5F70F4EA232F1D38ULL }; - XSUM_testXXH128_withSecret(NULL, 0, secret, secretSize, expected); /* empty string */ - } - { XXH128_hash_t const expected = { 0x8A52451418B2DA4DULL, 0x3A66AF5A9819198EULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 1, secret, secretSize, expected); /* 1-3 */ - } - { XXH128_hash_t const expected = { 0x0B61C8ACA7D4778FULL, 0x376BD91B6432F36DULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 6, secret, secretSize, expected); /* 4-8 */ - } - { XXH128_hash_t const expected = { 0xAF82F6EBA263D7D8ULL, 0x90A3C2D839F57D0FULL }; - XSUM_testXXH128_withSecret(sanityBuffer, 12, secret, secretSize, expected); /* 9-16 */ - } + for (i = 0; i < (sizeof(XSUM_XXH128_withSecret_testdata)/sizeof(XSUM_XXH128_withSecret_testdata[0])); i++) { + XSUM_testXXH128_withSecret(sanityBuffer, secret, secretSize, &XSUM_XXH128_withSecret_testdata[i]); } - /* secret generator */ - { verifSample_t const expected = { { 0xB8, 0x26, 0x83, 0x7E } }; - XSUM_testSecretGenerator(NULL, 0, expected); - } - - { verifSample_t const expected = { { 0xA6, 0x16, 0x06, 0x7B } }; - XSUM_testSecretGenerator(sanityBuffer, 1, expected); - } - - { verifSample_t const expected = { { 0xDA, 0x2A, 0x12, 0x11 } }; - XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_SIZE_MIN - 1, expected); - } - - { verifSample_t const expected = { { 0x7E, 0x48, 0x0C, 0xA7 } }; - XSUM_testSecretGenerator(sanityBuffer, XXH3_SECRET_DEFAULT_SIZE + 500, expected); + for (i = 0; i < (sizeof(XSUM_XXH3_generateSecret_testdata)/sizeof(XSUM_XXH3_generateSecret_testdata[0])); i++) { + XSUM_testSecretGenerator(sanityBuffer, &XSUM_XXH3_generateSecret_testdata[i]); } XSUM_logVerbose(3, "\r%70s\r", ""); /* Clean display line */ From 15f61dbe5504b755f752b04aa8e294ce46dda07a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 26 Sep 2020 10:24:57 -0700 Subject: [PATCH 025/187] fix compilation on macOS added macOS test to travisCI --- .travis.yml | 10 +++++++++- cli/xsum_sanity_check.h | 5 ++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 2f3a2168..db7380e5 100644 --- a/.travis.yml +++ b/.travis.yml @@ -42,6 +42,14 @@ matrix: - CPPFLAGS=-DXXH_REROLL=1 make check # reroll code path (#240) - make -C tests/bench + - name: macOS General Test + os: osx + compiler: clang + script: + - make # test library build + - make clean + - make test MOREFLAGS='-Werror' | tee # test scenario where `stdout` is not the console + - name: ARM compilation and consistency checks (Qemu) dist: xenial arch: amd64 @@ -111,7 +119,7 @@ matrix: - CPPFLAGS=-DXXH_VECTOR=XXH_VSX CFLAGS="-O3 -maltivec -mvsx -mpower8-vector -mcpu=power8" LDFLAGS="-static" make check # altivec.h redefinition issue #426 - make clean - - CPPFLAGS=-DXXH_VECTOR=XXH_VSX CFLAGS="-maltivec -mvsx -mcpu=power8 -mpower8-vector" make -C tests test_ppc_redefine + - CPPFLAGS=-DXXH_VECTOR=XXH_VSX CFLAGS="-maltivec -mvsx -mcpu=power8 -mpower8-vector" make -C tests test_ppc_redefine - name: IBM s390x compilation and consistency checks dist: bionic diff --git a/cli/xsum_sanity_check.h b/cli/xsum_sanity_check.h index a3f57a16..9f3f2b85 100644 --- a/cli/xsum_sanity_check.h +++ b/cli/xsum_sanity_check.h @@ -26,7 +26,10 @@ #ifndef XSUM_SANITY_CHECK_H #define XSUM_SANITY_CHECK_H -#include "xsum_config.h" +#include "xsum_config.h" /* XSUM_API, XSUM_U8 */ + +#include /* size_t */ + #ifdef __cplusplus extern "C" { From 56ea9135893dea4205833e8948bf32aec632f168 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Sat, 26 Sep 2020 10:38:27 -0700 Subject: [PATCH 026/187] fixed cat /proc/cpuinfo for macOS --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index db7380e5..675965f7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -2,7 +2,7 @@ language: c # Dump CPU info before start before_install: - - cat /proc/cpuinfo + - cat /proc/cpuinfo || echo /proc/cpuinfo is not present matrix: fast_finish: true From c6eb8210738ad4e3e6c74497288580a20231eee2 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Mon, 28 Sep 2020 09:21:51 -0700 Subject: [PATCH 027/187] added repology status --- README.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 1207969a..b976e3d1 100644 --- a/README.md +++ b/README.md @@ -220,10 +220,17 @@ thanks to great contributors. They are [listed here](http://www.xxhash.com/#other-languages). -### Special Thanks +### Packaging status + +Many distributions bundle a package manager +which allows easy xxhash installation as both a `libxxhash` library +and `xxhsum` command line interface. -Takayuki Matsuoka, aka @t-mat, for creating `xxhsum -c` and general support during early xxh releases +[![Packaging status](https://repology.org/badge/vertical-allrepos/xxhash.svg)](https://repology.org/project/xxhash/versions) -Mathias Westerdahl, aka @JCash, for introducing the first version of `XXH64` -Devin Hussey, aka @easyaspi314, for excellent low-level optimizations on `XXH3` and `XXH128` +### Special Thanks + +- Takayuki Matsuoka, aka @t-mat, for creating `xxhsum -c` and great support during early xxh releases +- Mathias Westerdahl, aka @JCash, for introducing the first version of `XXH64` +- Devin Hussey, aka @easyaspi314, for incredible low-level optimizations on `XXH3` and `XXH128` From d83cce5b91acf299535b8d3762285cc451b5c34a Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Wed, 30 Sep 2020 15:56:07 -0700 Subject: [PATCH 028/187] fix prefetching for x86 32-bit target on MSVC --- xxhash.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xxhash.h b/xxhash.h index 2ba034a5..b5e57b1b 100644 --- a/xxhash.h +++ b/xxhash.h @@ -2481,7 +2481,7 @@ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mule(xxh_u32x4 a, xxh_u32x4 b) #if defined(XXH_NO_PREFETCH) # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ #else -# if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */ +# if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) /* _mm_prefetch() not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define XXH_PREFETCH(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) From ba6fdd9d21e16fdea5a28560bedc69dc91bcf846 Mon Sep 17 00:00:00 2001 From: Alex Pankratov Date: Thu, 8 Oct 2020 12:33:13 +0200 Subject: [PATCH 029/187] Fix xxh_x86dispatch build with MSVC --- xxh_x86dispatch.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/xxh_x86dispatch.c b/xxh_x86dispatch.c index a618ae89..384004d2 100644 --- a/xxh_x86dispatch.c +++ b/xxh_x86dispatch.c @@ -44,8 +44,19 @@ extern "C" { # error "Dispatching is currently only supported on x86 and x86_64." #endif -#ifndef __GNUC__ -# error "Dispatching requires __attribute__((__target__)) capability" +#if defined(__GNUC__) +# include /* sse2 */ +# include /* avx2 */ +# define XXH_TARGET_AVX512 __attribute__((__target__("avx512f"))) +# define XXH_TARGET_AVX2 __attribute__((__target__("avx2"))) +# define XXH_TARGET_SSE2 __attribute__((__target__("sse2"))) +#elif defined(_MSC_VER) +# include +# define XXH_TARGET_AVX512 +# define XXH_TARGET_AVX2 +# define XXH_TARGET_SSE2 +#else +# error "Dispatching is currently not supported for your compiler." #endif #define XXH_DISPATCH_AVX2 /* enable dispatch towards AVX2 */ @@ -62,18 +73,8 @@ extern "C" { #endif #include -#if defined(__GNUC__) -# include /* sse2 */ -# include /* avx2 */ -#elif defined(_MSC_VER) -# include -#endif - #define XXH_INLINE_ALL #define XXH_X86DISPATCH -#define XXH_TARGET_AVX512 __attribute__((__target__("avx512f"))) -#define XXH_TARGET_AVX2 __attribute__((__target__("avx2"))) -#define XXH_TARGET_SSE2 __attribute__((__target__("sse2"))) #include "xxhash.h" /* From 45b2cb406dcd16ebdc5887f33a18c7a1c9c7f6fa Mon Sep 17 00:00:00 2001 From: Alex Pankratov Date: Thu, 8 Oct 2020 12:36:27 +0200 Subject: [PATCH 030/187] remove duplicate 'intrin.h' include, move Intel's guide link to a proper place --- xxh_x86dispatch.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/xxh_x86dispatch.c b/xxh_x86dispatch.c index 384004d2..1fc6fac8 100644 --- a/xxh_x86dispatch.c +++ b/xxh_x86dispatch.c @@ -77,14 +77,6 @@ extern "C" { #define XXH_X86DISPATCH #include "xxhash.h" -/* - * Modified version of Intel's guide - * https://software.intel.com/en-us/articles/how-to-detect-new-instruction-support-in-the-4th-generation-intel-core-processor-family - */ -#if defined(_MSC_VER) -# include -#endif - /* * Support both AT&T and Intel dialects * @@ -103,7 +95,6 @@ extern "C" { # define I_ATT(intel, att) "{" att "|" intel "}\n\t" #endif - static void XXH_cpuid(xxh_u32 eax, xxh_u32 ecx, xxh_u32* abcd) { #if defined(_MSC_VER) @@ -134,6 +125,11 @@ static void XXH_cpuid(xxh_u32 eax, xxh_u32 ecx, xxh_u32* abcd) #endif } +/* + * Modified version of Intel's guide + * https://software.intel.com/en-us/articles/how-to-detect-new-instruction-support-in-the-4th-generation-intel-core-processor-family + */ + #if defined(XXH_DISPATCH_AVX2) || defined(XXH_DISPATCH_AVX512) /* * While the CPU may support AVX2, the operating system might not properly save From e1784dda98688221509d9562a18827fae685e6f0 Mon Sep 17 00:00:00 2001 From: "easyaspi314 (Devin)" Date: Thu, 15 Oct 2020 18:33:27 -0400 Subject: [PATCH 031/187] [WIP] Doxygenize and redocument some funcs XXH32's API, XXH*_state_s, the config macros, and some other things are documented in Doxygen format, basic Doxyfile added. Some functions were also redocumented in detail. Still need to work on structuring the main page (set it to README.md?). How to view docs for now: $ sudo apt install doxygen $ npm install -g http-server # or you can use LLVM's scan-view $ doxygen && hs doxygen/html -s -c-1 -o --- Doxyfile | 2589 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ xxhash.h | 881 +++++++++++++++---- 2 files changed, 3302 insertions(+), 168 deletions(-) create mode 100644 Doxyfile diff --git a/Doxyfile b/Doxyfile new file mode 100644 index 00000000..373986a5 --- /dev/null +++ b/Doxyfile @@ -0,0 +1,2589 @@ +# Doxyfile 1.8.20 + +# This file describes the settings to be used by the documentation system +# doxygen (www.doxygen.org) for a project. +# +# All text after a double hash (##) is considered a comment and is placed in +# front of the TAG it is preceding. +# +# All text after a single hash (#) is considered a comment and will be ignored. +# The format is: +# TAG = value [value, ...] +# For lists, items can also be appended using: +# TAG += value [value, ...] +# Values that contain spaces should be placed between quotes (\" \"). + +#--------------------------------------------------------------------------- +# Project related configuration options +#--------------------------------------------------------------------------- + +# This tag specifies the encoding used for all characters in the configuration +# file that follow. The default is UTF-8 which is also the encoding used for all +# text before the first occurrence of this tag. Doxygen uses libiconv (or the +# iconv built into libc) for the transcoding. See +# https://www.gnu.org/software/libiconv/ for the list of possible encodings. +# The default value is: UTF-8. + +DOXYFILE_ENCODING = UTF-8 + +# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by +# double-quotes, unless you are using Doxywizard) that should identify the +# project for which the documentation is generated. This name is used in the +# title of most generated pages and in a few other places. +# The default value is: My Project. + +PROJECT_NAME = "xxHash" + +# The PROJECT_NUMBER tag can be used to enter a project or revision number. This +# could be handy for archiving the generated documentation or if some version +# control system is used. + +PROJECT_NUMBER = "0.8.0" + +# Using the PROJECT_BRIEF tag one can provide an optional one line description +# for a project that appears at the top of each page and should give viewer a +# quick idea about the purpose of the project. Keep the description short. + +PROJECT_BRIEF = "Extremely fast non-cryptographic hash function" + +# With the PROJECT_LOGO tag one can specify a logo or an icon that is included +# in the documentation. The maximum height of the logo should not exceed 55 +# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy +# the logo to the output directory. + +PROJECT_LOGO = + +# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path +# into which the generated documentation will be written. If a relative path is +# entered, it will be relative to the location where doxygen was started. If +# left blank the current directory will be used. + +OUTPUT_DIRECTORY = doxygen + +# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub- +# directories (in 2 levels) under the output directory of each output format and +# will distribute the generated files over these directories. Enabling this +# option can be useful when feeding doxygen a huge amount of source files, where +# putting all generated files in the same directory would otherwise causes +# performance problems for the file system. +# The default value is: NO. + +CREATE_SUBDIRS = NO + +# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII +# characters to appear in the names of generated files. If set to NO, non-ASCII +# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode +# U+3044. +# The default value is: NO. + +ALLOW_UNICODE_NAMES = NO + +# The OUTPUT_LANGUAGE tag is used to specify the language in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all constant output in the proper language. +# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese, +# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States), +# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian, +# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages), +# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian, +# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian, +# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish, +# Ukrainian and Vietnamese. +# The default value is: English. + +OUTPUT_LANGUAGE = English + +# The OUTPUT_TEXT_DIRECTION tag is used to specify the direction in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all generated output in the proper direction. +# Possible values are: None, LTR, RTL and Context. +# The default value is: None. + +OUTPUT_TEXT_DIRECTION = None + +# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member +# descriptions after the members that are listed in the file and class +# documentation (similar to Javadoc). Set to NO to disable this. +# The default value is: YES. + +BRIEF_MEMBER_DESC = YES + +# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief +# description of a member or function before the detailed description +# +# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the +# brief descriptions will be completely suppressed. +# The default value is: YES. + +REPEAT_BRIEF = YES + +# This tag implements a quasi-intelligent brief description abbreviator that is +# used to form the text in various listings. Each string in this list, if found +# as the leading text of the brief description, will be stripped from the text +# and the result, after processing the whole list, is used as the annotated +# text. Otherwise, the brief description is used as-is. If left blank, the +# following values are used ($name is automatically replaced with the name of +# the entity):The $name class, The $name widget, The $name file, is, provides, +# specifies, contains, represents, a, an and the. + +ABBREVIATE_BRIEF = "The $name class" \ + "The $name widget" \ + "The $name file" \ + is \ + provides \ + specifies \ + contains \ + represents \ + a \ + an \ + the + +# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then +# doxygen will generate a detailed section even if there is only a brief +# description. +# The default value is: NO. + +ALWAYS_DETAILED_SEC = NO + +# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all +# inherited members of a class in the documentation of that class as if those +# members were ordinary class members. Constructors, destructors and assignment +# operators of the base classes will not be shown. +# The default value is: NO. + +INLINE_INHERITED_MEMB = NO + +# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path +# before files name in the file list and in the header files. If set to NO the +# shortest path that makes the file name unique will be used +# The default value is: YES. + +FULL_PATH_NAMES = YES + +# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path. +# Stripping is only done if one of the specified strings matches the left-hand +# part of the path. The tag can be used to show relative paths in the file list. +# If left blank the directory from which doxygen is run is used as the path to +# strip. +# +# Note that you can specify absolute paths here, but also relative paths, which +# will be relative from the directory where doxygen is started. +# This tag requires that the tag FULL_PATH_NAMES is set to YES. + +STRIP_FROM_PATH = + +# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the +# path mentioned in the documentation of a class, which tells the reader which +# header file to include in order to use a class. If left blank only the name of +# the header file containing the class definition is used. Otherwise one should +# specify the list of include paths that are normally passed to the compiler +# using the -I flag. + +STRIP_FROM_INC_PATH = + +# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but +# less readable) file names. This can be useful is your file systems doesn't +# support long names like on DOS, Mac, or CD-ROM. +# The default value is: NO. + +SHORT_NAMES = NO + +# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the +# first line (until the first dot) of a Javadoc-style comment as the brief +# description. If set to NO, the Javadoc-style will behave just like regular Qt- +# style comments (thus requiring an explicit @brief command for a brief +# description.) +# The default value is: NO. + +JAVADOC_AUTOBRIEF = NO + +# If the JAVADOC_BANNER tag is set to YES then doxygen will interpret a line +# such as +# /*************** +# as being the beginning of a Javadoc-style comment "banner". If set to NO, the +# Javadoc-style will behave just like regular comments and it will not be +# interpreted by doxygen. +# The default value is: NO. + +JAVADOC_BANNER = NO + +# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first +# line (until the first dot) of a Qt-style comment as the brief description. If +# set to NO, the Qt-style will behave just like regular Qt-style comments (thus +# requiring an explicit \brief command for a brief description.) +# The default value is: NO. + +QT_AUTOBRIEF = NO + +# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a +# multi-line C++ special comment block (i.e. a block of //! or /// comments) as +# a brief description. This used to be the default behavior. The new default is +# to treat a multi-line C++ comment block as a detailed description. Set this +# tag to YES if you prefer the old behavior instead. +# +# Note that setting this tag to YES also means that rational rose comments are +# not recognized any more. +# The default value is: NO. + +MULTILINE_CPP_IS_BRIEF = NO + +# By default Python docstrings are displayed as preformatted text and doxygen's +# special commands cannot be used. By setting PYTHON_DOCSTRING to NO the +# doxygen's special commands can be used and the contents of the docstring +# documentation blocks is shown as doxygen documentation. +# The default value is: YES. + +PYTHON_DOCSTRING = YES + +# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the +# documentation from any documented member that it re-implements. +# The default value is: YES. + +INHERIT_DOCS = YES + +# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new +# page for each member. If set to NO, the documentation of a member will be part +# of the file/class/namespace that contains it. +# The default value is: NO. + +SEPARATE_MEMBER_PAGES = NO + +# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen +# uses this value to replace tabs by spaces in code fragments. +# Minimum value: 1, maximum value: 16, default value: 4. + +TAB_SIZE = 4 + +# This tag can be used to specify a number of aliases that act as commands in +# the documentation. An alias has the form: +# name=value +# For example adding +# "sideeffect=@par Side Effects:\n" +# will allow you to put the command \sideeffect (or @sideeffect) in the +# documentation, which will result in a user-defined paragraph with heading +# "Side Effects:". You can put \n's in the value part of an alias to insert +# newlines (in the resulting output). You can put ^^ in the value part of an +# alias to insert a newline as if a physical newline was in the original file. +# When you need a literal { or } or , in the value part of an alias you have to +# escape them by means of a backslash (\), this can lead to conflicts with the +# commands \{ and \} for these it is advised to use the version @{ and @} or use +# a double escape (\\{ and \\}) + +ALIASES = + +# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources +# only. Doxygen will then generate output that is more tailored for C. For +# instance, some of the names that are used will be different. The list of all +# members will be omitted, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_FOR_C = YES + +# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or +# Python sources only. Doxygen will then generate output that is more tailored +# for that language. For instance, namespaces will be presented as packages, +# qualified scopes will look different, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_JAVA = NO + +# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran +# sources. Doxygen will then generate output that is tailored for Fortran. +# The default value is: NO. + +OPTIMIZE_FOR_FORTRAN = NO + +# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL +# sources. Doxygen will then generate output that is tailored for VHDL. +# The default value is: NO. + +OPTIMIZE_OUTPUT_VHDL = NO + +# Set the OPTIMIZE_OUTPUT_SLICE tag to YES if your project consists of Slice +# sources only. Doxygen will then generate output that is more tailored for that +# language. For instance, namespaces will be presented as modules, types will be +# separated into more groups, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_SLICE = NO + +# Doxygen selects the parser to use depending on the extension of the files it +# parses. With this tag you can assign which parser to use for a given +# extension. Doxygen has a built-in mapping, but you can override or extend it +# using this tag. The format is ext=language, where ext is a file extension, and +# language is one of the parsers supported by doxygen: IDL, Java, JavaScript, +# Csharp (C#), C, C++, D, PHP, md (Markdown), Objective-C, Python, Slice, VHDL, +# Fortran (fixed format Fortran: FortranFixed, free formatted Fortran: +# FortranFree, unknown formatted Fortran: Fortran. In the later case the parser +# tries to guess whether the code is fixed or free formatted code, this is the +# default for Fortran type files). For instance to make doxygen treat .inc files +# as Fortran files (default is PHP), and .f files as C (default is Fortran), +# use: inc=Fortran f=C. +# +# Note: For files without extension you can use no_extension as a placeholder. +# +# Note that for custom extensions you also need to set FILE_PATTERNS otherwise +# the files are not read by doxygen. + +EXTENSION_MAPPING = + +# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments +# according to the Markdown format, which allows for more readable +# documentation. See https://daringfireball.net/projects/markdown/ for details. +# The output of markdown processing is further processed by doxygen, so you can +# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in +# case of backward compatibilities issues. +# The default value is: YES. + +MARKDOWN_SUPPORT = YES + +# When the TOC_INCLUDE_HEADINGS tag is set to a non-zero value, all headings up +# to that level are automatically included in the table of contents, even if +# they do not have an id attribute. +# Note: This feature currently applies only to Markdown headings. +# Minimum value: 0, maximum value: 99, default value: 5. +# This tag requires that the tag MARKDOWN_SUPPORT is set to YES. + +TOC_INCLUDE_HEADINGS = 5 + +# When enabled doxygen tries to link words that correspond to documented +# classes, or namespaces to their corresponding documentation. Such a link can +# be prevented in individual cases by putting a % sign in front of the word or +# globally by setting AUTOLINK_SUPPORT to NO. +# The default value is: YES. + +AUTOLINK_SUPPORT = YES + +# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want +# to include (a tag file for) the STL sources as input, then you should set this +# tag to YES in order to let doxygen match functions declarations and +# definitions whose arguments contain STL classes (e.g. func(std::string); +# versus func(std::string) {}). This also make the inheritance and collaboration +# diagrams that involve STL classes more complete and accurate. +# The default value is: NO. + +BUILTIN_STL_SUPPORT = NO + +# If you use Microsoft's C++/CLI language, you should set this option to YES to +# enable parsing support. +# The default value is: NO. + +CPP_CLI_SUPPORT = NO + +# Set the SIP_SUPPORT tag to YES if your project consists of sip (see: +# https://www.riverbankcomputing.com/software/sip/intro) sources only. Doxygen +# will parse them like normal C++ but will assume all classes use public instead +# of private inheritance when no explicit protection keyword is present. +# The default value is: NO. + +SIP_SUPPORT = NO + +# For Microsoft's IDL there are propget and propput attributes to indicate +# getter and setter methods for a property. Setting this option to YES will make +# doxygen to replace the get and set methods by a property in the documentation. +# This will only work if the methods are indeed getting or setting a simple +# type. If this is not the case, or you want to show the methods anyway, you +# should set this option to NO. +# The default value is: YES. + +IDL_PROPERTY_SUPPORT = YES + +# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC +# tag is set to YES then doxygen will reuse the documentation of the first +# member in the group (if any) for the other members of the group. By default +# all members of a group must be documented explicitly. +# The default value is: NO. + +DISTRIBUTE_GROUP_DOC = NO + +# If one adds a struct or class to a group and this option is enabled, then also +# any nested class or struct is added to the same group. By default this option +# is disabled and one has to add nested compounds explicitly via \ingroup. +# The default value is: NO. + +GROUP_NESTED_COMPOUNDS = NO + +# Set the SUBGROUPING tag to YES to allow class member groups of the same type +# (for instance a group of public functions) to be put as a subgroup of that +# type (e.g. under the Public Functions section). Set it to NO to prevent +# subgrouping. Alternatively, this can be done per class using the +# \nosubgrouping command. +# The default value is: YES. + +SUBGROUPING = YES + +# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions +# are shown inside the group in which they are included (e.g. using \ingroup) +# instead of on a separate page (for HTML and Man pages) or section (for LaTeX +# and RTF). +# +# Note that this feature does not work in combination with +# SEPARATE_MEMBER_PAGES. +# The default value is: NO. + +INLINE_GROUPED_CLASSES = NO + +# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions +# with only public data fields or simple typedef fields will be shown inline in +# the documentation of the scope in which they are defined (i.e. file, +# namespace, or group documentation), provided this scope is documented. If set +# to NO, structs, classes, and unions are shown on a separate page (for HTML and +# Man pages) or section (for LaTeX and RTF). +# The default value is: NO. + +INLINE_SIMPLE_STRUCTS = NO + +# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or +# enum is documented as struct, union, or enum with the name of the typedef. So +# typedef struct TypeS {} TypeT, will appear in the documentation as a struct +# with name TypeT. When disabled the typedef will appear as a member of a file, +# namespace, or class. And the struct will be named TypeS. This can typically be +# useful for C code in case the coding convention dictates that all compound +# types are typedef'ed and only the typedef is referenced, never the tag name. +# The default value is: NO. + +TYPEDEF_HIDES_STRUCT = NO + +# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This +# cache is used to resolve symbols given their name and scope. Since this can be +# an expensive process and often the same symbol appears multiple times in the +# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small +# doxygen will become slower. If the cache is too large, memory is wasted. The +# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range +# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536 +# symbols. At the end of a run doxygen will report the cache usage and suggest +# the optimal cache size from a speed point of view. +# Minimum value: 0, maximum value: 9, default value: 0. + +LOOKUP_CACHE_SIZE = 0 + +# The NUM_PROC_THREADS specifies the number threads doxygen is allowed to use +# during processing. When set to 0 doxygen will based this on the number of +# cores available in the system. You can set it explicitly to a value larger +# than 0 to get more control over the balance between CPU load and processing +# speed. At this moment only the input processing can be done using multiple +# threads. Since this is still an experimental feature the default is set to 1, +# which efficively disables parallel processing. Please report any issues you +# encounter. Generating dot graphs in parallel is controlled by the +# DOT_NUM_THREADS setting. +# Minimum value: 0, maximum value: 32, default value: 1. + +NUM_PROC_THREADS = 1 + +#--------------------------------------------------------------------------- +# Build related configuration options +#--------------------------------------------------------------------------- + +# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in +# documentation are documented, even if no documentation was available. Private +# class members and static file members will be hidden unless the +# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES. +# Note: This will also disable the warnings about undocumented members that are +# normally produced when WARNINGS is set to YES. +# The default value is: NO. + +EXTRACT_ALL = NO + +# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will +# be included in the documentation. +# The default value is: NO. + +EXTRACT_PRIVATE = NO + +# If the EXTRACT_PRIV_VIRTUAL tag is set to YES, documented private virtual +# methods of a class will be included in the documentation. +# The default value is: NO. + +EXTRACT_PRIV_VIRTUAL = NO + +# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal +# scope will be included in the documentation. +# The default value is: NO. + +EXTRACT_PACKAGE = NO + +# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be +# included in the documentation. +# The default value is: NO. + +EXTRACT_STATIC = YES + +# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined +# locally in source files will be included in the documentation. If set to NO, +# only classes defined in header files are included. Does not have any effect +# for Java sources. +# The default value is: YES. + +EXTRACT_LOCAL_CLASSES = YES + +# This flag is only useful for Objective-C code. If set to YES, local methods, +# which are defined in the implementation section but not in the interface are +# included in the documentation. If set to NO, only methods in the interface are +# included. +# The default value is: NO. + +EXTRACT_LOCAL_METHODS = NO + +# If this flag is set to YES, the members of anonymous namespaces will be +# extracted and appear in the documentation as a namespace called +# 'anonymous_namespace{file}', where file will be replaced with the base name of +# the file that contains the anonymous namespace. By default anonymous namespace +# are hidden. +# The default value is: NO. + +EXTRACT_ANON_NSPACES = NO + +# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all +# undocumented members inside documented classes or files. If set to NO these +# members will be included in the various overviews, but no documentation +# section is generated. This option has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_MEMBERS = NO + +# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all +# undocumented classes that are normally visible in the class hierarchy. If set +# to NO, these classes will be included in the various overviews. This option +# has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_CLASSES = NO + +# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend +# declarations. If set to NO, these declarations will be included in the +# documentation. +# The default value is: NO. + +HIDE_FRIEND_COMPOUNDS = NO + +# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any +# documentation blocks found inside the body of a function. If set to NO, these +# blocks will be appended to the function's detailed documentation block. +# The default value is: NO. + +HIDE_IN_BODY_DOCS = NO + +# The INTERNAL_DOCS tag determines if documentation that is typed after a +# \internal command is included. If the tag is set to NO then the documentation +# will be excluded. Set it to YES to include the internal documentation. +# The default value is: NO. + +INTERNAL_DOCS = YES + +# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file +# names in lower-case letters. If set to YES, upper-case letters are also +# allowed. This is useful if you have classes or files whose names only differ +# in case and if your file system supports case sensitive file names. Windows +# (including Cygwin) and Mac users are advised to set this option to NO. +# The default value is: system dependent. + +CASE_SENSE_NAMES = YES + +# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with +# their full class and namespace scopes in the documentation. If set to YES, the +# scope will be hidden. +# The default value is: NO. + +HIDE_SCOPE_NAMES = NO + +# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will +# append additional text to a page's title, such as Class Reference. If set to +# YES the compound reference will be hidden. +# The default value is: NO. + +HIDE_COMPOUND_REFERENCE= NO + +# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of +# the files that are included by a file in the documentation of that file. +# The default value is: YES. + +SHOW_INCLUDE_FILES = YES + +# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each +# grouped member an include statement to the documentation, telling the reader +# which file to include in order to use the member. +# The default value is: NO. + +SHOW_GROUPED_MEMB_INC = NO + +# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include +# files with double quotes in the documentation rather than with sharp brackets. +# The default value is: NO. + +FORCE_LOCAL_INCLUDES = NO + +# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the +# documentation for inline members. +# The default value is: YES. + +INLINE_INFO = YES + +# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the +# (detailed) documentation of file and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. +# The default value is: YES. + +SORT_MEMBER_DOCS = NO + +# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief +# descriptions of file, namespace and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. Note that +# this will also influence the order of the classes in the class list. +# The default value is: NO. + +SORT_BRIEF_DOCS = NO + +# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the +# (brief and detailed) documentation of class members so that constructors and +# destructors are listed first. If set to NO the constructors will appear in the +# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS. +# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief +# member documentation. +# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting +# detailed member documentation. +# The default value is: NO. + +SORT_MEMBERS_CTORS_1ST = NO + +# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy +# of group names into alphabetical order. If set to NO the group names will +# appear in their defined order. +# The default value is: NO. + +SORT_GROUP_NAMES = NO + +# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by +# fully-qualified names, including namespaces. If set to NO, the class list will +# be sorted only by class name, not including the namespace part. +# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. +# Note: This option applies only to the class list, not to the alphabetical +# list. +# The default value is: NO. + +SORT_BY_SCOPE_NAME = NO + +# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper +# type resolution of all parameters of a function it will reject a match between +# the prototype and the implementation of a member function even if there is +# only one candidate or it is obvious which candidate to choose by doing a +# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still +# accept a match between prototype and implementation in such cases. +# The default value is: NO. + +STRICT_PROTO_MATCHING = NO + +# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo +# list. This list is created by putting \todo commands in the documentation. +# The default value is: YES. + +GENERATE_TODOLIST = YES + +# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test +# list. This list is created by putting \test commands in the documentation. +# The default value is: YES. + +GENERATE_TESTLIST = YES + +# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug +# list. This list is created by putting \bug commands in the documentation. +# The default value is: YES. + +GENERATE_BUGLIST = YES + +# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO) +# the deprecated list. This list is created by putting \deprecated commands in +# the documentation. +# The default value is: YES. + +GENERATE_DEPRECATEDLIST= YES + +# The ENABLED_SECTIONS tag can be used to enable conditional documentation +# sections, marked by \if ... \endif and \cond +# ... \endcond blocks. + +ENABLED_SECTIONS = + +# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the +# initial value of a variable or macro / define can have for it to appear in the +# documentation. If the initializer consists of more lines than specified here +# it will be hidden. Use a value of 0 to hide initializers completely. The +# appearance of the value of individual variables and macros / defines can be +# controlled using \showinitializer or \hideinitializer command in the +# documentation regardless of this setting. +# Minimum value: 0, maximum value: 10000, default value: 30. + +MAX_INITIALIZER_LINES = 30 + +# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at +# the bottom of the documentation of classes and structs. If set to YES, the +# list will mention the files that were used to generate the documentation. +# The default value is: YES. + +SHOW_USED_FILES = YES + +# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This +# will remove the Files entry from the Quick Index and from the Folder Tree View +# (if specified). +# The default value is: YES. + +SHOW_FILES = YES + +# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces +# page. This will remove the Namespaces entry from the Quick Index and from the +# Folder Tree View (if specified). +# The default value is: YES. + +SHOW_NAMESPACES = YES + +# The FILE_VERSION_FILTER tag can be used to specify a program or script that +# doxygen should invoke to get the current version for each file (typically from +# the version control system). Doxygen will invoke the program by executing (via +# popen()) the command command input-file, where command is the value of the +# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided +# by doxygen. Whatever the program writes to standard output is used as the file +# version. For an example see the documentation. + +FILE_VERSION_FILTER = + +# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed +# by doxygen. The layout file controls the global structure of the generated +# output files in an output format independent way. To create the layout file +# that represents doxygen's defaults, run doxygen with the -l option. You can +# optionally specify a file name after the option, if omitted DoxygenLayout.xml +# will be used as the name of the layout file. +# +# Note that if you run doxygen from a directory containing a file called +# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE +# tag is left empty. + +LAYOUT_FILE = + +# The CITE_BIB_FILES tag can be used to specify one or more bib files containing +# the reference definitions. This must be a list of .bib files. The .bib +# extension is automatically appended if omitted. This requires the bibtex tool +# to be installed. See also https://en.wikipedia.org/wiki/BibTeX for more info. +# For LaTeX the style of the bibliography can be controlled using +# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the +# search path. See also \cite for info how to create references. + +CITE_BIB_FILES = + +#--------------------------------------------------------------------------- +# Configuration options related to warning and progress messages +#--------------------------------------------------------------------------- + +# The QUIET tag can be used to turn on/off the messages that are generated to +# standard output by doxygen. If QUIET is set to YES this implies that the +# messages are off. +# The default value is: NO. + +QUIET = YES + +# The WARNINGS tag can be used to turn on/off the warning messages that are +# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES +# this implies that the warnings are on. +# +# Tip: Turn warnings on while writing the documentation. +# The default value is: YES. + +WARNINGS = YES + +# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate +# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag +# will automatically be disabled. +# The default value is: YES. + +WARN_IF_UNDOCUMENTED = NO + +# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for +# potential errors in the documentation, such as not documenting some parameters +# in a documented function, or documenting parameters that don't exist or using +# markup commands wrongly. +# The default value is: YES. + +WARN_IF_DOC_ERROR = YES + +# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that +# are documented, but have no documentation for their parameters or return +# value. If set to NO, doxygen will only warn about wrong or incomplete +# parameter documentation, but not about the absence of documentation. If +# EXTRACT_ALL is set to YES then this flag will automatically be disabled. +# The default value is: NO. + +WARN_NO_PARAMDOC = NO + +# If the WARN_AS_ERROR tag is set to YES then doxygen will immediately stop when +# a warning is encountered. +# The default value is: NO. + +WARN_AS_ERROR = NO + +# The WARN_FORMAT tag determines the format of the warning messages that doxygen +# can produce. The string should contain the $file, $line, and $text tags, which +# will be replaced by the file and line number from which the warning originated +# and the warning text. Optionally the format may contain $version, which will +# be replaced by the version of the file (if it could be obtained via +# FILE_VERSION_FILTER) +# The default value is: $file:$line: $text. + +WARN_FORMAT = "$file:$line: $text" + +# The WARN_LOGFILE tag can be used to specify a file to which warning and error +# messages should be written. If left blank the output is written to standard +# error (stderr). + +WARN_LOGFILE = + +#--------------------------------------------------------------------------- +# Configuration options related to the input files +#--------------------------------------------------------------------------- + +# The INPUT tag is used to specify the files and/or directories that contain +# documented source files. You may enter file names like myfile.cpp or +# directories like /usr/src/myproject. Separate the files or directories with +# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING +# Note: If this tag is empty the current directory is searched. + +INPUT = + +# This tag can be used to specify the character encoding of the source files +# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses +# libiconv (or the iconv built into libc) for the transcoding. See the libiconv +# documentation (see: https://www.gnu.org/software/libiconv/) for the list of +# possible encodings. +# The default value is: UTF-8. + +INPUT_ENCODING = UTF-8 + +# If the value of the INPUT tag contains directories, you can use the +# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and +# *.h) to filter out the source-files in the directories. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# read by doxygen. +# +# If left blank the following patterns are tested:*.c, *.cc, *.cxx, *.cpp, +# *.c++, *.java, *.ii, *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, +# *.hh, *.hxx, *.hpp, *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, +# *.m, *.markdown, *.md, *.mm, *.dox (to be provided as doxygen C comment), +# *.doc (to be provided as doxygen C comment), *.txt (to be provided as doxygen +# C comment), *.py, *.pyw, *.f90, *.f95, *.f03, *.f08, *.f18, *.f, *.for, *.vhd, +# *.vhdl, *.ucf, *.qsf and *.ice. + +FILE_PATTERNS = xxhash.h +# +#FILE_PATTERNS = *.c \ +# *.cc \ +# *.cxx \ +# *.cpp \ +# *.c++ \ +# *.java \ +# *.ii \ +# *.ixx \ +# *.ipp \ +# *.i++ \ +# *.inl \ +# *.idl \ +# *.ddl \ +# *.odl \ +# *.h \ +# *.hh \ +# *.hxx \ +# *.hpp \ +# *.h++ \ +# *.cs \ +# *.d \ +# *.php \ +# *.php4 \ +# *.php5 \ +# *.phtml \ +# *.inc \ +# *.m \ +# *.markdown \ +# *.md \ +# *.mm \ +# *.dox \ +# *.doc \ +# *.txt \ +# *.py \ +# *.pyw \ +# *.f90 \ +# *.f95 \ +# *.f03 \ +# *.f08 \ +# *.f18 \ +# *.f \ +# *.for \ +# *.vhd \ +# *.vhdl \ +# *.ucf \ +# *.qsf \ +# *.ice + +# The RECURSIVE tag can be used to specify whether or not subdirectories should +# be searched for input files as well. +# The default value is: NO. + +RECURSIVE = NO + +# The EXCLUDE tag can be used to specify files and/or directories that should be +# excluded from the INPUT source files. This way you can easily exclude a +# subdirectory from a directory tree whose root is specified with the INPUT tag. +# +# Note that relative paths are relative to the directory from which doxygen is +# run. + +EXCLUDE = + +# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or +# directories that are symbolic links (a Unix file system feature) are excluded +# from the input. +# The default value is: NO. + +EXCLUDE_SYMLINKS = NO + +# If the value of the INPUT tag contains directories, you can use the +# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude +# certain files from those directories. +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories for example use the pattern */test/* + +EXCLUDE_PATTERNS = + +# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names +# (namespaces, classes, functions, etc.) that should be excluded from the +# output. The symbol name can be a fully qualified name, a word, or if the +# wildcard * is used, a substring. Examples: ANamespace, AClass, +# AClass::ANamespace, ANamespace::*Test +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories use the pattern */test/* + +EXCLUDE_SYMBOLS = + +# The EXAMPLE_PATH tag can be used to specify one or more files or directories +# that contain example code fragments that are included (see the \include +# command). + +EXAMPLE_PATH = + +# If the value of the EXAMPLE_PATH tag contains directories, you can use the +# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and +# *.h) to filter out the source-files in the directories. If left blank all +# files are included. + +EXAMPLE_PATTERNS = * + +# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be +# searched for input files to be used with the \include or \dontinclude commands +# irrespective of the value of the RECURSIVE tag. +# The default value is: NO. + +EXAMPLE_RECURSIVE = NO + +# The IMAGE_PATH tag can be used to specify one or more files or directories +# that contain images that are to be included in the documentation (see the +# \image command). + +IMAGE_PATH = + +# The INPUT_FILTER tag can be used to specify a program that doxygen should +# invoke to filter for each input file. Doxygen will invoke the filter program +# by executing (via popen()) the command: +# +# +# +# where is the value of the INPUT_FILTER tag, and is the +# name of an input file. Doxygen will then use the output that the filter +# program writes to standard output. If FILTER_PATTERNS is specified, this tag +# will be ignored. +# +# Note that the filter must not add or remove lines; it is applied before the +# code is scanned, but not when the output code is generated. If lines are added +# or removed, the anchors will not be placed correctly. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +INPUT_FILTER = + +# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern +# basis. Doxygen will compare the file name with each pattern and apply the +# filter if there is a match. The filters are a list of the form: pattern=filter +# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how +# filters are used. If the FILTER_PATTERNS tag is empty or if none of the +# patterns match the file name, INPUT_FILTER is applied. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +FILTER_PATTERNS = + +# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using +# INPUT_FILTER) will also be used to filter the input files that are used for +# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES). +# The default value is: NO. + +FILTER_SOURCE_FILES = NO + +# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file +# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and +# it is also possible to disable source filtering for a specific pattern using +# *.ext= (so without naming a filter). +# This tag requires that the tag FILTER_SOURCE_FILES is set to YES. + +FILTER_SOURCE_PATTERNS = + +# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that +# is part of the input, its contents will be placed on the main page +# (index.html). This can be useful if you have a project on for instance GitHub +# and want to reuse the introduction page also for the doxygen output. + +USE_MDFILE_AS_MAINPAGE = + +#--------------------------------------------------------------------------- +# Configuration options related to source browsing +#--------------------------------------------------------------------------- + +# If the SOURCE_BROWSER tag is set to YES then a list of source files will be +# generated. Documented entities will be cross-referenced with these sources. +# +# Note: To get rid of all source code in the generated output, make sure that +# also VERBATIM_HEADERS is set to NO. +# The default value is: NO. + +SOURCE_BROWSER = NO + +# Setting the INLINE_SOURCES tag to YES will include the body of functions, +# classes and enums directly into the documentation. +# The default value is: NO. + +INLINE_SOURCES = NO + +# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any +# special comment blocks from generated source code fragments. Normal C, C++ and +# Fortran comments will always remain visible. +# The default value is: YES. + +STRIP_CODE_COMMENTS = YES + +# If the REFERENCED_BY_RELATION tag is set to YES then for each documented +# entity all documented functions referencing it will be listed. +# The default value is: NO. + +REFERENCED_BY_RELATION = NO + +# If the REFERENCES_RELATION tag is set to YES then for each documented function +# all documented entities called/used by that function will be listed. +# The default value is: NO. + +REFERENCES_RELATION = NO + +# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set +# to YES then the hyperlinks from functions in REFERENCES_RELATION and +# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will +# link to the documentation. +# The default value is: YES. + +REFERENCES_LINK_SOURCE = YES + +# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the +# source code will show a tooltip with additional information such as prototype, +# brief description and links to the definition and documentation. Since this +# will make the HTML file larger and loading of large files a bit slower, you +# can opt to disable this feature. +# The default value is: YES. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +SOURCE_TOOLTIPS = YES + +# If the USE_HTAGS tag is set to YES then the references to source code will +# point to the HTML generated by the htags(1) tool instead of doxygen built-in +# source browser. The htags tool is part of GNU's global source tagging system +# (see https://www.gnu.org/software/global/global.html). You will need version +# 4.8.6 or higher. +# +# To use it do the following: +# - Install the latest version of global +# - Enable SOURCE_BROWSER and USE_HTAGS in the configuration file +# - Make sure the INPUT points to the root of the source tree +# - Run doxygen as normal +# +# Doxygen will invoke htags (and that will in turn invoke gtags), so these +# tools must be available from the command line (i.e. in the search path). +# +# The result: instead of the source browser generated by doxygen, the links to +# source code will now point to the output of htags. +# The default value is: NO. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +USE_HTAGS = NO + +# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a +# verbatim copy of the header file for each class for which an include is +# specified. Set to NO to disable this. +# See also: Section \class. +# The default value is: YES. + +VERBATIM_HEADERS = YES + +#--------------------------------------------------------------------------- +# Configuration options related to the alphabetical class index +#--------------------------------------------------------------------------- + +# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all +# compounds will be generated. Enable this if the project contains a lot of +# classes, structs, unions or interfaces. +# The default value is: YES. + +ALPHABETICAL_INDEX = YES + +# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in +# which the alphabetical index list will be split. +# Minimum value: 1, maximum value: 20, default value: 5. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +COLS_IN_ALPHA_INDEX = 5 + +# In case all classes in a project start with a common prefix, all classes will +# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag +# can be used to specify a prefix (or a list of prefixes) that should be ignored +# while generating the index headers. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +IGNORE_PREFIX = + +#--------------------------------------------------------------------------- +# Configuration options related to the HTML output +#--------------------------------------------------------------------------- + +# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output +# The default value is: YES. + +GENERATE_HTML = YES + +# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a +# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of +# it. +# The default directory is: html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_OUTPUT = html + +# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each +# generated HTML page (for example: .htm, .php, .asp). +# The default value is: .html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FILE_EXTENSION = .html + +# The HTML_HEADER tag can be used to specify a user-defined HTML header file for +# each generated HTML page. If the tag is left blank doxygen will generate a +# standard header. +# +# To get valid HTML the header file that includes any scripts and style sheets +# that doxygen needs, which is dependent on the configuration options used (e.g. +# the setting GENERATE_TREEVIEW). It is highly recommended to start with a +# default header using +# doxygen -w html new_header.html new_footer.html new_stylesheet.css +# YourConfigFile +# and then modify the file new_header.html. See also section "Doxygen usage" +# for information on how to generate the default header that doxygen normally +# uses. +# Note: The header is subject to change so you typically have to regenerate the +# default header when upgrading to a newer version of doxygen. For a description +# of the possible markers and block names see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_HEADER = + +# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each +# generated HTML page. If the tag is left blank doxygen will generate a standard +# footer. See HTML_HEADER for more information on how to generate a default +# footer and what special commands can be used inside the footer. See also +# section "Doxygen usage" for information on how to generate the default footer +# that doxygen normally uses. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FOOTER = + +# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style +# sheet that is used by each HTML page. It can be used to fine-tune the look of +# the HTML output. If left blank doxygen will generate a default style sheet. +# See also section "Doxygen usage" for information on how to generate the style +# sheet that doxygen normally uses. +# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as +# it is more robust and this tag (HTML_STYLESHEET) will in the future become +# obsolete. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_STYLESHEET = + +# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined +# cascading style sheets that are included after the standard style sheets +# created by doxygen. Using this option one can overrule certain style aspects. +# This is preferred over using HTML_STYLESHEET since it does not replace the +# standard style sheet and is therefore more robust against future updates. +# Doxygen will copy the style sheet files to the output directory. +# Note: The order of the extra style sheet files is of importance (e.g. the last +# style sheet in the list overrules the setting of the previous ones in the +# list). For an example see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_STYLESHEET = + +# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or +# other source files which should be copied to the HTML output directory. Note +# that these files will be copied to the base HTML output directory. Use the +# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these +# files. In the HTML_STYLESHEET file, use the file name only. Also note that the +# files will be copied as-is; there are no commands or markers available. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_FILES = + +# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen +# will adjust the colors in the style sheet and background images according to +# this color. Hue is specified as an angle on a colorwheel, see +# https://en.wikipedia.org/wiki/Hue for more information. For instance the value +# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300 +# purple, and 360 is red again. +# Minimum value: 0, maximum value: 359, default value: 220. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_HUE = 220 + +# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors +# in the HTML output. For a value of 0 the output will use grayscales only. A +# value of 255 will produce the most vivid colors. +# Minimum value: 0, maximum value: 255, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_SAT = 100 + +# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the +# luminance component of the colors in the HTML output. Values below 100 +# gradually make the output lighter, whereas values above 100 make the output +# darker. The value divided by 100 is the actual gamma applied, so 80 represents +# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not +# change the gamma. +# Minimum value: 40, maximum value: 240, default value: 80. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_GAMMA = 100 + +# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML +# page will contain the date and time when the page was generated. Setting this +# to YES can help to show when doxygen was last run and thus if the +# documentation is up to date. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_TIMESTAMP = NO + +# If the HTML_DYNAMIC_MENUS tag is set to YES then the generated HTML +# documentation will contain a main index with vertical navigation menus that +# are dynamically created via JavaScript. If disabled, the navigation index will +# consists of multiple levels of tabs that are statically embedded in every HTML +# page. Disable this option to support browsers that do not have JavaScript, +# like the Qt help browser. +# The default value is: YES. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_DYNAMIC_MENUS = YES + +# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML +# documentation will contain sections that can be hidden and shown after the +# page has loaded. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_DYNAMIC_SECTIONS = YES + +# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries +# shown in the various tree structured indices initially; the user can expand +# and collapse entries dynamically later on. Doxygen will expand the tree to +# such a level that at most the specified number of entries are visible (unless +# a fully collapsed tree already exceeds this amount). So setting the number of +# entries 1 will produce a full collapsed tree by default. 0 is a special value +# representing an infinite number of entries and will result in a full expanded +# tree by default. +# Minimum value: 0, maximum value: 9999, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_INDEX_NUM_ENTRIES = 100 + +# If the GENERATE_DOCSET tag is set to YES, additional index files will be +# generated that can be used as input for Apple's Xcode 3 integrated development +# environment (see: https://developer.apple.com/xcode/), introduced with OSX +# 10.5 (Leopard). To create a documentation set, doxygen will generate a +# Makefile in the HTML output directory. Running make will produce the docset in +# that directory and running make install will install the docset in +# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at +# startup. See https://developer.apple.com/library/archive/featuredarticles/Doxy +# genXcode/_index.html for more information. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_DOCSET = NO + +# This tag determines the name of the docset feed. A documentation feed provides +# an umbrella under which multiple documentation sets from a single provider +# (such as a company or product suite) can be grouped. +# The default value is: Doxygen generated docs. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_FEEDNAME = "Doxygen generated docs" + +# This tag specifies a string that should uniquely identify the documentation +# set bundle. This should be a reverse domain-name style string, e.g. +# com.mycompany.MyDocSet. Doxygen will append .docset to the name. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_BUNDLE_ID = org.doxygen.Project + +# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify +# the documentation publisher. This should be a reverse domain-name style +# string, e.g. com.mycompany.MyDocSet.documentation. +# The default value is: org.doxygen.Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_ID = org.doxygen.Publisher + +# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher. +# The default value is: Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_NAME = Publisher + +# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three +# additional HTML index files: index.hhp, index.hhc, and index.hhk. The +# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop +# (see: https://www.microsoft.com/en-us/download/details.aspx?id=21138) on +# Windows. +# +# The HTML Help Workshop contains a compiler that can convert all HTML output +# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML +# files are now used as the Windows 98 help format, and will replace the old +# Windows help format (.hlp) on all Windows platforms in the future. Compressed +# HTML files also contain an index, a table of contents, and you can search for +# words in the documentation. The HTML workshop also contains a viewer for +# compressed HTML files. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_HTMLHELP = NO + +# The CHM_FILE tag can be used to specify the file name of the resulting .chm +# file. You can add a path in front of the file if the result should not be +# written to the html output directory. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_FILE = + +# The HHC_LOCATION tag can be used to specify the location (absolute path +# including file name) of the HTML help compiler (hhc.exe). If non-empty, +# doxygen will try to run the HTML help compiler on the generated index.hhp. +# The file has to be specified with full path. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +HHC_LOCATION = + +# The GENERATE_CHI flag controls if a separate .chi index file is generated +# (YES) or that it should be included in the main .chm file (NO). +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +GENERATE_CHI = NO + +# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc) +# and project file content. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_INDEX_ENCODING = + +# The BINARY_TOC flag controls whether a binary table of contents is generated +# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it +# enables the Previous and Next buttons. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +BINARY_TOC = NO + +# The TOC_EXPAND flag can be set to YES to add extra items for group members to +# the table of contents of the HTML help documentation and to the tree view. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +TOC_EXPAND = NO + +# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and +# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that +# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help +# (.qch) of the generated HTML documentation. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_QHP = NO + +# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify +# the file name of the resulting .qch file. The path specified is relative to +# the HTML output folder. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QCH_FILE = + +# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help +# Project output. For more information please see Qt Help Project / Namespace +# (see: https://doc.qt.io/archives/qt-4.8/qthelpproject.html#namespace). +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_NAMESPACE = org.doxygen.Project + +# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt +# Help Project output. For more information please see Qt Help Project / Virtual +# Folders (see: https://doc.qt.io/archives/qt-4.8/qthelpproject.html#virtual- +# folders). +# The default value is: doc. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_VIRTUAL_FOLDER = doc + +# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom +# filter to add. For more information please see Qt Help Project / Custom +# Filters (see: https://doc.qt.io/archives/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_NAME = + +# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the +# custom filter to add. For more information please see Qt Help Project / Custom +# Filters (see: https://doc.qt.io/archives/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_ATTRS = + +# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this +# project's filter section matches. Qt Help Project / Filter Attributes (see: +# https://doc.qt.io/archives/qt-4.8/qthelpproject.html#filter-attributes). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_SECT_FILTER_ATTRS = + +# The QHG_LOCATION tag can be used to specify the location of Qt's +# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the +# generated .qhp file. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHG_LOCATION = + +# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be +# generated, together with the HTML files, they form an Eclipse help plugin. To +# install this plugin and make it available under the help contents menu in +# Eclipse, the contents of the directory containing the HTML and XML files needs +# to be copied into the plugins directory of eclipse. The name of the directory +# within the plugins directory should be the same as the ECLIPSE_DOC_ID value. +# After copying Eclipse needs to be restarted before the help appears. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_ECLIPSEHELP = NO + +# A unique identifier for the Eclipse help plugin. When installing the plugin +# the directory name containing the HTML and XML files should also have this +# name. Each documentation set should have its own identifier. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES. + +ECLIPSE_DOC_ID = org.doxygen.Project + +# If you want full control over the layout of the generated HTML pages it might +# be necessary to disable the index and replace it with your own. The +# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top +# of each HTML page. A value of NO enables the index and the value YES disables +# it. Since the tabs in the index contain the same information as the navigation +# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +DISABLE_INDEX = NO + +# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index +# structure should be generated to display hierarchical information. If the tag +# value is set to YES, a side panel will be generated containing a tree-like +# index structure (just like the one that is generated for HTML Help). For this +# to work a browser that supports JavaScript, DHTML, CSS and frames is required +# (i.e. any modern browser). Windows users are probably better off using the +# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can +# further fine-tune the look of the index. As an example, the default style +# sheet generated by doxygen has an example that shows how to put an image at +# the root of the tree instead of the PROJECT_NAME. Since the tree basically has +# the same information as the tab index, you could consider setting +# DISABLE_INDEX to YES when enabling this option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_TREEVIEW = YES + +# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that +# doxygen will group on one line in the generated HTML documentation. +# +# Note that a value of 0 will completely suppress the enum values from appearing +# in the overview section. +# Minimum value: 0, maximum value: 20, default value: 4. +# This tag requires that the tag GENERATE_HTML is set to YES. + +ENUM_VALUES_PER_LINE = 4 + +# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used +# to set the initial width (in pixels) of the frame in which the tree is shown. +# Minimum value: 0, maximum value: 1500, default value: 250. +# This tag requires that the tag GENERATE_HTML is set to YES. + +TREEVIEW_WIDTH = 250 + +# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to +# external symbols imported via tag files in a separate window. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +EXT_LINKS_IN_WINDOW = NO + +# If the HTML_FORMULA_FORMAT option is set to svg, doxygen will use the pdf2svg +# tool (see https://github.com/dawbarton/pdf2svg) or inkscape (see +# https://inkscape.org) to generate formulas as SVG images instead of PNGs for +# the HTML output. These images will generally look nicer at scaled resolutions. +# Possible values are: png (the default) and svg (looks nicer but requires the +# pdf2svg or inkscape tool). +# The default value is: png. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FORMULA_FORMAT = png + +# Use this tag to change the font size of LaTeX formulas included as images in +# the HTML documentation. When you change the font size after a successful +# doxygen run you need to manually remove any form_*.png images from the HTML +# output directory to force them to be regenerated. +# Minimum value: 8, maximum value: 50, default value: 10. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_FONTSIZE = 10 + +# Use the FORMULA_TRANSPARENT tag to determine whether or not the images +# generated for formulas are transparent PNGs. Transparent PNGs are not +# supported properly for IE 6.0, but are supported on all modern browsers. +# +# Note that when changing this option you need to delete any form_*.png files in +# the HTML output directory before the changes have effect. +# The default value is: YES. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_TRANSPARENT = YES + +# The FORMULA_MACROFILE can contain LaTeX \newcommand and \renewcommand commands +# to create new LaTeX commands to be used in formulas as building blocks. See +# the section "Including formulas" for details. + +FORMULA_MACROFILE = + +# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see +# https://www.mathjax.org) which uses client side JavaScript for the rendering +# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX +# installed or if you want to formulas look prettier in the HTML output. When +# enabled you may also need to install MathJax separately and configure the path +# to it using the MATHJAX_RELPATH option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +USE_MATHJAX = NO + +# When MathJax is enabled you can set the default output format to be used for +# the MathJax output. See the MathJax site (see: +# http://docs.mathjax.org/en/latest/output.html) for more details. +# Possible values are: HTML-CSS (which is slower, but has the best +# compatibility), NativeMML (i.e. MathML) and SVG. +# The default value is: HTML-CSS. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_FORMAT = HTML-CSS + +# When MathJax is enabled you need to specify the location relative to the HTML +# output directory using the MATHJAX_RELPATH option. The destination directory +# should contain the MathJax.js script. For instance, if the mathjax directory +# is located at the same level as the HTML output directory, then +# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax +# Content Delivery Network so you can quickly see the result without installing +# MathJax. However, it is strongly recommended to install a local copy of +# MathJax from https://www.mathjax.org before deployment. +# The default value is: https://cdn.jsdelivr.net/npm/mathjax@2. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_RELPATH = https://cdn.jsdelivr.net/npm/mathjax@2 + +# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax +# extension names that should be enabled during MathJax rendering. For example +# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_EXTENSIONS = + +# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces +# of code that will be used on startup of the MathJax code. See the MathJax site +# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an +# example see the documentation. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_CODEFILE = + +# When the SEARCHENGINE tag is enabled doxygen will generate a search box for +# the HTML output. The underlying search engine uses javascript and DHTML and +# should work on any modern browser. Note that when using HTML help +# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET) +# there is already a search function so this one should typically be disabled. +# For large projects the javascript based search engine can be slow, then +# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to +# search using the keyboard; to jump to the search box use + S +# (what the is depends on the OS and browser, but it is typically +# , /