Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks #112

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ set_target_properties(clar PROPERTIES
C_EXTENSIONS OFF
)

if(NOT WIN32)
set(CLAR_LIBRARIES m)
endif()

if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME)
include(CTest)
if(BUILD_TESTING)
Expand Down
112 changes: 96 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,29 @@
Come out and Clar
=================

In Catalan, "clar" means clear, easy to perceive. Using clar will make it
easy to test and make clear the quality of your code.
Clar is a minimal C unit testing and benchmarking framework. It
provides a simple mechanism for writing tests and asserting
postconditions, while providing support for TAP and JUnit style
outputs.

> _Historical note_
>
> Originally the clar project was named "clay" because the word "test" has its
> roots in the latin word *"testum"*, meaning "earthen pot", and *"testa"*,
> meaning "piece of burned clay"?
>
> This is because historically, testing implied melting metal in a pot to
> check its quality. Clay is what tests are made of.
In Catalan, "clar" means clear, easy to perceive. Using Clar will make it
easy to test and make clear the quality of your code.

## Quick Usage Overview

Clar is a minimal C unit testing framework. It's been written to replace the
old framework in [libgit2][libgit2], but it's both very versatile and
straightforward to use.

Can you count to funk?

- **Zero: Initialize test directory**
1. **Initialize test directory**

~~~~ sh
$ mkdir tests
$ cp -r $CLAR_ROOT/clar* tests
$ cp $CLAR_ROOT/example/*.c tests
~~~~

- **One: Write some tests**
2. **Write some tests**

File: tests/adding.c:

Expand Down Expand Up @@ -59,7 +53,7 @@ Can you count to funk?
}
~~~~~

- **Two: Build the test executable**
3. **Build the test executable**

~~~~ sh
$ cd tests
Expand All @@ -68,7 +62,7 @@ Can you count to funk?
$ gcc -I. clar.c main.c adding.c -o testit
~~~~

- **Funk: Funk it.**
4. **Run the tests**

~~~~ sh
$ ./testit
Expand Down Expand Up @@ -319,6 +313,92 @@ void test_example__a_test_with_auxiliary_methods(void)
}
~~~~

## Benchmarks

The clar mixer (`generate.py`) and runner can also be used to support
simple benchmark capabilities. When running in benchmark mode, Clar
will run each test multiple times in succession, using a high-resolution
platform timer to measure the elapsed time of each run.

By default, Clar will run each test repeatedly for 3 seconds (with
a minimum of 10 runs), but you can define the explicit number of
runs for each test in the definition.

By default, Clar will run the initialization and cleanup functions
before _each_ test run. This allows for consistent setup and teardown
behavior, and predictability with existing test setups. However, you
can avoid this additional overhead by defining a _reset_ function.
This will be called between test runs instead of the cleanup and
re-initialization; in this case, initialization will occur only
before all test runs, and cleanup will be performed only when all
test runs are complete.

To configure a benchmark application instead of a test application:

1. **Set clar into benchmark mode in your main function**

~~~~ c
int main(int argc, char *argv[])
{
clar_test_set_mode(CL_TEST_BENCHMARK);
clar_test_init(argc, argv);
res = clar_test_run();
clar_test_shutdown();
return res;
}
~~~~

2. **Optionally, set up your initialization, cleanup, and reset
functions**

~~~~ c
void test_foo__initialize(void)
{
global_data = malloc(1024 * 1024 * 1024);
memset(global_data, 0, 1024 * 1024 * 1024);
}

void test_foo__reset(void)
{
memset(global_data, 0, 1024 * 1024 * 1024);
}

void test_foo__cleanup(void)
{
global_data = malloc(1024 * 1024 * 1024);
}
~~~~

3. **Optionally, configure tests with a specific run number**

~~~~ c
/* Run this test 500 times */
void test_foo__bar(void)
/* [clar]:runs=500 */
{
bar();
}
~~~~

3. **Run the benchmarks**

When running in benchmark mode, you'll see timings output; if you
write a summary file, it will be a JSON file that contains the
time information.

~~~~ sh
$ ./benchmarks -r/path/to/results.json
Started benchmarks (mean time ± stddev / min time … max time):

foo::bar: 24.75 ms ± 1.214 ms / range: 24.41 ms … 38.06 ms (500 runs)
foo::baz: 24.67 ms ± 248.2 μs / range: 24.41 ms … 25.41 ms (478 runs)
foo::qux: 25.98 ms ± 333.0 μs / range: 25.64 ms … 26.82 ms (112 runs)
~~~~

Note: you can change the prefix of the test function names from `test_`
to something of your choice by using the `--prefix=...` option for
the `generate.py` mixer script.

About Clar
==========

Expand Down
Loading