Skip to content

Commit

Permalink
Add CMake build system for valkey (#1196)
Browse files Browse the repository at this point in the history
With this commit, users are able to build valkey using `CMake`.

## Example usage:

Build `valkey-server` in Release mode with TLS enabled and using
`jemalloc` as the allocator:

```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_INSTALL_PREFIX=/tmp/valkey-install \
         -DBUILD_MALLOC=jemalloc -DBUILD_TLS=1
make -j$(nproc) install

# start valkey
/tmp/valkey-install/bin/valkey-server
```

Build `valkey-unit-tests`:

```bash
mkdir build-release-ut
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DBUILD_MALLOC=jemalloc -DBUILD_UNIT_TESTS=1
make -j$(nproc)

# Run the tests
./bin/valkey-unit-tests 
```

Current features supported by this PR:

- Building against different allocators: (`jemalloc`, `tcmalloc`,
`tcmalloc_minimal` and `libc`), e.g. to enable `jemalloc` pass
`-DBUILD_MALLOC=jemalloc` to `cmake`
- OpenSSL builds (to enable TLS, pass `-DBUILD_TLS=1` to `cmake`)
- Sanitizier: pass `-DBUILD_SANITIZER=<address|thread|undefined>` to
`cmake`
- Install target + redis symbolic links
- Build `valkey-unit-tests` executable
- Standard CMake variables are supported. e.g. to install `valkey` under
`/home/you/root` pass `-DCMAKE_INSTALL_PREFIX=/home/you/root`

Why using `CMake`? To list *some* of the advantages of using `CMake`:

- Superior IDE integrations: cmake generates the file
`compile_commands.json` which is required by `clangd` to get a compiler
accuracy code completion (in other words: your VScode will thank you)
- Out of the source build tree: with the current build system, object
files are created all over the place polluting the build source tree,
the best practice is to build the project on a separate folder
- Multiple build types co-existing: with the current build system, it is
often hard to have multiple build configurations. With cmake you can do
it easily:
- It is the de-facto standard for C/C++ project these days

More build examples: 

ASAN build:

```bash
mkdir build-asan
cd $_
cmake .. -DBUILD_SANITIZER=address -DBUILD_MALLOC=libc
make -j$(nproc)
```

ASAN with jemalloc:

```bash
mkdir build-asan-jemalloc
cd $_
cmake .. -DBUILD_SANITIZER=address -DBUILD_MALLOC=jemalloc 
make -j$(nproc)
```

As seen by the previous examples, any combination is allowed and
co-exist on the same source tree.

## Valkey installation

With this new `CMake`, it is possible to install the binary by running
`make install` or creating a package `make package` (currently supported
on Debian like distros)

### Example 1: build & install using `make install`:

```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/valkey-install -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) install
# valkey is now installed under $HOME/valkey-install
```

### Example 2: create a `.deb` installer:

```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) package
# ... CPack deb generation output
sudo gdebi -n ./valkey_8.1.0_amd64.deb
# valkey is now installed under /opt/valkey
```

### Example 3: create installer for non Debian systems (e.g. FreeBSD or
macOS):

```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) package
mkdir -p /opt/valkey && ./valkey-8.1.0-Darwin.sh --prefix=/opt/valkey  --exclude-subdir
# valkey-server is now installed under /opt/valkey

```

Signed-off-by: Eran Ifrah <[email protected]>
  • Loading branch information
eifrah-aws authored Nov 8, 2024
1 parent 3672f9b commit 07b3e7a
Show file tree
Hide file tree
Showing 22 changed files with 1,252 additions and 35 deletions.
76 changes: 76 additions & 0 deletions .cmake-format.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
format:
_help_line_width:
- How wide to allow formatted cmake files
line_width: 120
_help_tab_size:
- How many spaces to tab for indent
tab_size: 4
_help_use_tabchars:
- If true, lines are indented using tab characters (utf-8
- 0x09) instead of <tab_size> space characters (utf-8 0x20).
- In cases where the layout would require a fractional tab
- character, the behavior of the fractional indentation is
- governed by <fractional_tab_policy>
use_tabchars: false
_help_separate_ctrl_name_with_space:
- If true, separate flow control names from their parentheses
- with a space
separate_ctrl_name_with_space: true
_help_min_prefix_chars:
- If the statement spelling length (including space and
- parenthesis) is smaller than this amount, then force reject
- nested layouts.
min_prefix_chars: 4
_help_max_prefix_chars:
- If the statement spelling length (including space and
- parenthesis) is larger than the tab width by more than this
- amount, then force reject un-nested layouts.
max_prefix_chars: 10
_help_max_lines_hwrap:
- If a candidate layout is wrapped horizontally but it exceeds
- this many lines, then reject the layout.
max_lines_hwrap: 2
_help_line_ending:
- What style line endings to use in the output.
line_ending: unix
_help_command_case:
- Format command names consistently as 'lower' or 'upper' case
command_case: lower
_help_keyword_case:
- Format keywords consistently as 'lower' or 'upper' case
keyword_case: unchanged
_help_always_wrap:
- A list of command names which should always be wrapped
always_wrap: []
_help_enable_sort:
- If true, the argument lists which are known to be sortable
- will be sorted lexicographicall
enable_sort: true
_help_autosort:
- If true, the parsers may infer whether or not an argument
- list is sortable (without annotation).
autosort: false
_help_require_valid_layout:
- By default, if cmake-format cannot successfully fit
- everything into the desired linewidth it will apply the
- last, most agressive attempt that it made. If this flag is
- True, however, cmake-format will print error, exit with non-
- zero status code, and write-out nothing
require_valid_layout: false
_help_layout_passes:
- A dictionary mapping layout nodes to a list of wrap
- decisions. See the documentation for more information.
layout_passes: {}
encode:
_help_emit_byteorder_mark:
- If true, emit the unicode byte-order mark (BOM) at the start
- of the file
emit_byteorder_mark: false
_help_input_encoding:
- Specify the encoding of the input file. Defaults to utf-8
input_encoding: utf-8
_help_output_encoding:
- Specify the encoding of the output file. Defaults to utf-8.
- Note that cmake only claims to support utf-8 so be careful
- when using anything else
output_encoding: utf-8
25 changes: 25 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,31 @@ jobs:
run: |
./src/valkey-unit-tests
test-ubuntu-latest-cmake:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
- name: cmake and make
run: |
sudo apt-get install -y cmake libssl-dev
mkdir -p build-release
cd build-release
cmake -DCMAKE_BUILD_TYPE=Release .. -DBUILD_TLS=yes -DBUILD_UNIT_TESTS=yes
make -j$(nproc)
- name: test
run: |
sudo apt-get install -y tcl8.6 tclx
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-server
ln -sf $(pwd)/build-release/bin/valkey-cli $(pwd)/src/valkey-cli
ln -sf $(pwd)/build-release/bin/valkey-benchmark $(pwd)/src/valkey-benchmark
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-check-aof
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-check-rdb
ln -sf $(pwd)/build-release/bin/valkey-server $(pwd)/src/valkey-sentinel
./runtest --verbose --tags -slow --dump-logs
- name: unit tests
run: |
./build-release/bin/valkey-unit-tests
test-sanitizer-address:
runs-on: ubuntu-latest
steps:
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,5 @@ nodes*.conf
tests/cluster/tmp/*
tests/rdma/rdma-test
tags
build-debug/
build-release/
43 changes: 43 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
cmake_minimum_required(VERSION 3.20)

# Must be done first
if (APPLE)
# Force clang compiler on macOS
find_program(CLANGPP "clang++")
find_program(CLANG "clang")
if (CLANG AND CLANGPP)
message(STATUS "Found ${CLANGPP}, ${CLANG}")
set(CMAKE_CXX_COMPILER ${CLANGPP})
set(CMAKE_C_COMPILER ${CLANG})
endif ()
endif ()

# Options
option(BUILD_UNIT_TESTS "Build valkey-unit-tests" OFF)
option(BUILD_TEST_MODULES "Build all test modules" OFF)
option(BUILD_EXAMPLE_MODULES "Build example modules" OFF)

set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/")
project("valkey")

set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED ON)
set(CMAKE_C_EXTENSIONS ON)

include(ValkeySetup)
add_subdirectory(src)
add_subdirectory(tests)

# Include the packaging module
include(Packaging)

# Clear cached variables from the cache
unset(BUILD_TESTS CACHE)
unset(CLANGPP CACHE)
unset(CLANG CACHE)
unset(BUILD_RDMA_MODULE CACHE)
unset(BUILD_TLS_MODULE CACHE)
unset(BUILD_UNIT_TESTS CACHE)
unset(BUILD_TEST_MODULES CACHE)
unset(BUILD_EXAMPLE_MODULES CACHE)
unset(USE_TLS CACHE)
124 changes: 90 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@ This project was forked from the open source Redis project right before the tran

This README is just a fast *quick start* document. More details can be found under [valkey.io](https://valkey.io/)

What is Valkey?
--------------
# What is Valkey?

Valkey is a high-performance data structure server that primarily serves key/value workloads.
It supports a wide range of native structures and an extensible plugin system for adding new data structures and access patterns.

Building Valkey
--------------
# Building Valkey using `Makefile`

Valkey can be compiled and used on Linux, OSX, OpenBSD, NetBSD, FreeBSD.
We support big endian and little endian architectures, and both 32 bit
Expand Down Expand Up @@ -43,7 +42,7 @@ supports RDMA as connection module mode. Run:

% make BUILD_RDMA=module

To build with systemd support, you'll need systemd development libraries (such
To build with systemd support, you'll need systemd development libraries (such
as libsystemd-dev on Debian/Ubuntu or systemd-devel on CentOS) and run:

% make USE_SYSTEMD=yes
Expand Down Expand Up @@ -71,8 +70,7 @@ More about running the integration tests can be found in
[tests/README.md](tests/README.md) and for unit tests, see
[src/unit/README.md](src/unit/README.md).

Fixing build problems with dependencies or cached build options
---------
## Fixing build problems with dependencies or cached build options

Valkey has some dependencies which are included in the `deps` directory.
`make` does not automatically rebuild dependencies even if something in
Expand All @@ -91,8 +89,7 @@ optimizations (for debugging purposes), and other similar build time options,
those options are cached indefinitely until you issue a `make distclean`
command.

Fixing problems building 32 bit binaries
---------
## Fixing problems building 32 bit binaries

If after building Valkey with a 32 bit target you need to rebuild it
with a 64 bit target, or the other way around, you need to perform a
Expand All @@ -105,8 +102,7 @@ the following steps:
* Try using the following command line instead of `make 32bit`:
`make CFLAGS="-m32 -march=native" LDFLAGS="-m32"`

Allocator
---------
## Allocator

Selecting a non-default memory allocator when building Valkey is done by setting
the `MALLOC` environment variable. Valkey is compiled and linked against libc
Expand All @@ -122,28 +118,25 @@ To compile against jemalloc on Mac OS X systems, use:

% make MALLOC=jemalloc

Monotonic clock
---------------
## Monotonic clock

By default, Valkey will build using the POSIX clock_gettime function as the
monotonic clock source. On most modern systems, the internal processor clock
can be used to improve performance. Cautions can be found here:
can be used to improve performance. Cautions can be found here:
http://oliveryang.net/2015/09/pitfalls-of-TSC-usage/

To build with support for the processor's internal instruction clock, use:

% make CFLAGS="-DUSE_PROCESSOR_CLOCK"

Verbose build
-------------
## Verbose build

Valkey will build with a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:

% make V=1

Running Valkey
-------------
# Running Valkey

To run Valkey with the default configuration, just type:

Expand All @@ -165,10 +158,10 @@ as options using the command line. Examples:
All the options in valkey.conf are also supported as options using the command
line, with exactly the same name.

Running Valkey with TLS:
------------------
# Running Valkey with TLS:

## Running manually

### Running manually
To manually run a Valkey server with TLS mode (assuming `./gen-test-certs.sh` was invoked so sample certificates/keys are available):

* TLS built-in mode:
Expand Down Expand Up @@ -204,8 +197,7 @@ Specifying `--tls-replication yes` makes a replica connect to the primary.

Using `--tls-cluster yes` makes Valkey Cluster use TLS across nodes.

Running Valkey with RDMA:
------------------
# Running Valkey with RDMA:

Note that Valkey Over RDMA is an experimental feature.
It may be changed or removed in any minor or major version.
Expand Down Expand Up @@ -236,8 +228,7 @@ Or:
% ibv_devices


Playing with Valkey
------------------
# Playing with Valkey

You can use valkey-cli to play with Valkey. Start a valkey-server instance,
then in another terminal try the following:
Expand All @@ -256,8 +247,7 @@ then in another terminal try the following:
(integer) 2
valkey>

Installing Valkey
-----------------
# Installing Valkey

In order to install Valkey binaries into /usr/local/bin, just use:

Expand Down Expand Up @@ -289,16 +279,82 @@ system reboots.
You'll be able to stop and start Valkey using the script named
`/etc/init.d/valkey_<portnumber>`, for instance `/etc/init.d/valkey_6379`.

Code contributions
-----------------
# Building using `CMake`

In addition to the traditional `Makefile` build, Valkey supports an alternative, **experimental**, build system using `CMake`.

To build and install `Valkey`, in `Release` mode (an optimized build), type this into your terminal:

```bash
mkdir build-release
cd $_
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/opt/valkey
sudo make install
# Valkey is now installed under /opt/valkey
```

Other options supported by Valkey's `CMake` build system:

## Special build flags

- `-DBUILD_TLS=<on|off|module>` enable TLS build for Valkey
- `-DBUILD_RDMA=<off|module>` enable RDMA module build (only module mode supported)
- `-DBUILD_MALLOC=<libc|jemalloc|tcmalloc|tcmalloc_minimal>` choose the allocator to use. Default on Linux: `jemalloc`, for other OS: `libc`
- `-DBUILD_SANITIZER=<address|thread|undefined>` build with address sanitizer enabled
- `-DBUILD_UNIT_TESTS=[1|0]` when set, the build will produce the executable `valkey-unit-tests`
- `-DBUILD_TEST_MODULES=[1|0]` when set, the build will include the modules located under the `tests/modules` folder
- `-DBUILD_EXAMPLE_MODULES=[1|0]` when set, the build will include the example modules located under the `src/modules` folder

## Common flags

- `-DCMAKE_BUILD_TYPE=<Debug|Release...>` define the build type, see CMake manual for more details
- `-DCMAKE_INSTALL_PREFIX=/installation/path` override this value to define a custom install prefix. Default: `/usr/local`
- `-G<Generator Name>` generate build files for "Generator Name". By default, CMake will generate `Makefile`s.

## Verbose build

`CMake` generates a user-friendly colorized output by default.
If you want to see a more verbose output, use the following:

```bash
make VERBOSE=1
```

## Troubleshooting

During the `CMake` stage, `CMake` caches variables in a local file named `CMakeCache.txt`. All variables generated by Valkey
are removed from the cache once consumed (this is done by calling to `unset(VAR-NAME CACHE)`). However, some variables,
like the compiler path, are kept in cache. To start a fresh build either remove the cache file `CMakeCache.txt` from the
build folder, or delete the build folder completely.

**It is important to re-run `CMake` when adding new source files.**

## Integration with IDE

During the `CMake` stage of the build, `CMake` generates a JSON file named `compile_commands.json` and places it under the
build folder. This file is used by many IDEs and text editors for providing code completion (via `clangd`).

A small caveat is that these tools will look for `compile_commands.json` under the Valkey's top folder.
A common workaround is to create a symbolic link to it:

```bash
cd /path/to/valkey/
# We assume here that your build folder is `build-release`
ln -sf $(pwd)/build-release/compile_commands.json $(pwd)/compile_commands.json
```

Restart your IDE and voila

# Code contributions

Please see the [CONTRIBUTING.md][2]. For security bugs and vulnerabilities, please see [SECURITY.md][3].

[1]: https://github.com/valkey-io/valkey/blob/unstable/COPYING
[2]: https://github.com/valkey-io/valkey/blob/unstable/CONTRIBUTING.md
[3]: https://github.com/valkey-io/valkey/blob/unstable/SECURITY.md
# Valkey is an open community project under LF Projects

Valkey is an open community project under LF Projects
-----------------
Valkey a Series of LF Projects, LLC
2810 N Church St, PMB 57274
Wilmington, Delaware 19802-4447

[1]: https://github.com/valkey-io/valkey/blob/unstable/COPYING
[2]: https://github.com/valkey-io/valkey/blob/unstable/CONTRIBUTING.md
[3]: https://github.com/valkey-io/valkey/blob/unstable/SECURITY.md
Loading

0 comments on commit 07b3e7a

Please sign in to comment.