Installation Manual

Installation

SPNC comprises two main parts: xspn, a small library to help with the serialization of SPFlow models, and spnc, which is the compiler itself. The installation procedure for both packages is explained in the following.

xspn

xspn is implemented completely in Python and uses Cap'n'Proto via pycapnp.

To install xspn, there are two options:

Install the pre-built wheel provided on the Github repository. As xspn is pure Python, this package should be platform-independent.
Build xspn from source:

git clone https://github.com/esa-tu-darmstadt/spn-compiler.git
cd spn-compiler/xspn
pip install wheel setuptools
pip install .

spnc

The compiler itself is mostly implemented in C++, so the setup is a little more involved.

Pre-Built Packages

The easiest way to obtain spnc, which comprises the compiler and the Python interface to work with the compiler directly from Python, is through the pre-built package provided on the Github repository. As the compiler is implemented in C++, the package is tied to the Linux platform and depends on the following libraries:

libgomp.so.1
libpthread.so.0
libdl.so.2
libz.so.1
libtinfo.so.6
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libc.so.6

This list is mostly identical to the many-linux requirements of Python, with the exception of libgomp.so.1, which is GCC's OpenMP library and should be shipped with most GCC installations.

If you download the pre-built package with CUDA GPU support, the package will additionally depend on libcuda.so.1 and librt.so.1.

In case you want to use SPNC on a non-Linux platform, follow the instructions for building from source below.

Building from Source

Prerequisites

spnc requires a C++ compiler that supports at least the C++14 standard as well as a modern CMake (>= version 3.14)).

spnc and its dependencies require a number of libraries & tools to build. On Ubuntu 20.04, these can be installed with:

apt install -y git gcc clang cmake ninja-build zlib1g zlib1g-dev python3 lld doxygen graphviz autoconf automake libtool python3-venv python3-pip python3-dev pkg-config libelf-dev libelf1

The following procedure has been tested on Ubuntu 20.04. $BASE_DIR is used as a placeholder for a directory of your choice, replace it in the following commands or make it available via export BASE_DIR=[...].

If you want to enable CUDA support in the compiler, make sure that you have installed the Nvidia driver and a working CUDA toolkit, and that nvcc is available on your PATH.

For End-Users

The following procedure is intended for users that wish to install SPNC once.

1. Build LLVM from Source

Make sure to replace XYZ in the following code with the LLVM base commit specified by SPNC's README.

cd $BASE_DIR
mkdir llvm
cd llvm
git clone https://github.com/llvm/llvm-project.git llvm-src
cd llvm-src
# Check out specific commit. Other versions of LLVM/MLIR might work, but the commit ID from the README has been tested
git checkout XYZ
cd ..
mkdir llvm-bin
cd llvm-bin

# Make sure to adapt LLVM_PARALLEL_COMPILE_JOBS and LLVM_PARALLEL_LINK_JOBS to your machine's RAM and CPU.
cmake -G Ninja\
    -DLLVM_ENABLE_PROJECTS="mlir;clang;compiler-rt"\
    -DLLVM_BUILD_EXAMPLES=ON\
    -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF\
    -DLLVM_ENABLE_LLD=ON\
    -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_RTTI=ON\
    -DLLVM_PARALLEL_COMPILE_JOBS=16 -DLLVM_PARALLEL_LINK_JOBS=3\
    -DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++\
    -DLLVM_OPTIMIZED_TABLEGEN=ON\
    ../llvm-src/llvm/
        
# Build LLVM & MLIR. This step might take some time, depending on your machine.
ninja

If you want to enable support for CUDA GPUs in the compiler, add -DMLIR_CUDA_CONVERSIONS_ENABLED=ON -DMLIR_CUDA_RUNNER_ENABLED=ON to the cmake command above.

2. Build Pybind11 from Source

Next, install and build pybind11 for the Python-to-C++ interface:

cd $BASE_DIR
git clone https://github.com/pybind/pybind11.git
cd pybind11
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$BASE_DIR/pybind11/install \
     -DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3 ..
make
make install

3. Build spdlog from Source

Install and build spdlog for logging in the compiler:

cd $BASE_DIR
git clone https://github.com/gabime/spdlog.git
cd spdlog
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$BASE_DIR/spdlog/install -DCMAKE_POSITION_INDEPENDENT_CODE=ON ..
make
make install

4. Build Cap'n'Proto from Source

Install and build capnproto for binary serialization of SPN graphs:

cd $BASE_DIR
git clone https://github.com/sandstorm-io/capnproto.git
cd capnproto/c++
autoreconf -i
./configure --prefix=$BASE_DIR/capnproto/install --disable-shared --with-pic
# Make sure to adapt the number of parallel jobs to your machine.
make -j 16
make install

5. Build spnc from Source

Now, download and build spnc:

cd $BASE_DIR
git clone [email protected]:esa-tu-darmstadt/spn-compiler.git
cd spn-compiler
git checkout develop
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH="$BASE_DIR/llvm/llvm-bin/lib/cmake/llvm;$BASE_DIR/llvm/llvm-bin/lib/cmake/mlir;$BASE_DIR/pybind11/install/share/cmake/pybind11;$BASE_DIR/spdlog/install/lib/cmake/spdlog;$BASE_DIR/capnproto/install"\
    -DLLVM_ENABLE_LLD=ON -DLLVM_ENABLE_ASSERTIONS=ON\
    -DSPNC_BUILD_DOC=OFF\
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DBUILD_SHARED_LIBS=OFF\
    ..

# Make sure to adapt the number of parallel jobs to your machine.
make -j 16

If you want to enable support for CUDA GPUs in the compiler, add -DCUDA_GPU_SUPPORT=ON to the cmake command above.

6. Build Python Package

In the last step, a Python package (binary wheel) can be created to install spnc via pip, also in virtual environments.

cd $BASE_DIR/spn-compiler/python-interface
python setup.py bdist_wheel

Copy the created wheel (e.g., spnc-0.1-py3-none-any.whl) from dist/ to any location and use it to install spnc. If you have saved the wheel to some other location, you can delete the whole content of BASE_DIR, the wheel should contain all necessary files.

For developers

For developers actively developing on spnc, we recommend a slightly different installation procedure. The main difference is that all dependencies and the parts of spnc are compiled into shared libraries, to avoid long linking times and large files.

The remaining installation procedure is similar to the one described above.