Local compilation guide for Milvus on arch/manjaro linux

[douglarek]

Note: This is not a request for help post, but rather a sharing of compilation instructions.

Preparation

This compilation tutorial is intended for compiling on the unofficially recommended Linux distributions Arch/Manjaro, excluding Docker. Theoretically, other distributions should also work. Please note that this tutorial is specifically for version v2.4.0.

First, this tutorial will encounter several issues:

1. The gcc compiler version currently only supports up to gcc12. gcc12 can pass by modifying one line of code, but for gcc13 and above, you'll need to modify more code, not only within Milvus itself but also potentially in related code like RocksDB. It's not recommended.
2. The issue with Conan. During compilation, Conan searches for the full version number of gcc, but in its settings, it only has the major version number. This can be resolved by modifying Conan's configuration file.
3. Problem with protoc code generation. Milvus uses an older version of protoc. Newer versions of protoc no longer support plugins with go_out. This can be easily solved.
4. You need unrestricted internet access (Chinese users probably understand this well, no need to elaborate).
5. After compilation, the Milvus source directory will occupy approximately 6GB of disk space. Make sure you have enough disk space reserved.

This compilation tutorial assumes you understand how to configure Go and use the distribution.

Getting Started

Installing Necessary Software

Installation on Arch is relatively simple; the required software is available in the official repositories. Just install them directly:

sudo pacman -S cmake base-devel gcc12 openblas clang go python

Installing Conan may vary; you may need to use pipx:

pipx install conan==1.64.0 --force

Setting up gcc

This step involves setting up the previously installed gcc12. As of the date of this tutorial (2024-04-19), the default gcc version in Arch is gcc13. So, you need to change the default gcc version to gcc12:

sudo ln -s $(which g++-12) /usr/local/bin/g++ && sudo ln -s $(which gcc-12) /usr/local/bin/gcc

It's straightforward and won't delete your defaults. It just prioritizes gcc12 under /usr/local. If you don't need it later, just delete this soft link.

Setting up cmake

It seems that the default gcc12 setting mentioned above doesn't work for cmake. Therefore, before building, you can set an environment variable to force cmake to use your gcc12:

export CC=/usr/local/bin/gcc && export CXX=/usr/local/bin/g++

Compilation

Now we can proceed with source code compilation. Execute make in the source code root directory:

make

Depending on your machine configuration, if everything goes smoothly, it should take less than half an hour for a 16-core 32-thread AMD 3950x.

gcc

When compiling with gcc12, you might encounter a missing import of cstdint, which is easy to fix:

diff --git a/internal/core/src/config/ConfigKnowhere.h b/internal/core/src/config/ConfigKnowhere.h
index d9e8df1fd..73bcebb61 100644
--- a/internal/core/src/config/ConfigKnowhere.h
+++ b/internal/core/src/config/ConfigKnowhere.h
@@ -15,6 +15,7 @@
 // limitations under the License.
 
 #pragma once
+#include <cstdint>
 #include <string>
 
 namespace milvus::config {

protoc

If your system is relatively new and you're not using a distribution-installed protoc, you'll likely encounter issues with protoc code generation. It's also easy to fix. Either temporarily remove protoc from your PATH or modify scripts/generate_proto.sh:

${protoc_opt} --go_out=plugins=grpc,paths=source_relative:./etcdpb etcd_meta.proto || { echo 'generate etcd_meta.proto failed'; exit 1; }
...

Assuming you know how to make these modifications. I haven't made any modifications here.

With this, there should be no major issues. Just wait for the compilation to finish.

[xiaofan-luan]

wow that's nice.
Really appreciate the sharing!