Binary packaging ideas

[gyorb]

Some options to consider if we want to create a binary package.

• pypi and virtualenv: Linux, Windows, Mac
  install with pip in python, all the python dependencies would be downloaded and installed, but there are still missing dependencies which the user needs to install manually (clang, clang-tidy, postgresql ...)
• appimage: RHEL, CentOS, Ubuntu, Fedora, Debian
  create a universal binary package in an old enough Linux release, custom compile scripts are needed to build dependencies (clang, clang-tidy, psycopg, ldap, psutil ...)
  no extra installation is required by the user
• docker image: Mac, Windows, Linux
  more heavyweight than the previous options and docker is needed to be installed which is might not possible for some older releases.
• HomeBrew Mac

I still think pypi package should be created, this would enable easier installation in the other cases too.
Appimage seems promising but this is a Linux only option.

What do you think? Do you know other options we should consider?

[whisperity]

I agree with support pypi, that's the "closest" to the current installation method and still easier, and the most "lightweight" one, enabling support for arbitrary clang installs present on the system.

Could there be any way to "convert" between appimage (and thus, apt, yum, rpm and other packaging methods) and homebrew?
Homebrew specifications look much like Dockerfiles.

I think all of the options above have upsides to them and there are reasons for supporting them as binary package... Though I'd like to add a fifth option, a totally custom "install script" to the list.

If libncurses is installed (I'm sure it has a py binding), we could make a nice install "GUI" in the terminal, just like yast on SuSE works, which could, in theory, detect everything needed for CodeChecker on the system, and download-build-install-etc. the rest. Lots of config options which works for everyone...

... and having such a script, we could also streamline the generation of the initial four release types.

[WallStProg]

At least for me, the problem is not so much the dependencies per se, but that at least some of the dependencies are broken/different/non-existent on at least some OS's (including apparently RH/CentOS, which I try to stick with since that is my production environment). I'm perfectly OK downloading and installing packages from source, and in fact I tend to prefer that approach, since it gives me more control -- for instance, I refuse to install anything in /bin etc. unless it's part of the distro.

The appimage would be a lot of work, I imagine, and I tend not to like "lowest common denominator" approaches. Docker seems like it might work, but again the the issue is compatibility with the host OS, I think. Personally I'm OK with Homebrew, but that's because I'm on Mac, and tend to use that for GUI-heavy applications.

A good start would be to remove hard-coded dependencies from the codechecker setup scripts. For instance, I think my original problem can be traced to pip install explicitly invoking /usr/bin/python, which is the wrong version on RH6. (I've got python 2.7 installed, but it's not in /usr, because I'm OCD about keeping the OS "clean").

One last thought -- please consider that many who would like to use the software don't have root access on their machines, and end up working around that by installing software in their home directories, etc.

In any event, thanks for taking the time and trouble to consider how to make codechecker work in "other" environments.

[whisperity]

I think your problem roots in virtualenv wasn't being sourced (or a too old version not working properly?) as it should be. The install script does not require root privileges at all, we are running it on our development machines all the time without sudo.

[WallStProg]

That's not my point -- on RH6 /usr/bin/python is 2.6, and that's what pip install tries to use (and fails, of course, since it requires 2.7). I can put python 2.7 on my PATH, but if the install script doesn't honor that setting (by hard-coding the path), then the install script fails.

And, of course, installing the prerequisites does require root privileges -- unless they are built from source and installed in a non-standard location. But with hard-coded paths that doesn't work either.

While I'm not particularly familiar with virtualenv, I don't believe that it uses chroot, so it cannot change /usr/bin to something else. And yes, I did follow the instructions (several times), including sourcing the activate script. See #484 for details.

Again, the problems appear to come down to (a) I'm on a different OS than what is used to develop codechecker, and (b) some of the codechecker scripts make assumptions that are not necessarily valid in different environments.

It would be much simpler if I could use Ubuntu, but that would cause other problems, since it's not my production environment.

[cleeland]

Are the things that land in the virtual environment needed for build, run, or both?
Are there things created at build time that are installed in the virtualenv?

I'm working on packaging via rpm and trying to get an understanding of what goes where so that I can package.

[gyorb]

Hi,

The direction we are planning with the debian packaging is to create two separate deb packages:

1. analyzer package

• dependencies are the static analyzers (clang, clang-tidy)
• tools (plist_to_html, tu_collector)

2. web package

• client (to store/diff the reports to a server)
• server (web server, postgresql/sqlite ..)

Analyzer
Packaging:

• needs a python virtual environment (should be the same as the runtime)
• gcc is needed to build the ldlogger
• javascript dependecies are downloaded automatically (we plan to switch to npm later)
• doxygen to build the documentation (we plan to replace this)

make -C analyzer package builds the analyzer package only, under analyzer/build/

Runtime:

• static analyzers: clang/clang-tidy
• the requirements.txt contains the runtime dependencies

Web
Packaging:

• javascript dependecies are downloaded automatically (we plan to switch to npm later)
• doxygen to build the documentation (we plan to replace this)
• thirft-compiler (v0.9.1) to generate api stubs between the client and the server
• the python virtualenv could be less than the runtime environment but we usually use that for building the package

make -C web package builds the web package only, under web/build/

Runtime:

• the requirements.txt contains only the basic runtime requirements if you want to run a package without the ldap authentication feature. If you want that too you will need the auth requirements too
• if SQLite is used as a database nothing extra is needed
• if PostgreSQL is used as a database:
  • for the CodeChecker client part the postgresql-client is needed
  • for the server part a PostgreSQL server is needed.

For the web part a multi stage dockerfile was introduced which could be help you to see the different dependencies for packaging and runtime.

The virtual environments under the dev directories contain extra dependencies to run the tests.

I would recommend to start trying to create a package for the analyzer part.
We started to split up the source code (analyzer and web) #1830 but it is not finished yet.
There is the codechecker_common python package which is still required by both sides.
We might need some further refactoring to build the deb and rpm packages easier.

Please let us know if you bump into problems or have more questions.

[cleeland]

I conceptually divided the tool along the same lines, but did not make any effort to divide the packaging because I didn't want to have to support any such changes myself in a fork. At the moment, my .spec file mimics the "quick start" build recommendations and packages that up. The packaging is actually done in a multi-stage docker file.

The resulting RPM is then usable for both analyzer and server/web. My intent is to use the server/web side in a docker container composed with another container running postgresql, since I want this to be usable in a multi-user environment.

How close is your packaging division to being available in master? If it's imminent, I'll just set this aside and wait.

[gyorb]

We introduced a new documentation recently, how you can build a docker image from the web server. We are still fine tuning the image builder Dockefile see #2109.
There are some additional docker compose examples for easier database setup for the server.

Do you plan to publish the rpm package officially?

[cleeland]

My plan was to contribute the packaging somehow, although the exact mechanism of contribution is uncertain. Currently, I have an internal project that only does the packaging, and I would likely at least publish that (presuming I receive clearance from my employer, which seems reasonably likely) so that others could leverage it to create their own packaged version. I have no affiliation with any of the official distros or RPM providers, so I don't know what effort would be involved or required to make an "official" package.

Ideally any work I do would get contributed back into this project as a PR, which is why I wanted to start a conversation about packaging before I got too far into what I am doing. I'd rather we work together, or that work I am doing moves the project towards its ultimate goals, rather than competing--assuming we have similar goals. Based on what you've said, our goals and philosophies sound similar enough that there should be a lot of synergy.

Is this the best way forum in which to collaborate?

[cleeland]

The PostgreSQL (authentication) composition is the closest to what I envision using, although I plan on anchoring to a specific released version of codechecker, and putting the database files not in a user's home directory. In my specific case I'd like to figure out a way to connect codechecker into our LDAP server, so the codechecker config would also need to be in a separate volume.

[gyorb]

Thanks for working on this.

For the official packages we would like to have the same packaging for the rpm and deb packages (analyzer, web).
We will try our best to finish splitting up the two parts in the next weeks or month. There are some parts which can be moved from codechecker_common easily but there are parts which might need further refactoring on both sides.

To collaborate we use mainly the GitHub issues and we have a gitter channel.