Use glib as base library

[mgautierfr]

This is a open question. I wonder if ... I'm not sure we have to do it. But I open the issue to discuss about this.
This issue is related to almost all our projects but I have to open the issue somewhere.

glib (https://developer.gnome.org/glib/stable/) is the low level library of the gnome project.
While it is in the gnome project, it is not related at all with graphic/ui (handle by gdk, gtk, clutter, ...).

The thing is that glib provide a lot of functionality we use/need (in parenthesis, the library or code we have) :

• Byte order manipulation (https://github.com/openzim/libzim/blob/master/src/endian_tools.h)
• File system (https://github.com/openzim/libzim/blob/master/src/fs.h, https://github.com/openzim/libzim/blob/master/src/fs_unix.cpp, https://github.com/openzim/libzim/blob/master/src/fs_unix.h, https://github.com/openzim/libzim/blob/master/src/fs_windows.cpp, https://github.com/openzim/libzim/blob/master/src/fs_windows.h, https://github.com/kiwix/kiwix-lib/blob/master/src/tools/pathTools.cpp, all the mmap things on libzim (and we have nothing for now on windows))
• Threads (pthread, but specific implementation on windows https://github.com/kiwix/kiwix-build/blob/master/appveyor/install_pthread.cmd)
• unicode and regex (icu4c)
• spawning process (https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess.h, https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess.cpp, https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess_unix.cpp, https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess_unix.h, https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess_windows.cpp, https://github.com/kiwix/kiwix-lib/blob/master/src/subprocess_windows.h)
• command line parsing option (Almost all our binary, sometime with getopt (with specific windows version to have getopt_long), sometime directly reading argv, sometime custom parser (https://github.com/openzim/zim-tools/blob/master/src/arg.h))
• bookmarks storage and manipulation (https://github.com/kiwix/kiwix-lib/blob/master/include/bookmark.h, https://github.com/kiwix/kiwix-lib/blob/master/src/bookmark.cpp, https://github.com/kiwix/kiwix-lib/blob/master/src/libxml_dumper.cpp#L124-L137, ...)
• configuration file
• cherksum (https://github.com/openzim/libzim/blob/master/src/md5.c)
• Internationalization (nothing for now. Kiwix-desktop has Qt and we will not change this)
• testing (gtest)
• uuid (https://github.com/openzim/libzim/blob/master/src/uuid.cpp, e2fsprogs)

The compilation system of glib is meson. We already have everything we need on our side to compile it. glib depends on few other libraries:

• iconv. Present on gnu libc since 1999 so not a problem on linux. On windows we can use gnu's libiconv)
• gettext. The de facto standard internationalization tool. We will need it anyway.
• a thread library. pthread on unix or win32 threads (but we don't need pthread on windows)

By using glib, we could remove the use of other dependencies (icu and all the icu data file manipulation, e2fsprogs, gtest) and almost all our "tools" code.
We would also be (more) platform independent using well established library instead of our (probably buggy) code.

[kelson42]

@mgautierfr I agree with the principle idea to rely on a well established base lib.

I have a few questions:

• I hardly understand how glib would avoid using libicu. Does glib for example provide function to remove accents?
• What is the complete list of glib dependences?
• How much work would be needed to get all of this compiles properly in CI?
• Where is the list of supported architectures? Does it works fine on all our supported systems?
• glib is C, why not using a baselib in C++ like Boost?
• How much bigger would become our static binaries?

[mgautierfr]

hardly understand how glib would avoid using libicu. Does glib for example provide function to remove accents?

Not directly, but the algorithm is pretty simple: Normalize using a decomposed form, remove accent chars.
This is explain here (https://sigquit.wordpress.com/tag/unaccenting/) with an implantation here (https://mail.gnome.org/archives/commits-list/2010-June/msg04595.html)
It already has functions to upper or lower case a unicode string.

What is the complete list of glib dependences?

The ones I've listed. (See https://developer.gnome.org/glib/stable/glib-building.html)
From the meson.build file (https://gitlab.gnome.org/GNOME/glib/blob/master/meson.build) the mandatory dependencies are :

• libffi
• some function from gettext
• threading support
• iconv

How much work would be needed to get all of this compiles properly in CI?

Hopefully not so much (if everything is good with next question). We can start to compile glib on the CI and slowly move our code base to glib.

Where is the list of supported architectures? Does it works fine on all our supported systems?

I don't know. Linux, Windows and mac platform should be ok. As the different archs.
There is this blog post to cross-compile for android (https://zwyuan.github.io/2016/07/17/cross-compile-glib-for-android/) this is pretty old (still use the autotools build system) but it should be better now than before :)

glib is C, why not using a baselib in C++ like Boost?

Because I've not investigate boost :)
But boost is also a solution yes.

But C has an advantage : C abi is standardize, it allows us to link to the lib whatever the compiler used to compile the lib and the binary (as opposite to c++ which need that the same compile is used)

How much bigger would become our static binaries?

No clue at all. We need to test.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

One thing we need as well is something to parse HTTP header properly, see kiwix/kiwix-tools#148. Boost provides one, but not glib AFAIK.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@mgautierfr @veloman-yunkan I think after 4 years a refresh of this ticket would be beneficial.