HDF5 1.12.0 compatibility

[kelbstf]

Hello @grimbough,
is there any option to compile rhdf5 against HDF5 1.12.0 instead of HDF5 1.10.x(7) (in R 4.x)?
Our stack makes heavy use of HDF5 1.12.0 and it would be a blessing to simply add "/rhdf5" to that same stack, using that same HDF5 1.12.0.
This would spare running multiple R-(Server)versions per HDF5 interface in a multiuser environment.
HDFGroup hints that one could compile applications against HDF5 1.12.0, but on a 1.10.x feature set level, as described here [https://www.hdfgroup.org/downloads/hdf5], but i am not shure whether this would possibly harm the integrity of "rhdf5".
Not to mention i currently would not know how to provide the described build options to the compiler from within R?
Best regards

[grimbough]

Hi @kelbstf

Configuration of rhdf5 is pretty tied to the Rhdf5lib package and the 1.10.7 version of the library distributed with that. This design decision was deliberately taken to try and simplify the installation of HDF5 for users and to tie the package to a single version of HDF5 to make debugging easier for developers. It's pretty hard (probably impossible) to override that at the moment.

However at some point I did consider this, and you can set the environment variable RHDF5LIB_RPATH to the path for an alternative location of libhdf5.a outside of the Rhdf5lib installation location. It's not documented and I can't remember whether that's because it wasn't successful or I just forgot, but you can give it a try. You should do this before trying to install rhdf5. I expect you'll run into the compatibility problems you mentioned, but I'd be interested to see what happens and maybe we can figure a way to provide this option going forward.

[kelbstf]

Hi @grimbough,,

thank you very much for looking into this, i really appreciate it.
I just followed you proposal to set the custom path for "RHDF5LIB_RPAT",
and it looks fundamentally better,
but now it fails at this step:

> BiocManager::install("rhdf5")
Bioconductor version 3.13 (BiocManager 1.30.16), R 4.1.0 (2021-05-18)
Installing package(s) 'rhdf5'
trying URL 'https://bioconductor.org/packages/3.13/bioc/src/contrib/rhdf5_2.36.0.tar.gz'
Content type 'application/x-gzip' length 1096041 bytes (1.0 MB)
==================================================
downloaded 1.0 MB

* installing *source* package ‘rhdf5’ ...
** using staged installation
configure: creating ./config.status
config.status: creating src/external_filters.h
** libs
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5.c -o H5.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5A.c -o H5A.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5D.c -o H5D.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5E.c -o H5E.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5F.c -o H5F.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5G.c -o H5G.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5I.c -o H5I.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5L.c -o H5L.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5O.c -o H5O.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5P.c -o H5P.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5S.c -o H5S.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5S_extras.c -o H5S_extras.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5T.c -o H5T.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5Z.c -o H5Z.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c H5constants.c -o H5constants.o
g++ -std=gnu++11 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c HandleList.cpp -o HandleList.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c HandleListWrap.c -o HandleListWrap.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c external_filters.c -o external_filters.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c h5dump.c -o h5dump.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c h5ls.c -o h5ls.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c h5testLock.c -o h5testLock.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c h5writeDataFrame.c -o h5writeDataFrame.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c printdatatype.c -o printdatatype.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c utils.c -o utils.o
gcc -std=gnu99 -I"/opt/R/4.1.0/lib/R/include" -DNDEBUG  -I'/opt/R/4.1.0/lib/R/library/Rhdf5lib/include' -I/usr/local/include   -fpic  -g -O2  -c wrap.c -o wrap.o
g++ -std=gnu++11 -shared -L/opt/R/4.1.0/lib/R/lib -L/usr/local/lib -o rhdf5.so H5.o H5A.o H5D.o H5E.o H5F.o H5G.o H5I.o H5L.o H5O.o H5P.o H5S.o H5S_extras.o H5T.o H5Z.o H5constants.o HandleList.o HandleListWrap.o external_filters.o h5dump.o h5ls.o h5testLock.o h5writeDataFrame.o printdatatype.o utils.o wrap.o /opt/hdf5-1.10.7/lib/libhdf5.a /opt/hdf5-1.10.7/lib/libsz.a -lcrypto -lcurl -lz -L/opt/R/4.1.0/lib/R/lib -lR
g++: error: /opt/hdf5-1.10.7/lib/libsz.a: No such file or directory
make: *** [rhdf5.so] Error 1
ERROR: compilation failed for package ‘rhdf5’

I guess this relates to "szlib" and how 1.10.7 has been compiled?
"libsz.a" is in fact missing from the HDF5 1.10.7 distribution provided by hdfgroup, which only seem to ship with the object files....
Rhdf5lib has it though: "/usr/lib64/R/library/Rhdf5lib/lib/libsz.a"...

Before i am digging deeper in this direction:
i wonder how likely this approach would actually work in a "real world" environment,
assuming that a use case might arise where users make use of rhdf5 (compiled against 1.10.7)
and are then loading additional dependencies, which themselves might have been compiled earlier and against HDF5 1.12.8, without the user considering this aspect?

Would the R package management be able to keep track of those implicit, "2nd-level" compatibility issues?
I.o.w.: would this approach depend on library combinations which are all known to NOT have been compiled against different HDF5 APIs, if i see this right?

This might be hard to handle on a multiuser server...