Rocky8temp #14
Open
Rocky8temp #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- XrdOucPinLoader.hh is needed by XrdClPlugInManager.hh - XrdZipDataDescriptor.hh is needed by XrdZipCDFH.hh - XrdCryptoFactory.hh is needed by XrdCryptosslAux.hh - XrdOucCRC32C.hh is needed by XrdEcUtilities.hh - XrdOucTUtils.hh is needed by XrdClUtils.hh
XrdEc headers include <isa-l/crc.h>, which is only available when isa-l is taken from the system as an external dependency.
Some tests sporadically fail on macOS in GitHub Actions, but so far I cannot reproduce these test failures locally on a macOS machine. Allow a few retries in the CI to avoid having to re-run workflows manually.
When all.pidpath is set to a non-existent directory and it cannot be created for some reason (permission denied as shown below), before this change the variable pidFN would be uninitialized and the error message would contain garbage instead of the name of the directory that could not be created. ... ++++++ xrootd [email protected] initialization started. Config using configuration file /tmp/xrootd.cfg =====> all.adminpath /var/spool/xrootd =====> all.pidpath /run/xrootd =====> xrd.port any 231116 10:57:41 8927 XrdSetIF: Skipping duplicate private interface [::172.17.0.1] 231116 10:57:41 8927 XrdConfig: Unable to create /run/xrootd; permission denied ...
.../src/XrdEc/XrdEcReader.cc:934:72: error: ‘info’ may be used uninitialized [-Werror=maybe-uninitialized]
934 | blockMap[blkid]->stripes[strpid].resize( info ->GetSize() );
| ^
.../src/XrdEc/XrdEcReader.cc:932:42: note: ‘info’ was declared here
932 | XrdCl::StatInfo* info;
| ^
Many features in xrootd require file system support for extended file attributes. Tests that run tests on these features fail if the file system does not support them. The /tmp directory in many Linux installations is using a tmpfs partition. The tmpfs file system does not support extended file attributes, so some tests that use /tmp to store files fail. This commit changes some affected tests so that they create the temporary directory containing the test files in the current working directory instead of /tmp. Example of failure: 17/34 Test xrootd#20: XrdEc::AlignedWriteTest ...................................................***Failed 0.07 sec You have selected: Selected tests/ Selected tests/MicroTest::AlignedWriteTest Running: .F MicroTest.cc:506:Assertion Test name: MicroTest::AlignedWriteTest assertion failed - Expression: _st.IsOK() - [*status]: [ERROR] Internal error: std::bad_alloc Failures !!! Run: 1 Failure total: 1 Failures: 1 Errors: 0 The following tests FAILED: 20 - XrdEc::AlignedWriteTest (Failed) 21 - XrdEc::SmallWriteTest (Failed) 22 - XrdEc::BigWriteTest (Failed) 23 - XrdEc::VectorReadTest (Failed) 24 - XrdEc::IllegalVectorReadTest (Failed) 25 - XrdEc::AlignedWrite1MissingTest (Failed) 26 - XrdEc::AlignedWrite2MissingTest (Failed) 27 - XrdEc::AlignedWriteTestIsalCrcNoMt (Failed) 28 - XrdEc::SmallWriteTestIsalCrcNoMt (Failed) 29 - XrdEc::BigWriteTestIsalCrcNoMt (Failed) 30 - XrdEc::AlignedWrite1MissingTestIsalCrcNoMt (Failed) 31 - XrdEc::AlignedWrite2MissingTestIsalCrcNoMt (Failed) In addition to addressing the above failures, the commit also addresses the following warnings during the XRootD::smoke-test test: 34: setfattr: /tmp/xrdfs-test-ChGSEb/01.ref: Operation not supported 34: Extended attributes not supported, using downloaded checksums for server checks 34: setfattr: /tmp/xrdfs-test-ChGSEb/02.ref: Operation not supported 34: Extended attributes not supported, using downloaded checksums for server checks 34: setfattr: /tmp/xrdfs-test-ChGSEb/03.ref: Operation not supported 34: Extended attributes not supported, using downloaded checksums for server checks
-ENOTSUP. When this small negative value is cast to a size_t in the call to new on the following line it becomes a very large integer and the request to allocate this enormous memory block fails with a std::bad_alloc exception. This commit adds a check on the returned size and returns an error if it is negative avoiding triggering the std::bad_alloc exception.
Outside most grid contexts, we need to present the certificate chain in the TLS handshake for the verification path through the intermediate CA to the root. This commit switches over to the corresponding OpenSSL function call. Fixes the ability to use most CAB CAs with XRootD.
tests/XrdCl/XrdClLocalFileHandlerTest.cc
Original commit message:
OpenFlags::Flags should not be used for the flags argument in a posix
open() call, since they use different values.
The OpenFlags::Flags values are defined in src/XrdCl/XrdClFileSystem.hh:
MakePath = kXR_mkpath, //!< Create directory path if it does not
//!< already exist
New = kXR_new, //!< Open the file only if it does not already
Update = kXR_open_updt, //!< Open for reading and writing
The kXR_* values are defined in src/XProtocol/XProtocol.hh:
kXR_new = 0x0008, // 8
kXR_mkpath = 0x0100, // 256
kXR_open_updt= 0x0020, // 32
As can be seen these do not correspond to the O_* values used in the
posix open() call. What they actually mean depends on the system and
varies between architectures.
On hppa Linux O_CREAT is defind in /usr/include/hppa-linux-gnu/asm/fcntl.h:
#define O_CREAT 000000400 /* not fcntl */
Since 0o400 = 0x100 = 256 this by chance matches kXR_mkpath and
triggers the error below. On other architectures the wrong flags are
just silently accepted.
In file included from /usr/include/fcntl.h:342,
from /<<PKGBUILDDIR>>/tests/XrdClTests/LocalFileHandlerTest.cc:25:
In function ‘int open(const char*, int, ...)’,
inlined from ‘void LocalFileHandlerTest::WriteMkdirTest()’ at /<<PKGBUILDDIR>>/tests/XrdClTests/LocalFileHandlerTest.cc:210:17:
/usr/include/hppa-linux-gnu/bits/fcntl2.h:50:31: error: call to ‘__open_missing_mode’ declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
50 | __open_missing_mode ();
| ~~~~~~~~~~~~~~~~~~~~^~
XrdSys/XrdSysPlatform.hh defines MAXPATHLEN if needed.
EXPECT_EQ( A, B ) provides more useful diagnostics than EXPECT_TRUE ( A == B ). EXPECT_NE( A, B ) provides more useful diagnostics than EXPECT_TRUE ( A != B ).
These break compilation with the error below:
xrootd-4.8.3/tests/XrdSsiTests/XrdShMap.cc:
In function ‘int DoA32(const char*)’:
xrootd-4.8.3/tests/XrdSsiTests/XrdShMap.cc:418:34:
error: expected initializer before ‘OF’
ZEXTERN uLong ZEXPORT adler32 OF((uLong adler, const Bytef *buf, uInt len));
Original patch:
https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=518dbf0b81b27225e09bc939ba6af14a15830922
The tests can now be run locally, without the need to build docker images.
Headers are installed into the subdirectory xrootd/. This was broken with commit c6e0e59 ("[CMake] Find only XRootD matching XRootDConfig.cmake installation path") released with version 5.6.3.
…cation (64 < scitag.flow < 65536)
…for expID and actID Also modified the validation of the scitag provided by the user. Checks whether the expID and actID are within the min and max values. In the case the provided value is not correct, the packets will be marked with a scitag = 0 (expId = 0 and actId = 0)
Fixes xrootd#2259. A new configuration is available "http.keepheadercase" allowing XrdHttp to pass client-provided non-lowercased headers to HTTP external handlers
…arsing We do not modify the header key coming from the client anymore. The reason for that is for HTTP/TPC where we may need to pass headers to the non-active server. If they are lower-cased, then we will break the compatibility between old xrootd server and new ones as the old ones have a case-sensitive header parsing logic! Fixes xrootd#2259
- Allow to optionally build documentation - Sync with downstream changes to use 64bit time_t - Restrict building ceph plugin and XrdEc to supported arches - Add option to enforce rootless builds (to run server tests) - Add xrootd metapackage to install server and client together
The Doxygen docs take a lot of space and increase the CI build time significantly.
This new load balancing algorithm randomizes the choice of node based on the pre-existing fuzz parameter for cms.sched. Selectable nodes are assigned weights inversely proportional to their loads, plus the fuzz factor to allow selection to be more evenly distributed. Please note that a fuzz factor of 100 will suppress the use of load information in scheduling decisions, as has been the case prior to this change. For this algorithm to take effect, a formula for load must be specified. For example, to use 50% CPU and 50% I/O as factors for load calculation, and 10% randomization factor, one can use the following: cms.sched cpu 50 io 50 affinity randomized fuzz 10
Homebrew now follows PEP668 (https://peps.python.org/pep-0668) after the move to Python 3.12, so we cannot install updates to pip, wheel, and setuptools with pip anymore. We need to use only brew.
The variable 'items' was multiplied by 2 to get n for the loop, but it was including the first item for the header, which was double counted later. This commit changes 'items' to count only operations, excluding the header, which fixes the buffer overrun. Note that IOV_MAX == 1024, which means maxCSSZ == 511, and the longest loop is n = 2 * 511 = 1022. The while loop then does i = 1, 3, 5, ..., 1021, in which we update the elements iov[1021] and iov[1022] and exit the loop with i = 1023 > n. Fixes: xrootd#2287
This allows genversion.sh to generate versions that are usable for DEBs, RPMs, and Python's pip package manager (See PEP440).
The administrator can now use this option in order to allow some client HTTP headers to be passed to the OFS layer as opaque information. It works exactly the same way as the http.header2cgi configuration. E.g: the active server of a HTTP/TPC PULL transfer receives the header 'ArchiveMetadata' and 'tpc.header2cgi ArchiveMetadata archivemetadata' is configured. If a client submits a 'ArchiveMetadata: test' header with their request, the opaque archivemetadata=test will be provided to the XrdSfsFile open() function. Fixes xrootd#2283
The string in 'pub' is not guaranteed to be null-terminated.
|
Jo-stfc
commented
Jul 29, 2024
| } | ||
|
|
||
| // If node passes filters, give it a weight | ||
| totWeight += Config.P_fuzz + (100 - np->myLoad); |
There was a problem hiding this comment.
to note in documentation: if fuzz value is 0, nodes at 100 load will not be selected
Jo-stfc
commented
Jul 29, 2024
| int selected = distr(generator); | ||
|
|
||
| for (int i = 0; i <= STHi; ++i) { | ||
| if (NodeWeight[i] < selected) |
There was a problem hiding this comment.
comment to explain - selection happens in intervals from previous value to the NodeWeight value
Jo-stfc
commented
Jul 29, 2024
src/XrdCms/XrdCmsCluster.cc
Outdated
| return sp ? sp : calcDelay(selR); | ||
| } | ||
|
|
||
|
|
There was a problem hiding this comment.
create documentation to provide high level overview on how this algorithm works
Jo-stfc
commented
Jul 29, 2024
| sp=NodeTab[i]; | ||
| break; | ||
| } | ||
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes for enabling xrootd to work correctly post rocky8 upgrade