Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Determine and display various versions of interest #521

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Determine and display various versions of interest #521

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7c4eeb1
Determine and display various versions of interest
iankouls-aws Jan 6, 2025
9e6a5a7
Add os info to versions script
iankouls-aws Jan 6, 2025
4aaba66
Add GDS version info, when it is installed
iankouls-aws Jan 7, 2025
d7931d7
Changed EFA Installer version -> EFA version
iankouls-aws Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions 4.validation_and_observability/versions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#!/bin/bash

if [ -f /etc/os-release ]; then
echo ""
echo "OS info:"
cat /etc/os-release | head -n 4
fi

echo ""
echo "Linux family:"
uname

echo ""
echo "Linux Kernel version:"
uname -r

echo ""
echo "nvidia-smi:"
which nvidia-smi
if [ "$?" == "0" ]; then
echo ""
echo "NVIDIA versions:"
nvidia-smi --version | grep DRIVER
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nvidia-smi --version | grep DRIVER
nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0

nvidia-smi --version | grep CUDA
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nvidia-smi --version | grep CUDA
nvcc --version | sed -n 's/^.*release \([0-9]\+\.[0-9]\+\).*$/\1/p

nvidia smi provices the cuda version support by the nvidia driver not the actual version of cuda


NCCL_LIB=$(ls /usr/local/cuda/lib/libnccl.so.*.*.*)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It points toward the default path of cuda install. However there can be many version cuda installed.

search="so."
NCCL_VERSION=${NCCL_LIB#*$search}
echo "NCCL version: $NCCL_VERSION"

OFI_NCCL_LIB=$(strings /opt/aws-ofi-nccl/lib/libnccl-net.so | grep "Initializing aws-ofi-nccl")
OFI_NCCL_VERSION=$(echo $OFI_NCCL_LIB | cut -d ' ' -f 4)
echo "AWS OFI NCCL version: $OFI_NCCL_VERSION"

if [ -f /usr/local/cuda/gds/tools/gdscheck ]; then
echo "NVIDIA GDS:"
/usr/local/cuda/gds/tools/gdscheck -v
fi
else
echo "not present"
fi

echo ""
echo "Lustre client version:"
LUSTRE_CLIENT_VERSION="not found"
which yum
if [ "$?" == "0" ]; then
LUSTRE_CLIENT_VERSION=$(yum list lustre-client | grep lustre-client | awk '{print $2}')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are checking installed version, not the actual version loaded.

Suggested change
LUSTRE_CLIENT_VERSION=$(yum list lustre-client | grep lustre-client | awk '{print $2}')
modinfo lustre | grep 'version:' | head -n 1 | awk '{print $2}'

else
LUSTRE_CLIENT_VERSION=$(apt list lustre-client | grep lustre-client | cut -d ' ' -f 2)
fi
echo $LUSTRE_CLIENT_VERSION

echo ""
if [ -f /opt/amazon/efa_installed_packages ]; then
echo "EFA version:"
EFA_LIBS=($(cat /opt/amazon/efa_installed_packages))
EFA_INSTALLER_VERSION=${EFA_LIBS[-1]}
echo $EFA_INSTALLER_VERSION

echo ""
echo "Libfabric version:"
LIBFABRIC_VERSION=$(/opt/amazon/efa/bin/fi_info --version | grep libfabric: | cut -d ' ' -f 2)
echo $LIBFABRIC_VERSION

else
echo "EFA Installer not found"
fi

echo ""