Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gyan output is empty apart from the headers #4

Open
rahul003 opened this issue Jan 5, 2019 · 0 comments
Open

Gyan output is empty apart from the headers #4

rahul003 opened this issue Jan 5, 2019 · 0 comments

Comments

@rahul003
Copy link

rahul003 commented Jan 5, 2019

I'm starting my job as follows

LD_PRELOAD=~/mpi-tools/mpi_t/gyan/libgyan.so mpirun -np 8 -hostfile hosts -mca plm_rsh_no_tree_spawn 1 -bind-to socket -map-by slot -x HOROVOD_HIERARCHICAL_ALLREDUCE=1 -x HOROVOD_FUSION_THRESHOLD=16777216 -x NCCL_MIN_NRINGS=4 -x LD_LIBRARY_PATH -x PATH -mca pml ob1 -mca btl ^openib -x TF_CPP_MIN_LOG_LEVEL=0 python -W ignore train_imagenet_resnet_hvd.py --synthetic -b 256 --num_batches 200 --clear_log

Output is

----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Performance profiling for the complete MPI job:
----------------------------------------------------------------------------------------
Variable Name                           	Type    Minimum(Rank)    Maximum(Rank)       Average
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant