CICD NeMo

Avoid init_ddp for inference #8172

You are viewing an older attempt in the history of this workflow run. View latest attempt.

Triggered via pull request February 7, 2025 22:08

athitten

labeled #12011

athitten/inference_bug

Status Cancelled

Total duration 7h 29m 36s

Artifacts –

cicd-main.yml

on: pull_request

pre-flight

cicd-test-container-build / main

49s

L0_Unit_Tests_GPU_ASR / main

13m 55s

L0_Unit_Tests_GPU_Audio / main

2m 18s

L0_Unit_Tests_GPU_Common / main

3m 42s

L0_Unit_Tests_GPU_LLM / main

3m 24s

L0_Unit_Tests_GPU_Multimodal / main

3m 3s

L0_Unit_Tests_GPU_NLP / main

5m 3s

L0_Unit_Tests_GPU_TTS / main

2m 40s

L0_Unit_Tests_GPU_Hydra / main

2m 25s

L0_Unit_Tests_GPU_Lightning / main

4m 19s

L0_Unit_Tests_GPU_Others / main

4m 42s

L0_Unit_Tests_CPU_ASR / main

9m 36s

L0_Unit_Tests_CPU_Audio / main

1m 49s

L0_Unit_Tests_CPU_Common / main

2m 50s

L0_Unit_Tests_CPU_LLM / main

1m 41s

L0_Unit_Tests_CPU_Multimodal / main

1m 31s

L0_Unit_Tests_CPU_NLP / main

2m 10s

L0_Unit_Tests_CPU_TTS / main

2m 42s

L0_Unit_Tests_CPU_Core / main

4m 15s

L0_Unit_Tests_CPU_Hydra / main

2m 4s

L0_Unit_Tests_CPU_Lightning / main

1m 47s

L0_Unit_Tests_CPU_Others / main

2m 4s

cicd-import-tests

1m 26s

L0_Unit_Tests_GPU_Core / main

7m 5s

L2_Community_LLM_Checkpoints_tests_Bert / main

1m 51s

L2_Community_LLM_Checkpoints_tests_Mamba2 / main

1m 48s

L2_Community_LLM_Checkpoints_tests_Llama / main

49s

L2_Community_LLM_Checkpoints_tests_Llama3 / main

1m 50s

L2_Community_LLM_Checkpoints_tests_StarCoder / main

1m 41s

L2_Community_LLM_Checkpoints_tests_Falcon / main

1m 43s

L2_Community_vita_Checkpoints_tests_Llama3 / main

4m 45s

L2_PTQ_Llama2_FP8 / main

2m 57s

L2_Distill_Llama2 / main

3m 4s

L2_Prune_Width_Llama2 / main

2m 25s

L2_Prune_Depth_Llama2 / main

2m 3s

ASR_dev_run_Speech_to_Text / main

52s

ASR_dev_run_Speech_to_Text_WPE_-_CitriNet / main

52s

ASR_dev_run_Speech_Pre-training_-_CitriNet / main

37s

ASR_dev_run_Speech_To_Text_Finetuning / main

1m 38s

ASR_dev_run_Speech_To_Text_HF_Finetuning / main

1m 30s

ASR_dev_run_Speech_to_Text_WPE_-_Conformer / main

1m 32s

ASR_dev_run-part_two_Speech_to_Text_WPE_-_Squeezeformer / main

36s

L2_Speech_to_Text_EMA / main

2m 1s

L2_Speech_to_Text_AED / main

1m 35s

L2_Speaker_dev_run_Speaker_Recognition / main

34s

L2_Speaker_dev_run_Speaker_Diarization / main

38s

L2_Speaker_dev_run_EndtoEnd_Speaker_Diarization_Sortformer / main

41s

L2_Speaker_dev_run_EndtoEnd_Diarizer_Inference / main

1m 24s

L2_Speaker_dev_run_Speech_to_Label / main

1m 27s

L2_Speaker_dev_run_Speaker_Diarization_with_ASR_Inference / main

1m 35s

L2_Speaker_dev_run_Clustering_Diarizer_Inference / main

1m 39s

L2_Speaker_dev_run_Neural_Diarizer_Inference / main

1m 42s

L2_Speaker_dev_run_Multispeaker_ASR_Data_Simulation / main

1m 27s

L2_ASR_Multi-dataloader_dev_run_Speech_to_Text_multi-dataloader / main

1m 42s

L2_ASR_Multi-dataloader_dev_run_Speech_to_Label_multi-dataloader / main

40s

L2_ASR_Adapters_Linear_Adapters / main

1m 34s

L2_ASR_Adapters_RelPos_MHA_Adapters / main

1m 29s

L2_Speech_Estimate_Duration_Bins / main

1m 45s

L2_Speech_Batch_Size_OOMptimizer / main

2m 38s

L2_Speech_Batch_Size_OOMptimizer_Canary / main

2m 8s

L2_Speech_Transcription_Speech_to_Text_Transcribe / main

1m 30s

L2_Speech_Transcription_Canary_Transcribe_Full_Manifest / main

2m 29s

L2_Speech_Transcription_Canary_Transcribe_With_Prompt / main

2m 40s

L2_Speech_Transcription_Canary_Transcribe_Audio_Dir / main

2m 29s

L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Eng_CitriNet_with_wav / main

1m 14s

L2_Segmentation_Tool_Parallel_ctc_segmentation_test_L2_Ru_QN_with_mp3 / main

1m 7s

L2_G2P_Models_G2P_Conformer_training_evaluation_and_inference / main

1m 54s

L2_G2P_Models_HeteronymClassificationModel_training_evaluation_and_inference / main

2m 41s

L2_Pretraining_BERT_pretraining_from_Text / main

25s

L2_Pretraining_BERT_from_Preprocessed / main

1m 46s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Post-LN / main

1m 10s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Training_Pre-LN / main

51s

L2_NMT_Attention_is_All_You_Need_Training_NMT_Multi-Validation / main

43s

L2_NMT_Attention_is_All_You_Need_Inference / main

1m 54s

L2_NMT_Attention_is_All_You_Need_Finetuning / main

1m 41s

L2_NMT_Tarred_Dataset_Creation_Auto_Tarred_Dataset_Creation / main

1m 53s

L2_NMT_Tarred_Dataset_Creation_Script_Tarred_Dataset_Creation / main

1m 44s

L2_Megatron_NMT_Training_TP2 / main

5m 4s

L2_Megatron_Bert_Pretraining_and_Resume_Training_with_Pipeline_Parallelism / main

3m 55s

L2_Megatron_Core_Bert_Pretraining_and_Resume_Training / main

4m 5s

L2_RAG_Pipeline_Indexing / main

1m 46s

L2_RAG_Pipeline_Generating / main

2m 15s

L2_Megatron_GPT_Pretraining_and_Resume_Training_TP2 / main

3m 48s

L2_Megatron_GPT_Skip_Train / main

3m 0s

L2_Megatron_GPT_with_Rope_Pretraining_and_Resume_Training_TP2 / main

3m 19s

L2_Megatron_LM_To_NeMo_Conversion / main

4m 18s

L2_Megatron_GPT_with_ResetLR_Pretraining_and_Resume_Training_TP2 / main

4m 37s

L2_Megatron_GPT_with_Drop_Optimizer_States_TP2 / main

3m 38s

L2_Megatron_GPT_with_ALiBi_Pretraining_and_Resume_Training_TP2 / main

3m 18s

L2_Megatron_GPT_with_KERPLE_Pretraining_and_Resume_Training_TP2 / main

3m 16s

L2_Megatron_GPT_Pretraining_and_Resume_Training_PP2 / main

4m 9s

L2_Megatron_GPT_Auto_Configurator_TP1_PP1_MBS124 / main

2m 28s

L2_Megatron_GPT_Finetuning_PP2 / main

4m 34s

L2_Megatron_GPT_Finetuning_StarCoder_PP1 / main

1m 17s

L2_Megatron_GPT_Reranker / main

1m 43s

L2_Megatron_GPT_Embedding / main

2m 14s

L2_Megatron_GPT_PEFT_Lora_PP2_O2 / main

2m 43s

L2_Megatron_GPT_PEFT_Lora_TP2_O1 / main

3m 2s

L2_Megatron_GPT_PEFT_Lora_TP2SP1 / main

1m 8s

L2_Megatron_GPT_Eval / main

1m 54s

L2_Megatron_GPT_Eval_PP2 / main

2m 24s

L2_Megatron_GPT_SFT_Eval_inference_seq_len_greaterThan_training_seq_len / main

2m 0s

L2_Megatron_Change_Partitions_Reduce_TP_Num_Partitions_-2_to_1-_and_PP_Num_Partitions_-1_to_2 / main

2m 7s

L2_Megatron_Change_Partitions_Increase_TP_Num_Partitions_-2_to_4-_and_PP_Num_Partitions_-1_to_2 / main

2m 13s

L2_Megatron_Core_T5_Pretraining_and_Resume_Training_TP2 / main

3m 38s

L2_Megatron_T5_with_ALiBi_Pretraining_and_Resume_Training_TP2 / main

3m 11s

L2_Megatron_T5_with_KERPLE_Pretraining_and_Resume_Training_TP2 / main

3m 8s

L2_Megatron_T5_w_Mixture_of_Expert_Pretraining / main

2m 23s

L2_Megatron_UL2_Pretraining_and_Resume_Training_TP2 / main

3m 11s

L2_Megatron_Core_T5_Eval / main

1m 52s

L2_Megatron_Core_T5_PEFT_Lora_TP2 / main

4m 3s

L2_VLM_HF_Transformer_PEFT / main

1m 25s

L2_VLM_HF_Transformer_PEFT_FSDP / main

2m 39s

L2_VLM_HF_Transformer_PEFT_4bit / main

1m 27s

L2_VLM_HF_Transformer_SFT_FSDP2 / main

40s

L2_HF_Transformer_PEFT / main

1m 23s

L2_HF_Transformer_PEFT_nemorun / main

1m 56s

L2_HF_Transformer_PEFT_2gpu / main

2m 58s

L2_HF_Transformer_PEFT_2gpu_nemorun / main

3m 5s

L2_HF_Transformer_SFT_2gpu / main

6m 30s

L2_HF_Transformer_SFT_FSDP2_2gpu / main

1m 44s

L2_HF_Transformer_PT_2gpu / main

5m 37s

L2_HF_Transformer_SFT_2gpu_nemorun / main

3m 26s

L2_HF_Transformer_SFT_2gpu_nemorun_fsdp2 / main

1m 42s

L2_HF_Transformer_PT_2gpu_nemorun / main

2m 31s

L2_HF_Transformer_PT / main

4m 49s

L2_HF_Transformer_PT_nemorun / main

1m 27s

L2_HF_Transformer_SFT / main

6m 8s

L2_HF_Transformer_SFT_nemorun / main

3m 5s

L2_HF_Transformer_SFT_TE_Acceleration / main

7m 4s

L2_HF_Transformer_PT_TE_Acceleration / main

5m 15s

L2_HF_Transformer_SpeechLM_SFT_2gpu / main

2m 23s

L2_Megatron_Mock_Data_Generation_MockGPTDataset / main

3m 4s

L2_Megatron_Mock_Data_Generation_MockT5Dataset / main

2m 8s

L2_TTS_Fast_dev_runs_1_Tacotron_2 / main

55s

L2_TTS_Fast_dev_runs_1_WaveGlow / main

1m 48s

L2_TTS_Fast_dev_runs_1_FastPitch / main

2m 1s

L2_TTS_Fast_dev_runs_1_Hifigan / main

1m 45s

Speech_Checkpoints_tests / main

3m 30s

L2_Stable_Diffusion_Training / main

4m 6s

L2_NeMo_2_GPT_Pretraining_no_transformer_engine / main

3m 48s

L2_NeMo_2_llama3_pretraining_recipe / main

4m 47s

L2_NeMo_2_llama3_fault_tolerance_plugin / main

7m 27s

L2_NeMo_2_llama3_straggler_detection / main

2m 44s

L2_NeMo_2_GPT_DDP_Param_Parity_check / main

3m 4s

L2_NeMo_2_SSM_Pretraining / main

2m 49s

L2_NeMo_2_SSM_Finetuning / main

3m 26s

L2_NeMo_2_HF_MODEL_IMPORT / main

4m 34s

L2_NeMo_2_jit_callback / main

1m 39s

L2_NeMo_2_T5_Pretraining / main

L2_NeMo_2_T5_Finetuning / main

2m 44s

L2_NeMo_2_T5_LoRA / main

1m 47s

L2_NeMo_2_NEVA_MOCK_TRAINING / main

2m 40s

L2_NeMo_2_NEVA_MOCK_PACKED_TRAINING / main

2m 41s

L2_NeMo_2_MLLAMA_MOCK_TRAINING / main

5m 46s

L2_NeMo_2_Mixtral_Pretraining / main

1m 57s

L2_NeMo_2_GPT_SFT_TP1PP1_MBS1 / main

L2_NeMo_2_GPT_SFT_TP1PP1_MBS2 / main

L2_NeMo_2_GPT_SFT_TP1PP2_MBS2 / main

L2_NeMo_2_GPT_SFT_TP2PP1_MBS2 / main

L2_NeMo_2_GPT_SFT_TP1PP1_MBS1_PACKED / main

L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1 / main

1m 36s

L2_NeMo_2_GPT_LoRA_TP1PP1_MBS2 / main

3m 39s

L2_NeMo_2_GPT_LoRA_TP1PP2_MBS2 / main

L2_NeMo_2_GPT_LoRA_TP2PP1_MBS2 / main

L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_PACKED / main

L2_NeMo_2_GPT_DoRA_TP1PP1_MBS1_PACKED / main

L2_NeMo_2_GPT_CLoRA_TP1PP1_MBS1_PACKED / main

L2_NeMo_2_GPT_LoRA_TP1PP1_MBS1_Chat / main

L2_NeMo_2_Mixtral_LoRA_EP2PP1_MBS2 / main

L2_NeMo_2_Mixtral_LoRA_TP1PP1_MBS1 / main

L2_NeMo_2_Mixtral_LoRA_TP2PP1_MBS1 / main

L2_NeMo_2_Mistral_LoRA_TP1PP1_MBS1 / main

L2_NeMo_2_Mistral_LoRA_TP2PP1_MBS1 / main

L2_NEMO_2_LoRA_MERGE / main

L2_NEMO_2_LoRA_Export / main

37s

L2_NEMO_2_LoRA_Inference / main

37s

L2_NeMo_2_NeMo_Mcore_Mixtral_bitexact / main

L2_NeMo_2_PTQ_Llama2_FP8 / main

L2_NeMo_2_Distill_Llama3_TP1PP2 / main

L2_NeMo_2_Export_In_Framework / main

L2_NeMo_2_LLAVA_NEXT_MOCK_TRAINING / main

L2_NeMo_2_VLLM_EXPORT / main

L0_Setup_Test_Data_And_Models / main

1m 23s

OPTIONAL_L2_Megatron_T5_Pretraining_and_Resume_Training_PP2 / main

3m 1s

L0_Unit_Tests_Coverage

Nemo_CICD_Test

Annotations

38 errors

OPTIONAL_L2_Megatron_T5_Pretraining_and_Resume_Training_PP2 / main

Process completed with exit code 1.

L2_NEMO_2_LoRA_Inference / main

Process completed with exit code 1.

L2_NeMo_2_GPT_SFT_TP1PP1_MBS1 / main