Reproduce Megatron-LM RETRO

[rstarmer]

As a CTO, I want to use immutable dependencies for our product engineering so that engineering has consistent dependencies.

---

Acceptance Criteria:

• Reproducible build of megatron-lm.
• Train a retro database with the 1.4T dataset.
• Inference the retro model.
• DOD version 0 met.

---

References:

[rstarmer]

• Install is missing required libraries
• NVIDIA APEX support needs to be removed (APEX is another issue all together)

[sdake]

@rstarmer I have paused this work.

I have improved the Megatron_LM codebase so that:

• Installation works
• I have removed APEX.

No idea if the APEX removal works, although it seemed correct when I made the changes. While your traveling, if you want to take a look, you may wish to take a look. For detail about how we are structuring our workloads e.g.: Megatron_LM retro Please see README.md.

I have placed the paused tarball into our cold storage.

Thank you,
-steve