DAPT playbooks - with NeMo 2.0 #12067
Conversation
jvamaraju
requested review from
pablo-garay,
ko3n1g,
thomasdhc and
chtruong814
as code owners
github-actions
bot
added
core
Signed-off-by: jvamaraju <[email protected]>
Signed-off-by: jvamaraju <[email protected]>
| "id": "c43ef563", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# NeMo Tools and Resources\n", |
There was a problem hiding this comment.
Let's change this to nvcr.io/nvidia/nemo:25.02
There was a problem hiding this comment.
@suiyoubi, @jvamaraju was running into issues when testing with 25.02, for now we should keep it to the container we've tested it on may be? Once you push the new container with the PR for the llama2 tutorial merged we can update this
There was a problem hiding this comment.
What issue were you encountered ? I am fine with leaving it as whatever container you have tested for now.
There was a problem hiding this comment.
Thank you, will leave it as is for now. @jvamaraju can you share details about the issue?
There was a problem hiding this comment.
Thank you. Yes lets leave it to 24.12 for now. When we add llama27b recipes...nemo fails to recompile using 25.02 container.
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
| "#### Benefits of bin/idx format for Pretraining:\n", | ||
| "\n", | ||
| "* **Optimized I/O Performance:** The binary format enables quicker data reads and reduces latency, allowing the model to continuously access data at high speeds.\n", | ||
| "* **Efficient Memory Usage:** Data in bin/idx format consumes less memory during loading, making it suitable for large datasets and enabling better use of available system resources.\n", |
There was a problem hiding this comment.
You are modifying these in the later cells, let's remove them here
There was a problem hiding this comment.
Which cell are you referring to?
There was a problem hiding this comment.
I mean you are overwriting the TP settings in the later cell, you don't want to overwrite it twice to confuse the reader
tutorials/llm/llama/domain-adaptive-pretraining/code/domain_adaptive_pretraining_nemo2.0.ipynb
Show resolved
Hide resolved
* DAPT with NeMo 2.0 * DAPT with NeMo 2.0 * Apply isort and black reformatting Signed-off-by: jvamaraju <[email protected]> * Deleting file not needed * Update README.md Signed-off-by: jvamaraju <[email protected]> * Addressing feedback from PR review for DAPT playbook with nemo 2.0 * Addressing feedback for DAPT with nemo 2.0 * Addressing feedback for DAPT with nemo 2.0- local executor * Add Copyright --------- Signed-off-by: jvamaraju <[email protected]> Signed-off-by: jvamaraju <[email protected]> Co-authored-by: jvamaraju <[email protected]> Co-authored-by: aastha <[email protected]> Co-authored-by: Ao Tang <[email protected]> Signed-off-by: Youngeun Kwon <[email protected]>
Added Custom tokenization + DAPT playbooks
Previous PRs (closed but not merged)