Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Good news] Training a 1.6B model on 24G VRAM through offload and additional support for AdamW8bit optimizer！ #156

Open
Pevernow opened this issue Jan 19, 2025 · 0 comments
Open

[Good news] Training a 1.6B model on 24G VRAM through offload and additional support for AdamW8bit optimizer！ #156

Pevernow opened this issue Jan 19, 2025 · 0 comments
Labels
documentation Improvements or additions to documentation

Comments

@Pevernow
Copy link
Contributor

Image

Image

The code is very messy, but it worked.
I may not have enough time to perfect these codes. If someone is willing to help me optimize the code structure and omit some unnecessary offloads, or adjust the timing and objects of offloads, I will be very grateful.
@Pevernow Pevernow changed the title [Good news] Training a 1.6B model on 24G video memory through offload and additional support for AdamW8bit optimizer！ [Good news] Training a 1.6B model on 24G VRAM through offload and additional support for AdamW8bit optimizer！ Jan 19, 2025
@lawrence-cj lawrence-cj added the documentation Improvements or additions to documentation label Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Pevernow @lawrence-cj