This repository is dedicated to translating the Python code from the book Build a Large Language Model (from Scratch) by Sebastian Raschka into Julia. The goal is to provide a Julia-based implementation of the concepts and techniques outlined in the book.
Sebastian Raschka's Build a Large Language Model (from Scratch) provides a hands-on approach to understanding the fundamental concepts behind large language models (LLMs), covering everything from tokenization to model training and inference. The book uses Python as the primary language, and this repository aims to make the same techniques accessible to Julia enthusiasts.
- Julia Translation: Translate the Python code examples from the book into idiomatic Julia, ensuring clarity and performance.
- Julia Ecosystem: Showcase how Julia's libraries and tools can be used to implement LLM components, such as tokenization, embedding layers, and model architectures.
- Learning Resource: Provide Julia users with a resource to learn about LLM development in their preferred language.
The code in this repository is explained through a series of Medium articles. Each new commit in this repository will be accompanied by a new Medium article. Below are the Medium articles published so far:
Section 3.3.2: Computing Attention Weights for All Input Tokens. Here is the friend link for non-Medium members.
Section 3.3.1: Implementing a Simple Self-Attention Mechanism with Python and Julia. Here is the friend link for non-Medium members.
Section 2.8: How to Add Positional Encodings to Token Embeddings in Python and Julia for LLM Training. Here is the friend link for non-Medium members.
Section 2.7: From Token IDs to Embeddings: LLM Training with Python and Julia. Here is the friend link for non-Medium members.
Section 2.6b: Creating a PyTorch-Style DataLoader in Julia for LLM Training Batches. Here is the friend link for non-Medium members.
Section 2.6a: Preparing a PyTorch-Style Dataset in Julia for LLM Training. Here is the friend link for non-Medium members.
Section 2.5: Byte Pair Encoding (BPE) in Action: Julia and Python Side-by-Side. Here is the friend link for non-Medium members.
Section 2.3: From Classes to Structs: Translating a Python Tokenizer into Julia’s Functional Style. Here is the friend link for non-Medium members.
Section 2.3: Creating a Vocabulary Dictionary for English Words in Julia. Here is the friend link for non-Medium members.
Section
Contributions are welcome! If you find a bug, have a suggestion, or want to improve the code, feel free to open an issue or submit a pull request.
If you have questions or feedback, feel free to open an issue or reach out to me on Twitter.
Sebastian Raschka for the incredible book and the Python code examples that inspired this repo.
Happy coding! 🚀