A: Word2Vec is a popular neural network model used to generate dense vector representations of words, called word embeddings. These word embeddings capture the semantic meaning of words based on the context in which they appear in a large corpus of text. Word2Vec was introduced by Tomas Mikolov and his team at Google in 2013, and it revolutionized natural language processing (NLP) by showing that words with similar meanings tend to have similar vector representations in a continuous vector space.
-
Word Embeddings: Word2Vec learns to map words into a high-dimensional continuous vector space such that words with similar meanings or usage contexts are closer together in the space. These vectors can be used for various NLP tasks, such as similarity comparison, word analogy, and more.
-
Contextual Similarity: Word2Vec assumes that words that share similar contexts in the text (i.e., they appear in similar neighboring words or contexts) have similar meanings. It is based on the idea that "you shall know a word by the company it keeps."
Word2Vec uses a shallow neural network to learn word embeddings. There are two primary architectures for training a Word2Vec model:
- The goal of the CBOW model is to predict a target word (center word) from a fixed-size context window of surrounding words. The context window contains words surrounding the target word, and the model uses the context to predict the center word.
- The input to the model is a set of context words, and the output is the probability distribution of the target word. CBOW is a "context to target" model.
CBOW Architecture:
- Input: A context window of words surrounding the target word (e.g., for a target word "dog," the context could be the words "the," "chased," "a").
- Output: The model predicts the target word from the context.
- The model averages the word embeddings of the context words and tries to predict the target word.
Training Process: The model adjusts its weights so that the probability of predicting the target word is maximized given the context words.
- In contrast to CBOW, the skip-gram model tries to predict the context words given a target word (center word). This model is generally better at handling smaller datasets and rare words because it leverages the target word to predict multiple context words.
- The skip-gram model is a "target to context" model.
Skip-gram Architecture:
- Input: A single target word (center word).
- Output: The model predicts the probability distribution of multiple context words surrounding the target word.
- The model learns to maximize the probability of observing the context words, given the target word.
Example: If the target word is "dog," the skip-gram model might predict the words "the," "chased," and "a" as its context words.
Both CBOW and skip-gram models are trained to maximize the probability of the context words given the target word (in skip-gram) or the target word given the context words (in CBOW). The training objective can be written as:
[ P(w_o | w_c) = \frac{\exp(v_{w_o}^T v_{w_c})}{\sum_{w=1}^{V} \exp(v_w^T v_{w_c})} ]
Where:
- ( w_o ) is the output word (target word),
- ( w_c ) is the context word,
- ( v_{w_o} ) and ( v_{w_c} ) are the vector representations (embeddings) of the output and context words,
- ( V ) is the vocabulary size.
This is essentially a softmax function over all the words in the vocabulary, which calculates the probability of each word given the context word.
The optimization process involves adjusting the word embeddings to maximize the likelihood of the correct word given the context.
One of the challenges in training Word2Vec is the softmax normalization over the entire vocabulary, which can be computationally expensive when the vocabulary is large (e.g., millions of words). To address this, negative sampling is often used.
- Negative sampling is a technique that simplifies the softmax calculation by only considering a small number of "negative" samples (words that are unlikely to be the context word).
- Instead of updating the weights for all words in the vocabulary, the model only updates the weights for the target word and a small number of negative samples.
- This dramatically speeds up training by reducing the number of computations per update.
Once the Word2Vec model is trained, the result is a set of word embeddings — high-dimensional vectors representing words. These embeddings capture semantic relationships between words.
- Words with similar meanings (like "dog" and "cat") will have embeddings that are close to each other in the vector space.
- Words that frequently appear in similar contexts (e.g., "bank" and "river") will also have similar embeddings.
Word2Vec embeddings are particularly known for their ability to handle word analogies. For example, the model can understand analogies like:
king - man + woman = queen
This happens because the difference between the vector representations of "king" and "man" is similar to the difference between "woman" and "queen" in the vector space. This property arises naturally from the training process.
- Efficient: Once trained, word embeddings are fast to compute and can be used for a variety of NLP tasks, such as sentiment analysis, machine translation, or clustering.
- Captures Semantic Meaning: Word2Vec embeddings capture word meanings based on context and can show relationships like synonyms, antonyms, or even word analogies.
- Flexible: Word embeddings can be transferred and fine-tuned for various downstream NLP tasks, making them versatile.
- Context Independence: Word2Vec assumes a fixed representation for each word regardless of context. This means that polysemy (words with multiple meanings, like "bank") is not well-handled.
- Out-of-Vocabulary Words: Words not seen during training cannot be represented by the model unless some form of subword-based approach (like FastText) is used.
- Pre-trained Embeddings: While pre-trained word embeddings (e.g., GloVe, Word2Vec) work well in many cases, they may not capture domain-specific meanings without further fine-tuning.
- GloVe (Global Vectors for Word Representation): Another popular word embedding model that is based on factorizing the word co-occurrence matrix.
- FastText: An extension of Word2Vec that represents words as bags of character n-grams, making it better at handling rare words or morphologically rich languages.
- ELMo, BERT: Modern pre-trained models like ELMo and BERT go beyond fixed word embeddings by incorporating contextual word representations, which allows them to handle polysemy and other complexities of language more effectively.
In summary, Word2Vec is a foundational model for learning word embeddings that represent words in a continuous vector space. It uses shallow neural networks (CBOW or skip-gram) to predict words based on context, allowing it to capture semantic relationships between words. While Word2Vec is still widely used, more recent models like BERT and GPT provide context-dependent embeddings and have become the new standard in NLP.