From 8f9914e1a80cb77cf34edf6600128e6ec956626a Mon Sep 17 00:00:00 2001 From: Jen Looper <jen.looper@gmail.com> Date: Thu, 12 May 2022 13:21:51 -0400 Subject: [PATCH] ch 15 first pass --- lessons/5-NLP/15-LanguageModeling/README.md | 30 +++++++++++++++------ 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/lessons/5-NLP/15-LanguageModeling/README.md b/lessons/5-NLP/15-LanguageModeling/README.md index 96a2e5c..489ea24 100644 --- a/lessons/5-NLP/15-LanguageModeling/README.md +++ b/lessons/5-NLP/15-LanguageModeling/README.md @@ -1,22 +1,36 @@ # Language Modeling -Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language. +Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language. -The main idea behind language modeling is training them on unlabeled datesets in unsupervised manner. It is important, because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample. +## [Pre-lecture quiz](tbd) -## Training embeddings +The main idea behind language modeling is training them on unlabeled datasets in an unsupervised manner. This is important because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we can build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample. -In our previous examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. +## Training Embeddings - +In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures. + + > Image from [this paper](https://arxiv.org/pdf/1301.3781.pdf) -The idea of CBoW is exactly predicting a missing word, however, to do this we take a small sliding window of text tokens (we can denote them from W<sub>-2</sub> to W<sub>2</sub>), and train a model to predict the central word W<sub>0</sub> from few surrounding words. +The idea underpinning CBoW involves how to predict a missing word, but to do this we take a small sliding window of text tokens. We can denote them from W<sub>-2</sub> to W<sub>2</sub>, and train a model to predict the central word W<sub>0</sub> from a few surrounding words. + +## Conclusion + +TBD + +## 馃殌 Challenge -## More Info +TBD + +## [Post-lecture quiz](tbd) + +## Review & Self Study * [Official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). * [Official TensorFlow tutorial on training Word2Vec model](https://www.TensorFlow.org/tutorials/text/word2vec). -* Using **gensim** framework to train most commonly used embeddings in a few lines of code is as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). +* Using the **gensim** framework to train most commonly used embeddings in a few lines of code is described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html). + +## [Assignment: Notebooks](assignment.md) - TBD -- GitLab