From 09c0211e3accc4d4101beba0c448574f1a6e98c0 Mon Sep 17 00:00:00 2001 From: Dmitri Soshnikov <dmitri@soshnikov.com> Date: Tue, 17 May 2022 15:02:30 +0300 Subject: [PATCH] Add text generation lab and takeaways --- README.md | 2 +- lessons/5-NLP/17-GenerativeNetworks/README.md | 14 +++++++++++++- lessons/5-NLP/17-GenerativeNetworks/lab/README.md | 12 ++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) create mode 100644 lessons/5-NLP/17-GenerativeNetworks/lab/README.md diff --git a/README.md b/README.md index e123bbc..9166eb9 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ For a gentle introduction to *AI in the Cloud* topics you may consider taking th <tr><td>14</td><td>Semantic word embeddings. Word2Vec and GloVe</td><td><a href="lessons/5-NLP/14-Embeddings/README.md">Text</td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb">TensorFlow</a></td><td></td></tr> <tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="lessons/5-NLP/15-LanguageModeling">Text</a></td><td>PyTorch</td><td>TensorFlow</td><td></td></tr> <tr><td>16</td><td>Recurrent Neural Networks</td><td><a href="lessons/5-NLP/16-RNN/README.md">Text</a></td><td><a href="lessons/5-NLP/16-RNN/RNNPyTorch.ipynb">PyTorch</a></td><td><a href="lessons/5-NLP/16-RNN/RNNTF.ipynb">TensorFlow</a></td><td></td></tr> -<tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="lessons/5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md">TensorFlow</a></td><td></td></tr> +<tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="lessons/5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.md">TensorFlow</a></td><td><a href="lessons/5-NLP/17-GenerativeNetworks/lab/README.md">Lab</a></td></tr> <tr><td>18</td><td>Transformers. BERT.</td><td><a href="lessons/5-NLP/18-Transformers/README.md">Text</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersPyTorch.md">PyTorch</a></td><td><a href="lessons/5-NLP/18-Transformers/TransformersTF.md">TensorFlow</a></td><td></td></tr> <tr><td>19</td><td>Named Entity Recognition</td><td>Text</td><td>PyTorch</td><td>TensorFlow</td><td></td></tr> <tr><td>20</td><td>Large Language Models, Prompt Programming and Few-Shot Tasks</td><td>Text</td><td>PyTorch</td><td>TensorFlow</td><td></td></tr> diff --git a/lessons/5-NLP/17-GenerativeNetworks/README.md b/lessons/5-NLP/17-GenerativeNetworks/README.md index 5b3002f..8a91433 100644 --- a/lessons/5-NLP/17-GenerativeNetworks/README.md +++ b/lessons/5-NLP/17-GenerativeNetworks/README.md @@ -50,4 +50,16 @@ Have a look at how this soft text generation is implemented in the notebooks. ## [Post-lecture quiz](https://black-ground-0cc93280f.1.azurestaticapps.net/quiz/217) -> ✅ Todo: conclusion, Assignment, challenge, reference. +## [Assignment](lab/README.md) + +We have seen how to generate text character-by-character. In the lab, you will explore word-level text generation. + +## References + +* Different approaches to text generation with Markov Chain, LSTM and GPT-2: [blog post](https://towardsdatascience.com/text-generation-gpt-2-lstm-markov-chain-9ea371820e1e) +* Learn Unit on Text Generation with [PyTorch](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-pytorch/6-generative-networks/?WT.mc_id=academic-15963-dmitryso)/[TensorFlow](https://docs.microsoft.com/learn/modules/intro-natural-language-processing-tensorflow/5-generative-networks/?WT.mc_id=academic-15963-dmitryso) +* Text generation sample in [Keras documentation](https://keras.io/examples/generative/lstm_character_level_text_generation/) + +## Takeaway + +While text generation may be useful in its own right, the major benefits come from the ability to generate text using RNNs from some initial feature vector. For example, text generation is used as part of machine translation (sequence-to-sequence, in this case state vector from *encoder* is used to generate or *decode* translated message), generating textual description of an image (in which case feature vector would come from CNN extractor), etc. diff --git a/lessons/5-NLP/17-GenerativeNetworks/lab/README.md b/lessons/5-NLP/17-GenerativeNetworks/lab/README.md new file mode 100644 index 0000000..6db7214 --- /dev/null +++ b/lessons/5-NLP/17-GenerativeNetworks/lab/README.md @@ -0,0 +1,12 @@ +# Word-level Text Generation using RNNs + +Lab Assignment from [AI for Beginners Curriculum](https://github.com/microsoft/ai-for-beginners). + +## Task + +In this lab, you need to take any book, and use it as a dataset to train word-level text generator. + +## The Dataset + +You are welcome to use any book. You can find a lot of free texts at [Project Gutenberg](https://www.gutenberg.org/), for example, here is a direct link to [Alice's Adventures in Wonderland](https://www.gutenberg.org/files/11/11-0.txt)) by Lewis Carroll. + -- GitLab