Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language.
Semantic embeddings, such as Word2Vec and GloVe, are in fact a first step towards **language modeling** - creating models that somehow *understand* (or *represent*) the nature of the language.
The main idea behind language modeling is training them on unlabeled datesets in unsupervised manner. It is important, because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample.
## [Pre-lecture quiz](tbd)
## Training embeddings
The main idea behind language modeling is training them on unlabeled datasets in an unsupervised manner. This is important because we have huge amounts of unlabeled text available, while the amount of labeled text would always be limited by the amount of effort we can spend on labeling. Most often, we can build language models that can **predict missing words** in the text, because it is easy to mask out a random word in text and use it as a training sample.
In our previous examples, we have been using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures.
In our previous examples, we used pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either CBoW, or Skip-gram architectures.
> Image from [this paper](https://arxiv.org/pdf/1301.3781.pdf)
The idea of CBoW is exactly predicting a missing word, however, to do this we take a small sliding window of text tokens (we can denote them from W<sub>-2</sub> to W<sub>2</sub>), and train a model to predict the central word W<sub>0</sub> from few surrounding words.
The idea underpinning CBoW involves how to predict a missing word, but to do this we take a small sliding window of text tokens. We can denote them from W<sub>-2</sub> to W<sub>2</sub>, and train a model to predict the central word W<sub>0</sub> from a few surrounding words.
## Conclusion
TBD
## 🚀 Challenge
## More Info
TBD
## [Post-lecture quiz](tbd)
## Review & Self Study
*[Official PyTorch tutorial on Language Modeling](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
*[Official TensorFlow tutorial on training Word2Vec model](https://www.TensorFlow.org/tutorials/text/word2vec).
* Using **gensim** framework to train most commonly used embeddings in a few lines of code is as described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
* Using the **gensim** framework to train most commonly used embeddings in a few lines of code is described [in this documentation](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html).
You have almost finished this course, and I hope that by now you clearly see that AI is based on a number of formal mathematical methods that allow us to find relationships in data and train models to replicate the human behavior in some areas. At this point in history, we consider AI to be a very powerful tool to extract patterns from data, and to apply those patterns to solve new problems.
You have almost finished this course, and I hope that by now you clearly see that AI is based on a number of formal mathematical methods that allow us to find relationships in data and train models to replicate some aspects of human behavior. At this point in history, we consider AI to be a very powerful tool to extract patterns from data, and to apply those patterns to solve new problems.
However, in science fiction we often see stories where AI presents a danger to the humankind. Usually those stories are centered around some sort of AI rebellion, when AI decides to confront human beings. This implies that AI has some sort of emotions, or can take decisions unforeseen by its developers.
The kind of AI that we have learned about in this course is nothing more than large matrix arithmetics. It is a very powerful tool to help us solve our problems, and as any other powerful tool - it can be used for good and for bad purposes. What's also important, it can be *misused*.
However, in science fiction we often see stories where AI presents a danger to humankind. Usually those stories are centered around some sort of AI rebellion, when AI decides to confront human beings. This implies that AI has some sort of emotion or can take decisions unforeseen by its developers.
The kind of AI that we have learned about in this course is nothing more than large matrix arithmetic. It is a very powerful tool to help us solve our problems, and as any other powerful tool - it can be used for good and for bad purposes. Importantly, it can be *misused*.
## Principles of Responsible AI
To avoid this accidental misuse of AI, Microsoft states important [Principles of Responsible AI](https://www.microsoft.com/ai/responsible-ai).
To avoid this accidental or purposeful misuse of AI, Microsoft states the important [Principles of Responsible AI](https://www.microsoft.com/ai/responsible-ai?WT.mc_id=academic-57639-dmitryso). The following concepts underpin these principles:
***Fairness** is related to the important problem of *model biases*, which can be caused by using biased data for training. For example, when we try to predict the probability of getting a software developer job for a person, the model is likely to give higher preference to males - just because the training dataset was likely biased towards male audience. We need to carefully balance training data and investigate the model to avoid biases, and make sure that the model takes into account more relevant features.
***Reliability and Safety**. By their nature, AI models can make mistakes. A neural network returns probabilities, and we need to take it into account when making decisions. Every model has some precision and recall, and we need to understand that to prevent harm that a wrong advice can cause.
***Privacy and Security** have some AI-specific flavour. For example, when we use some data for training a model, this data becomes somehow "integrated" into the model. On one hand, that increases security and privacy, on the other - we need to remember which data the model was trained on.
***Fairness** is related to the important problem of *model biases*, which can be caused by using biased data for training. For example, when we try to predict the probability of getting a software developer job for a person, the model is likely to give higher preference to males - just because the training dataset was likely biased towards a male audience. We need to carefully balance training data and investigate the model to avoid biases, and make sure that the model takes into account more relevant features.
***Reliability and Safety**. By their nature, AI models can make mistakes. A neural network returns probabilities, and we need to take it into account when making decisions. Every model has some precision and recall, and we need to understand that to prevent harm that wrong advice can cause.
***Privacy and Security** have some AI-specific implications. For example, when we use some data for training a model, this data becomes somehow "integrated" into the model. On one hand, that increases security and privacy, on the other - we need to remember which data the model was trained on.
***Inclusiveness** means that we are not building AI to replace people, but rather to augment people and make our work more creative. It is also related to fairness, because when dealing with underrepresented communities, most of the datasets we collect are likely to be biased, and we need to make sure that those communities are included and correctly handled by AI.
***Transparency**. This includes making sure that we are always clear about AI being used. Also, wherever possible, we want to use AI systems that are *interpretable*.
***Accountability**. When AI models come up with some decisions, it is not always clear who is responsible for those decisions. We need to make sure that we understand the responsibility of AI decisions. In most of the cases we would want to include human being into the loop of taking important decisions, and people are made accountable.
***Accountability**. When AI models come up with some decisions, it is not always clear who is responsible for those decisions. We need to make sure that we understand where responsibility of AI decisions lies. In most cases we would want to include human beings into the loop of making important decisions, so that actual people are made accountable.
## Tools for Responsible AI
At Microsoft, we have developed [Responsible AI Toolbox](https://github.com/microsoft/responsible-ai-toolbox), which contains a set of tools:
Microsoft has developed the [Responsible AI Toolbox](https://github.com/microsoft/responsible-ai-toolbox) which contains a set of tools:
* Interpretability Dashboard (InterpretML)
* Fairness Dashboard (FairLearn)
...
...
@@ -28,10 +30,10 @@ At Microsoft, we have developed [Responsible AI Toolbox](https://github.com/micr
- EconML - tool for Causal Analysis, which focuses on what-if questions
- DiCE - tool for Counterfactual Analysis allows you to see which features need to be changed to affect the decision of the model
## Model Interpretability
For more information about AI Ethics, please visit [this lesson](https://github.com/microsoft/ML-For-Beginners/tree/main/1-Introduction/3-fairness) on the Machine Learning Curriculum which includes assignments.
* Glass box models
* Black box models
## Review & Self Study
## Model Fairness
Take this [Learn Path](https://docs.microsoft.com/learn/modules/responsible-ai-principles/) to learn more about responsible AI.
