From 92e8f9ae061d058d06ea7a4024f8b32f767318ee Mon Sep 17 00:00:00 2001
From: Dmitri Soshnikov <dmitri@soshnikov.com>
Date: Fri, 14 Jan 2022 10:48:30 +0300
Subject: [PATCH] Add RNNs and Embedding readme
---
5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb | 5 +
5-NLP/14-Embeddings/README.md | 41 +
5-NLP/16-RNN/README.md | 58 +
5-NLP/16-RNN/RNNPyTorch.ipynb | 486 ++++++
5-NLP/16-RNN/RNNTF.ipynb | 443 ++++++
.../images/long-short-term-memory-cell.svg | 1334 +++++++++++++++++
5-NLP/16-RNN/images/multi-layer-lstm.jpg | Bin 0 -> 29122 bytes
5-NLP/16-RNN/images/rnn-anatomy.png | Bin 0 -> 16735 bytes
5-NLP/16-RNN/images/rnn.png | Bin 0 -> 18140 bytes
5-NLP/16-RNN/torchnlp.py | 104 ++
5-NLP/README.md | 4 +-
README.md | 6 +-
12 files changed, 2477 insertions(+), 4 deletions(-)
create mode 100644 5-NLP/16-RNN/README.md
create mode 100644 5-NLP/16-RNN/RNNPyTorch.ipynb
create mode 100644 5-NLP/16-RNN/RNNTF.ipynb
create mode 100644 5-NLP/16-RNN/images/long-short-term-memory-cell.svg
create mode 100644 5-NLP/16-RNN/images/multi-layer-lstm.jpg
create mode 100644 5-NLP/16-RNN/images/rnn-anatomy.png
create mode 100644 5-NLP/16-RNN/images/rnn.png
create mode 100644 5-NLP/16-RNN/torchnlp.py
diff --git a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb
index c77795c..c70f64a 100644
--- a/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb
+++ b/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb
@@ -686,6 +686,11 @@
"\n",
"The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models in the next unit.\n"
]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": []
}
],
"metadata": {
diff --git a/5-NLP/14-Embeddings/README.md b/5-NLP/14-Embeddings/README.md
index e69de29..b94abb0 100644
--- a/5-NLP/14-Embeddings/README.md
+++ b/5-NLP/14-Embeddings/README.md
@@ -0,0 +1,41 @@
+# Embeddings
+
+When training classifiers based on BoW or TF/IDF, we operated on high-dimensional bag-of-words vectors with length `vocab_size`, and we were explicitly converting from low-dimensional positional representation vectors into sparse one-hot representation. This one-hot representation is not memory-efficient, in addition, each word is treated independently from each other, i.e. one-hot encoded vectors do not express any semantic similarity between words.
+
+The idea of **embedding** is to represent words by lower-dimensional dense vectors, which somehow reflect semantic meaning of a word. We will later discuss how to build meaningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector.
+
+So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input, allowing us to avoid creating large one-hot-encoded vectors.
+
+By using embedding layer as a first layer in our classifier network, we can switch from bag-or-words to **embedding bag** model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `average` or `max`.
+
+
+
+## Continue in Notebooks
+
+* [Embeddings with PyTorch](EmbeddingsPyTorch.ipynb)
+* [Embeddings Tensorflow](EmbeddingsTF.ipynb)
+
+## Semantic Embeddings: Word2Vec
+
+While embedding layer learnt to map words to vector representation, however, this representation did not necessarily have much semantical meaning. It would be nice to learn such vector representation that similar words or symonims correspond to vectors that are close to each other in terms of some vector distance (eg. Euclidean distance).
+
+To do that, we need to pre-train our embedding model on a large collection of text in a specific way. One of the first ways to train semantic embeddings is called [Word2Vec](https://en.wikipedia.org/wiki/Word2vec). It is based on two main architectures that are used to produce a distributed representation of words:
+
+ - **Continuous bag-of-words** (CBoW) — in this architecture, we train the model to predict a word from surrounding context. Given the ngram $(W_{-2},W_{-1},W_0,W_1,W_2)$, the goal of the model is to predict $W_0$ from $(W_{-2},W_{-1},W_1,W_2)$.
+ - **Continuous skip-gram** is opposite to CBoW. The model uses surrounding window of context words to predict the current word.
+
+CBoW is faster, while skip-gram is slower, but does a better job of representing infrequent words.
+
+
+
+Word2Vec pre-trained embeddings (as well as other similar models, such as GloVe) can also be used in place of embedding layer in neural networks. However, we need to deal with vocabularies, because the vocabulary used to pre-train Word2Vec/GloVe is likely to differ from the vocabulary in our text corpus. Have a look into Notebooks to see how this problem can be resolved.
+
+## Contextual Embeddings
+
+One key limitation of tradition pretrained embedding representations such as Word2Vec is the problem of word sense disambiguation. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word 'play' have different meanings depending on the context they are used in.
+
+For example word 'play' in those two different sentences have quite different meaning:
+- I went to a **play** at the theature.
+- John wants to **play** with his friends.
+
+The pretrained embeddings above represent both of these meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the **language model**, which is trained on a large corpus of text, and *knows* how words can be put together in different contexts. Discussing contextual embeddings is out of scope for this tutorial, but we will come back to them when talking about language models later in the course.
diff --git a/5-NLP/16-RNN/README.md b/5-NLP/16-RNN/README.md
new file mode 100644
index 0000000..0f403d5
--- /dev/null
+++ b/5-NLP/16-RNN/README.md
@@ -0,0 +1,58 @@
+# Recurrent Neural Networks
+
+In the previous sections, we have been using rich semantic representations of text, and a simple linear classifier on top of the embeddings. What this architecture does is to capture aggregated meaning of words in a sentence, but it does not take into account the **order** of words, because aggregation operation on top of embeddings removed this information from the original text. Because these models are unable to model word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering.
+
+To capture the meaning of text sequence, we need to use another neural network architecture, which is called a **recurrent neural network**, or RNN. In RNN, we pass our sentence through the network one symbol at a time, and the network produces some **state**, which we then pass to the network again with the next symbol.
+
+
+
+Given the input sequence of tokens X<sub>0</sub>,...,X<sub>n</sub>, RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using back propagation. Each network block takes a pair (X<sub>i</sub>,S<sub>i</sub>) as an input, and produces S<sub>i+1</sub> as a result. Final state S<sub>n</sub> or (output Y<sub>n</sub>) goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass.
+
+Because state vectors S<sub>0</sub>,...,S<sub>n</sub> are passed through the network, it is able to learn the sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector, resulting in negation.
+
+> Since weights of all RNN blocks on the picture are shared, the same picture can be represented as one block (on the right) with a recurrent feedback loop, which passes output state of the network back to the input.
+
+## Anatomy of RNN Cell
+
+Let's see how simple RNN cell is organized. It accepts previous state S<sub>i-1</sub> and current symbol X<sub>i</sub> as inputs, and has to produce output state S<sub>i</sub> (and, sometimes, we are also interested in some other output Y<sub>i</sub>, as in case with generative networks).
+
+Simple RNN cell has two weight matrices inside: one transforms input symbol (let call it W), and another one transforms input state (H). In this case the output of the network is calculated as σ(W×X<sub>i</sub>+H×S<sub>i-1</sub>+b), where σ is the activation function, b is additional bias.
+
+
+
+In many cases, input tokens are passed through the embedding layer before entering the RNN to lower the dimensionality. In this case, if the dimension of the input vectors is *emb_size*, and state vector is *hid_size* - the size of W is *emb_size*×*hid_size*, and the size of H is *hid_size*×*hid_size*.
+
+## Long Short Term Memory (LSTM)
+
+One of the main problems of classical RNNs is so-called **vanishing gradients** problem. Because RNNs are trained end-to-end in one back-propagation pass, it is having hard times propagating error to the first layers of the network, and thus the network cannot learn relationships between distant tokens. One of the ways to avoid this problem is to introduce **explicit state management** by using so called **gates**. There are two most known architectures of this kind: **Long Short Term Memory** (LSTM) and **Gated Relay Unit** (GRU).
+
+
+
+LSTM Network is organized in a manner similar to RNN, but there are two states that are being passed from layer to layer: actual state C, and hidden vector H. At each unit, hidden vector H<sub>i</sub> is concatenated with input X<sub>i</sub>, and they control what happens to the state C via **gates**. Each gate is a neural network with sigmoid activation (output in the range [0,1]), which can be thought of as bitwise mask when multiplied by the state vector. There are the following gates (from left to right on the picture above):
+* **forget gate** takes hidden vector and determines, which components of the vector C we need to forget, and which to pass through.
+* **input gate** takes some information from the input and hidden vector, and inserts it into state.
+* **output gate** transforms state via some linear layer with *tanh* activation, then selects some of its components using hidden vector H<sub>i</sub> to produce new state C<sub>i+1</sub>.
+
+Components of the state C can be thought of as some flags that can be switched on and off. For example, when we encounter a name *Alice* in the sequence, we may want to assume that it refers to female character, and raise the flag in the state that we have female noun in the sentence. When we further encounter phrases *and Tom*, we will raise the flag that we have plural noun. Thus by manipulating state we can supposedly keep track of grammatical properties of sentence parts.
+
+> **Note**: A great resource for understanding internals of LSTM is this great article [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.
+
+## Bidirectional and multilayer RNNs
+
+We have discussed recurrent networks that operate in one direction, from beginning of a sequence to the end. It looks natural, because it resembles the way we read and listen to speech. However, since in many practical cases we have random access to the input sequence, it might make sense to run recurrent computation in both directions. Such networks are call **bidirectional** RNNs. When dealing with bidirectional network, we would need two hidden state vectors, one for each direction.
+
+Recurrent network, one-directional or bidirectional, captures certain patterns within a sequence, and can store them into state vector or pass into output. As with convolutional networks, we can build another recurrent layer on top of the first one to capture higher level patterns, build from low-level patterns extracted by the first layer. This leads us to the notion of **multi-layer RNN**, which consists of two or more recurrent networks, where output of the previous layer is passed to the next layer as input.
+
+
+
+*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López*
+
+## Continue to Notebooks
+
+* [RNNs with PyTorch](RNNPyTorch.ipynb)
+* [RNNs with Tensorflow](RNNTF.ipynb)
+
+## RNNs for other tasks
+
+In this unit, we have seen that RNNs can be used for sequence classification, but in fact, they can handle many more tasks, such as text generation, machine translation, and more. We will consider those tasks in the next unit.
+
diff --git a/5-NLP/16-RNN/RNNPyTorch.ipynb b/5-NLP/16-RNN/RNNPyTorch.ipynb
new file mode 100644
index 0000000..2f7f6d0
--- /dev/null
+++ b/5-NLP/16-RNN/RNNPyTorch.ipynb
@@ -0,0 +1,486 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Recurrent neural networks\n",
+ "\n",
+ "In the previous module, we have been using rich semantic representations of text, and a simple linear classifier on top of the embeddings. What this architecture does is to capture aggregated meaning of words in a sentence, but it does not take into account the **order** of words, because aggregation operation on top of embeddings removed this information from the original text. Because these models are unable to model word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering.\n",
+ "\n",
+ "To capture the meaning of text sequence, we need to use another neural network architecture, which is called a **recurrent neural network**, or RNN. In RNN, we pass our sentence through the network one symbol at a time, and the network produces some **state**, which we then pass to the network again with the next symbol.\n",
+ "\n",
+ "<img alt=\"RNN\" src=\"images/rnn.png\" width=\"60%\"/>\n",
+ "\n",
+ "Given the input sequence of tokens $X_0,\\dots,X_n$, RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using back propagation. Each network block takes a pair $(X_i,S_i)$ as an input, and produces $S_{i+1}$ as a result. Final state $S_n$ or output $X_n$ goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass.\n",
+ "\n",
+ "Because state vectors $S_0,\\dots,S_n$ are passed through the network, it is able to learn the sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector, resulting in negation. \n",
+ "\n",
+ "> Since weights of all RNN blocks on the picture are shared, the same picture can be represented as one block (on the right) with a recurrent feedback loop, which passes output state of the network back to the input.\n",
+ "\n",
+ "Let's see how recurrent neural networks can help us classify our news dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Loading dataset...\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "d:\\WORK\\ai-for-beginners\\5-NLP\\16-RNN\\data\\train.csv: 29.5MB [00:01, 28.3MB/s] \n",
+ "d:\\WORK\\ai-for-beginners\\5-NLP\\16-RNN\\data\\test.csv: 1.86MB [00:00, 9.72MB/s] \n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Building vocab...\n"
+ ]
+ }
+ ],
+ "source": [
+ "import torch\n",
+ "import torchtext\n",
+ "from torchnlp import *\n",
+ "train_dataset, test_dataset, classes, vocab = load_dataset()\n",
+ "vocab_size = len(vocab)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Simple RNN classifier\n",
+ "\n",
+ "In case of simple RNN, each recurrent unit is a simple linear network, which takes concatenated input vector and state vector, and produce a new state vector. PyTorch represents this unit with `RNNCell` class, and a networks of such cells - as `RNN` layer.\n",
+ "\n",
+ "To define an RNN classifier, we will first apply an embedding layer to lower the dimensionality of input vocabulary, and then have RNN layer on top of it: "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class RNNClassifier(torch.nn.Module):\n",
+ " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
+ " super().__init__()\n",
+ " self.hidden_dim = hidden_dim\n",
+ " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
+ " self.rnn = torch.nn.RNN(embed_dim,hidden_dim,batch_first=True)\n",
+ " self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
+ "\n",
+ " def forward(self, x):\n",
+ " batch_size = x.size(0)\n",
+ " x = self.embedding(x)\n",
+ " x,h = self.rnn(x)\n",
+ " return self.fc(x.mean(dim=1))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "> **Note:** We use untrained embedding layer here for simplicity, but for even better results we can use pre-trained embedding layer with Word2Vec or GloVe embeddings, as described in the previous unit. For better understanding, you might want to adapt this code to work with pre-trained embeddings.\n",
+ "\n",
+ "In our case, we will use padded data loader, so each batch will have a number of padded sequences of the same length. RNN layer will take the sequence of embedding tensors, and produce two outputs: \n",
+ "* $x$ is a sequence of RNN cell outputs at each step\n",
+ "* $h$ is a final hidden state for the last element of the sequence\n",
+ "\n",
+ "We then apply a fully-connected linear classifier to get the number of class.\n",
+ "\n",
+ "> **Note:** RNNs are quite difficult to train, because once the RNN cells are unrolled along the sequence length, the resulting number of layers involved in back propagation is quite large. Thus we need to select small learning rate, and train the network on larger dataset to produce good results. It can take quite a long time, so using GPU is preferred."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "3200: acc=0.3090625\n",
+ "6400: acc=0.38921875\n",
+ "9600: acc=0.4590625\n",
+ "12800: acc=0.511953125\n",
+ "16000: acc=0.5506875\n",
+ "19200: acc=0.57921875\n",
+ "22400: acc=0.6070089285714285\n",
+ "25600: acc=0.6304296875\n",
+ "28800: acc=0.6484027777777778\n",
+ "32000: acc=0.66509375\n",
+ "35200: acc=0.6790056818181818\n",
+ "38400: acc=0.6929166666666666\n",
+ "41600: acc=0.7035817307692308\n",
+ "44800: acc=0.7137276785714286\n",
+ "48000: acc=0.72225\n",
+ "51200: acc=0.73001953125\n",
+ "54400: acc=0.7372794117647059\n",
+ "57600: acc=0.7436631944444444\n",
+ "60800: acc=0.7503947368421052\n",
+ "64000: acc=0.75634375\n",
+ "67200: acc=0.7615773809523809\n",
+ "70400: acc=0.7662642045454545\n",
+ "73600: acc=0.7708423913043478\n",
+ "76800: acc=0.7751822916666666\n",
+ "80000: acc=0.7790625\n",
+ "83200: acc=0.7825\n",
+ "86400: acc=0.7858564814814815\n",
+ "89600: acc=0.7890513392857142\n",
+ "92800: acc=0.7920474137931034\n",
+ "96000: acc=0.7952708333333334\n",
+ "99200: acc=0.7982258064516129\n",
+ "102400: acc=0.80099609375\n",
+ "105600: acc=0.8037594696969697\n",
+ "108800: acc=0.8060569852941176\n"
+ ]
+ }
+ ],
+ "source": [
+ "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=padify, shuffle=True)\n",
+ "net = RNNClassifier(vocab_size,64,32,len(classes)).to(device)\n",
+ "train_epoch(net,train_loader, lr=0.001)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Long Short Term Memory (LSTM)\n",
+ "\n",
+ "One of the main problems of classical RNNs is so-called **vanishing gradients** problem. Because RNNs are trained end-to-end in one back-propagation pass, it is having hard times propagating error to the first layers of the network, and thus the network cannot learn relationships between distant tokens. One of the ways to avoid this problem is to introduce **explicit state management** by using so called **gates**. There are two most known architectures of this kind: **Long Short Term Memory** (LSTM) and **Gated Relay Unit** (GRU).\n",
+ "\n",
+ "\n",
+ "\n",
+ "LSTM Network is organized in a manner similar to RNN, but there are two states that are being passed from layer to layer: actual state $c$, and hidden vector $h$. At each unit, hidden vector $h_i$ is concatenated with input $x_i$, and they control what happens to the state $c$ via **gates**. Each gate is a neural network with sigmoid activation (output in the range $[0,1]$), which can be thought of as bitwise mask when multiplied by the state vector. There are the following gates (from left to right on the picture above):\n",
+ "* **forget gate** takes hidden vector and determines, which components of the vector $c$ we need to forget, and which to pass through. \n",
+ "* **input gate** takes some information from the input and hidden vector, and inserts it into state.\n",
+ "* **output gate** transforms state via some linear layer with $\\tanh$ activation, then selects some of its components using hidden vector $h_i$ to produce new state $c_{i+1}$.\n",
+ "\n",
+ "Components of the state $c$ can be thought of as some flags that can be switched on and off. For example, when we encounter a name *Alice* in the sequence, we may want to assume that it refers to female character, and raise the flag in the state that we have female noun in the sentence. When we further encounter phrases *and Tom*, we will raise the flag that we have plural noun. Thus by manipulating state we can supposedly keep track of grammatical properties of sentence parts.\n",
+ "\n",
+ "> **Note**: A great resource for understanding internals of LSTM is this great article [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.\n",
+ "\n",
+ "While internal structure of LSTM cell may look complex, PyTorch hides this implementation inside `LSTMCell` class, and provides `LSTM` object to represent the whole LSTM layer. Thus, implementation of LSTM classifier will be pretty similar to the simple RNN which we have seen above:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class LSTMClassifier(torch.nn.Module):\n",
+ " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
+ " super().__init__()\n",
+ " self.hidden_dim = hidden_dim\n",
+ " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
+ " self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
+ " self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
+ " self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
+ "\n",
+ " def forward(self, x):\n",
+ " batch_size = x.size(0)\n",
+ " x = self.embedding(x)\n",
+ " x,(h,c) = self.rnn(x)\n",
+ " return self.fc(h[-1])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now let's train our network. Note that training LSTM is also quite slow, and you may not seem much raise in accuracy in the beginning of training. Also, you may need to play with `lr` learning rate parameter to find the learning rate that results in reasonable training speed, and yet does not cause "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "3200: acc=0.259375\n",
+ "6400: acc=0.25859375\n",
+ "9600: acc=0.26177083333333334\n",
+ "12800: acc=0.2784375\n",
+ "16000: acc=0.313\n",
+ "19200: acc=0.3528645833333333\n",
+ "22400: acc=0.3965625\n",
+ "25600: acc=0.4385546875\n",
+ "28800: acc=0.4752777777777778\n",
+ "32000: acc=0.505375\n",
+ "35200: acc=0.5326704545454546\n",
+ "38400: acc=0.5557552083333334\n",
+ "41600: acc=0.5760817307692307\n",
+ "44800: acc=0.5954910714285714\n",
+ "48000: acc=0.6118333333333333\n",
+ "51200: acc=0.62681640625\n",
+ "54400: acc=0.6404779411764706\n",
+ "57600: acc=0.6520138888888889\n",
+ "60800: acc=0.662828947368421\n",
+ "64000: acc=0.673546875\n",
+ "67200: acc=0.6831547619047619\n",
+ "70400: acc=0.6917897727272727\n",
+ "73600: acc=0.6997146739130434\n",
+ "76800: acc=0.707109375\n",
+ "80000: acc=0.714075\n",
+ "83200: acc=0.7209134615384616\n",
+ "86400: acc=0.727037037037037\n",
+ "89600: acc=0.7326674107142858\n",
+ "92800: acc=0.7379633620689655\n",
+ "96000: acc=0.7433645833333333\n",
+ "99200: acc=0.7479032258064516\n",
+ "102400: acc=0.752119140625\n",
+ "105600: acc=0.7562405303030303\n",
+ "108800: acc=0.76015625\n",
+ "112000: acc=0.7641339285714286\n",
+ "115200: acc=0.7677777777777778\n",
+ "118400: acc=0.7711233108108108\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "(0.03487814127604167, 0.7728)"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = LSTMClassifier(vocab_size,64,32,len(classes)).to(device)\n",
+ "train_epoch(net,train_loader, lr=0.001)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Packed sequences\n",
+ "\n",
+ "In our example, we had to pad all sequences in the minibatch with zero vectors. While it results in some memory waste, with RNNs it is more critical that additional RNN cells are created for the padded input items, which take part in training, yet do not carry any important input information. It would be much better to train RNN only to the actual sequence size.\n",
+ "\n",
+ "To do that, a special format of padded sequence storage is introduced in PyTorch. Suppose we have input padded minibatch which looks like this:\n",
+ "```\n",
+ "[[1,2,3,4,5],\n",
+ " [6,7,8,0,0],\n",
+ " [9,0,0,0,0]]\n",
+ "```\n",
+ "Here 0 represents padded values, and the actual length vector of input sequences is `[5,3,1]`.\n",
+ "\n",
+ "In order to effectively train RNN with padded sequence, we want to begin training first group of RNN cells with large minibatch (`[1,6,9]`), but then end processing of third sequence, and continue training with shorted minibatches (`[2,7]`, `[3,8]`), and so on. Thus, packed sequence is represented as one vector - in our case `[1,6,9,2,7,3,8,4,5]`, and length vector (`[5,3,1]`), from which we can easily reconstruct the original padded minibatch.\n",
+ "\n",
+ "To produce packed sequence, we can use `torch.nn.utils.rnn.pack_padded_sequence` function. All recurrent layers, including RNN, LSTM and GRU, support packed sequences as input, and produce packed output, which can be decoded using `torch.nn.utils.rnn.pad_packed_sequence`.\n",
+ "\n",
+ "To be able to produce packed sequence, we need to pass length vector to the network, and thus we need a different function to prepare minibatches:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def pad_length(b):\n",
+ " # build vectorized sequence\n",
+ " v = [encode(x[1]) for x in b]\n",
+ " # compute max length of a sequence in this minibatch and length sequence itself\n",
+ " len_seq = list(map(len,v))\n",
+ " l = max(len_seq)\n",
+ " return ( # tuple of three tensors - labels, padded features, length sequence\n",
+ " torch.LongTensor([t[0]-1 for t in b]),\n",
+ " torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v]),\n",
+ " torch.tensor(len_seq)\n",
+ " )\n",
+ "\n",
+ "train_loader_len = torch.utils.data.DataLoader(train_dataset, batch_size=16, collate_fn=pad_length, shuffle=True)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Actual network would be very similar to `LSTMClassifier` above, but `forward` pass will receive both padded minibatch and the vector of sequence lengths. After computing the embedding, we compute packed sequence, pass it to LSTM layer, and then unpack the result back.\n",
+ "\n",
+ "> **Note**: We actually do not use unpacked result `x`, because we use output from the hidden layers in the following computations. Thus, we can remove the unpacking altogether from this code. The reason we place it here is for you to be able to modify this code easily, in case you should need to use network output in further computations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "class LSTMPackClassifier(torch.nn.Module):\n",
+ " def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):\n",
+ " super().__init__()\n",
+ " self.hidden_dim = hidden_dim\n",
+ " self.embedding = torch.nn.Embedding(vocab_size, embed_dim)\n",
+ " self.embedding.weight.data = torch.randn_like(self.embedding.weight.data)-0.5\n",
+ " self.rnn = torch.nn.LSTM(embed_dim,hidden_dim,batch_first=True)\n",
+ " self.fc = torch.nn.Linear(hidden_dim, num_class)\n",
+ "\n",
+ " def forward(self, x, lengths):\n",
+ " batch_size = x.size(0)\n",
+ " x = self.embedding(x)\n",
+ " pad_x = torch.nn.utils.rnn.pack_padded_sequence(x,lengths,batch_first=True,enforce_sorted=False)\n",
+ " pad_x,(h,c) = self.rnn(pad_x)\n",
+ " x, _ = torch.nn.utils.rnn.pad_packed_sequence(pad_x,batch_first=True)\n",
+ " return self.fc(h[-1])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now let's do the training:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "3200: acc=0.285625\n",
+ "6400: acc=0.33359375\n",
+ "9600: acc=0.3876041666666667\n",
+ "12800: acc=0.44078125\n",
+ "16000: acc=0.4825\n",
+ "19200: acc=0.5235416666666667\n",
+ "22400: acc=0.5559821428571429\n",
+ "25600: acc=0.58609375\n",
+ "28800: acc=0.6116666666666667\n",
+ "32000: acc=0.63340625\n",
+ "35200: acc=0.6525284090909091\n",
+ "38400: acc=0.668515625\n",
+ "41600: acc=0.6822596153846154\n",
+ "44800: acc=0.6948214285714286\n",
+ "48000: acc=0.7052708333333333\n",
+ "51200: acc=0.71521484375\n",
+ "54400: acc=0.7239889705882353\n",
+ "57600: acc=0.7315277777777778\n",
+ "60800: acc=0.7388486842105263\n",
+ "64000: acc=0.74571875\n",
+ "67200: acc=0.7518303571428572\n",
+ "70400: acc=0.7576988636363636\n",
+ "73600: acc=0.7628940217391305\n",
+ "76800: acc=0.7681510416666667\n",
+ "80000: acc=0.7728125\n",
+ "83200: acc=0.7772235576923077\n",
+ "86400: acc=0.7815393518518519\n",
+ "89600: acc=0.7857700892857142\n",
+ "92800: acc=0.7895043103448276\n",
+ "96000: acc=0.7930520833333333\n",
+ "99200: acc=0.7959072580645161\n",
+ "102400: acc=0.798994140625\n",
+ "105600: acc=0.802064393939394\n",
+ "108800: acc=0.8051378676470589\n",
+ "112000: acc=0.8077857142857143\n",
+ "115200: acc=0.8104600694444445\n",
+ "118400: acc=0.8128293918918919\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "(0.029785829671223958, 0.8138166666666666)"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "net = LSTMPackClassifier(vocab_size,64,32,len(classes)).to(device)\n",
+ "train_epoch_emb(net,train_loader_len, lr=0.001,use_pack_sequence=True)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "> **Note:** You may have noticed the parameter `use_pack_sequence` that we pass to the training function. Currently, `pack_padded_sequence` function requires length sequence tensor to be on CPU device, and thus training function needs to avoid moving the length sequence data to GPU when training. You can look into implementation of `train_emb` function in the [`torchnlp.py`](torchnlp.py) file."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Bidirectional and multilayer RNNs\n",
+ "\n",
+ "In our examples, all recurrent networks operated in one direction, from beginning of a sequence to the end. It looks natural, because it resembles the way we read and listen to speech. However, since in many practical cases we have random access to the input sequence, it might make sense to run recurrent computation in both directions. Such networks are call **bidirectional** RNNs, and they can be created by passing `bidirectional=True` parameter to RNN/LSTM/GRU constructor.\n",
+ "\n",
+ "When dealing with bidirectional network, we would need two hidden state vectors, one for each direction. PyTorch encodes those vectors as one vector of twice larger size, which is quite convenient, because you would normally pass the resulting hidden state to fully-connected linear layer, and you would just need to take this increase in size into account when creating the layer.\n",
+ "\n",
+ "Recurrent network, one-directional or bidirectional, captures certain patterns within a sequence, and can store them into state vector or pass into output. As with convolutional networks, we can build another recurrent layer on top of the first one to capture higher level patterns, build from low-level patterns extracted by the first layer. This leads us to the notion of **multi-layer RNN**, which consists of two or more recurrent networks, where output of the previous layer is passed to the next layer as input.\n",
+ "\n",
+ "\n",
+ "\n",
+ "*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López*\n",
+ "\n",
+ "PyTorch makes constructing such networks an easy task, because you just need to pass `num_layers` parameter to RNN/LSTM/GRU constructor to build several layers of recurrence automatically. This would also mean that the size of hidden/state vector would increase proportionally, and you would need to take this into account when handling the output of recurrent layers."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## RNNs for other tasks\n",
+ "\n",
+ "In this unit, we have seen that RNNs can be used for sequence classification, but in fact, they can handle many more tasks, such as text generation, machine translation, and more. We will consider those tasks in the next unit."
+ ]
+ }
+ ],
+ "metadata": {
+ "interpreter": {
+ "hash": "0cb620c6d4b9f7a635928804c26cf22403d89d98d79684e4529119355ee6d5a5"
+ },
+ "kernelspec": {
+ "display_name": "py37_pytorch",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/5-NLP/16-RNN/RNNTF.ipynb b/5-NLP/16-RNN/RNNTF.ipynb
new file mode 100644
index 0000000..c461ced
--- /dev/null
+++ b/5-NLP/16-RNN/RNNTF.ipynb
@@ -0,0 +1,443 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Recurrent neural networks\n",
+ "\n",
+ "In the previous module, we covered rich semantic representations of text. The architecture we've been using captures the aggregated meaning of words in a sentence, but it does not take into account the **order** of the words, because the aggregation operation that follows the embeddings removes this information from the original text. Because these models are unable to represent word ordering, they cannot solve more complex or ambiguous tasks such as text generation or question answering.\n",
+ "\n",
+ "To capture the meaning of a text sequence, we'll use a neural network architecture called **recurrent neural network**, or RNN. When using an RNN, we pass our sentence through the network one token at a time, and the network produces some **state**, which we then pass to the network again with the next token.\n",
+ "\n",
+ "\n",
+ "\n",
+ "Given the input sequence of tokens $X_0,\\dots,X_n$, the RNN creates a sequence of neural network blocks, and trains this sequence end-to-end using backpropagation. Each network block takes a pair $(X_i,S_i)$ as an input, and produces $S_{i+1}$ as a result. The final state $S_n$ or output $Y_n$ goes into a linear classifier to produce the result. All network blocks share the same weights, and are trained end-to-end using one backpropagation pass.\n",
+ "\n",
+ "> The figure above shows recurrent neural network in the unrolled form (on the left), and in more compact recurrent representation (on the right). It is important to realize that all RNN Cells have the same **shareable weights**.\n",
+ "\n",
+ "Because state vectors $S_0,\\dots,S_n$ are passed through the network, the RNN is able to learn sequential dependencies between words. For example, when the word *not* appears somewhere in the sequence, it can learn to negate certain elements within the state vector.\n",
+ "\n",
+ "Inside, each RNN cell contains two weight matrices: $W_H$ and $W_I$, and bias $b$. At each RNN step, given input $X_i$ and input state $S_i$, output state is calculated as $S_{i+1} = f(W_H\\times S_i + W_I\\times X_i+b)$, where $f$ is an activation function (often $\\tanh$).\n",
+ "\n",
+ "> For problems like text generation (that we will cover in the next unit) or machine translation we also want to get some output value at each RNN step. In this case, there is also another matrix $W_O$, and output is caluclated as $Y_i=f(W_O\\times S_i+b_O)$.\n",
+ "\n",
+ "Let's see how recurrent neural networks can help us classify our news dataset.\n",
+ "\n",
+ "> For the sandbox environment, we need to run the following cell to make sure the required library is installed, and data is prefetched. If you are running locally, you can skip the following cell."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import sys\n",
+ "!{sys.executable} -m pip install --quiet tensorflow_datasets==4.4.0\n",
+ "!cd ~ && wget -q -O - https://mslearntensorflowlp.blob.core.windows.net/data/tfds-ag-news.tgz | tar xz"
+ ],
+ "outputs": [],
+ "execution_count": 1,
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "import tensorflow as tf\n",
+ "from tensorflow import keras\n",
+ "import tensorflow_datasets as tfds\n",
+ "import numpy as np\n",
+ "\n",
+ "# We are going to be training pretty large models. In order not to face errors, we need\n",
+ "# to set tensorflow option to grow GPU memory allocation when required\n",
+ "physical_devices = tf.config.list_physical_devices('GPU') \n",
+ "if len(physical_devices)>0:\n",
+ " tf.config.experimental.set_memory_growth(physical_devices[0], True)\n",
+ "\n",
+ "ds_train, ds_test = tfds.load('ag_news_subset').values()"
+ ],
+ "outputs": [],
+ "execution_count": 2,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "When training large models, GPU memory allocation may become a problem. We also may need to experiment with different minibatch sizes, so that the data fits into our GPU memory, yet the training is fast enough. If you are running this code on your own GPU machine, you may experiment with adjusting minibatch size to speed up training.\r\n",
+ "\r\n",
+ "> **Note**: Certain versions of NVidia drivers are known not to release the memory after training the model. We are running several examples in this notebooks, and it might cause memory to be exhausted in certain setups, especially if you are doing your own experiments as part of the same notebook. If you encounter some weird errors when starting to train the model, you may want to restart notebook kernel."
+ ],
+ "metadata": {
+ "nteract": {
+ "transient": {
+ "deleting": false
+ }
+ }
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "batch_size = 16\r\n",
+ "embed_size = 64"
+ ],
+ "outputs": [],
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": true,
+ "jupyter": {
+ "source_hidden": false,
+ "outputs_hidden": false
+ },
+ "nteract": {
+ "transient": {
+ "deleting": false
+ }
+ }
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Simple RNN classifier\n",
+ "\n",
+ "In the case of a simple RNN, each recurrent unit is a simple linear network, which takes in an input vector and state vector, and produces a new state vector. In Keras, this can be represented by the `SimpleRNN` layer.\n",
+ "\n",
+ "While we can pass one-hot encoded tokens to the RNN layer directly, this is not a good idea because of their high dimensionality. Therefore, we will use an embedding layer to lower the dimensionality of word vectors, followed by an RNN layer, and finally a `Dense` classifier.\n",
+ "\n",
+ "> **Note**: In cases where the dimensionality isn't so high, for example when using character-level tokenization, it might make sense to pass one-hot encoded tokens directly into the RNN cell."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "vocab_size = 20000\n",
+ "\n",
+ "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(\n",
+ " max_tokens=vocab_size,\n",
+ " input_shape=(1,))\n",
+ "\n",
+ "model = keras.models.Sequential([\n",
+ " vectorizer,\n",
+ " keras.layers.Embedding(vocab_size, embed_size),\n",
+ " keras.layers.SimpleRNN(16),\n",
+ " keras.layers.Dense(4,activation='softmax')\n",
+ "])\n",
+ "\n",
+ "model.summary()"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Model: \"sequential\"\n",
+ "_________________________________________________________________\n",
+ "Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ "text_vectorization (TextVect (None, None) 0 \n",
+ "_________________________________________________________________\n",
+ "embedding (Embedding) (None, None, 64) 1280000 \n",
+ "_________________________________________________________________\n",
+ "simple_rnn (SimpleRNN) (None, 16) 1296 \n",
+ "_________________________________________________________________\n",
+ "dense (Dense) (None, 4) 68 \n",
+ "=================================================================\n",
+ "Total params: 1,281,364\n",
+ "Trainable params: 1,281,364\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "execution_count": 4,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "> **Note:** We use an untrained embedding layer here for simplicity, but for better results we can use a pretrained embedding layer using Word2Vec, as described in the previous unit. It would be a good exercise for you to adapt this code to work with pretrained embeddings.\n",
+ "\n",
+ "Now let's train our RNN. RNNs in general are quite difficult to train, because once the RNN cells are unrolled along the sequence length, the resulting number of layers involved in backpropagation is quite large. Thus we need to select a smaller learning rate, and train the network on a larger dataset to produce good results. This can take quite a long time, so using a GPU is preferred.\n",
+ "\n",
+ "To speed things up, we will only train the RNN model on news titles, omitting the description. You can try training with description and see if you can get the model to train."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def extract_title(x):\n",
+ " return x['title']\n",
+ "\n",
+ "def tupelize_title(x):\n",
+ " return (extract_title(x),x['label'])\n",
+ "\n",
+ "print('Training vectorizer')\n",
+ "vectorizer.adapt(ds_train.take(2000).map(extract_title))"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Training vectorizer\n"
+ ]
+ }
+ ],
+ "execution_count": 5,
+ "metadata": {
+ "scrolled": true
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
+ "model.fit(ds_train.map(tupelize_title).batch(batch_size),validation_data=ds_test.map(tupelize_title).batch(batch_size))"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "7500/7500 [==============================] - 82s 11ms/step - loss: 0.6629 - acc: 0.7623 - val_loss: 0.5559 - val_acc: 0.7995\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
+ ]
+ },
+ {
+ "output_type": "execute_result",
+ "execution_count": 6,
+ "data": {
+ "text/plain": "<tensorflow.python.keras.callbacks.History at 0x7f3e0030d350>"
+ },
+ "metadata": {}
+ }
+ ],
+ "execution_count": 6,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "> **Note** that accuracy is likely to be lower here, because we are training only on news titles."
+ ],
+ "metadata": {
+ "nteract": {
+ "transient": {
+ "deleting": false
+ }
+ }
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Revisiting variable sequences \n",
+ "\n",
+ "Remember that the `TextVectorization` layer will automatically pad sequences of variable length in a minibatch with pad tokens. It turns out that those tokens also take part in training, and they can complicate convergence of the model.\n",
+ "\n",
+ "There are several approaches we can take to minimize the amount of padding. One of them is to reorder the dataset by sequence length and group all sequences by size. This can be done using the `tf.data.experimental.bucket_by_sequence_length` function (see [documentation](https://www.tensorflow.org/api_docs/python/tf/data/experimental/bucket_by_sequence_length)). \n",
+ "\n",
+ "Another approach is to use **masking**. In Keras, some layers support additional input that shows which tokens should be taken into account when training. To incorporate masking into our model, we can either include a separate `Masking` layer ([docs](https://keras.io/api/layers/core_layers/masking/)), or we can specify the `mask_zero=True` parameter of our `Embedding` layer.\n",
+ "\n",
+ "> **Note**: This training will take around 5 minutes to complete one epoch on the whole dataset. Feel free to interrupt training at any time if you run out of patience. What you can also do is limit the amount of data used for training, by adding `.take(...)` clause after `ds_train` and `ds_test` datasets."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "def extract_text(x):\n",
+ " return x['title']+' '+x['description']\n",
+ "\n",
+ "def tupelize(x):\n",
+ " return (extract_text(x),x['label'])\n",
+ "\n",
+ "model = keras.models.Sequential([\n",
+ " vectorizer,\n",
+ " keras.layers.Embedding(vocab_size,embed_size,mask_zero=True),\n",
+ " keras.layers.SimpleRNN(16),\n",
+ " keras.layers.Dense(4,activation='softmax')\n",
+ "])\n",
+ "\n",
+ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
+ "model.fit(ds_train.map(tupelize).batch(batch_size),validation_data=ds_test.map(tupelize).batch(batch_size))"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "7500/7500 [==============================] - 371s 49ms/step - loss: 0.5401 - acc: 0.8079 - val_loss: 0.3780 - val_acc: 0.8822\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
+ ]
+ },
+ {
+ "output_type": "execute_result",
+ "execution_count": 7,
+ "data": {
+ "text/plain": "<tensorflow.python.keras.callbacks.History at 0x7f3dec118850>"
+ },
+ "metadata": {}
+ }
+ ],
+ "execution_count": 7,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Now that we're using masking, we can train the model on the whole dataset of titles and descriptions.\r\n",
+ "\r\n",
+ "> **Note**: Have you noticed that we have been using vectorizer trained on the news titles, and not the whole body of the article? Potentially, this can cause some of the the tokens to be ignored, so it is better to re-train the vectorizer. However, it might only have very small effect, so we will stick to the previous pre-trained vectorizer for the sake of simplicity."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## LSTM: Long short-term memory\n",
+ "\n",
+ "One of the main problems of RNNs is **vanishing gradients**. RNNs can be pretty long, and may have a hard time propagating the gradients all the way back to the first layer of the network during backpropagation. When this happens, the network cannot learn relationships between distant tokens. One way to avoid this problem is to introduce **explicit state management** by using **gates**. The two most common architectures that introduce gates are **long short-term memory** (LSTM) and **gated relay unit** (GRU). We'll cover LSTMs here.\n",
+ "\n",
+ "\n",
+ "\n",
+ "An LSTM network is organized in a manner similar to an RNN, but there are two states that are passed from layer to layer: the actual state $c$, and the hidden vector $h$. At each unit, the hidden vector $h_{t-1}$ is combined with input $x_t$, and together they control what happens to the state $c_t$ and output $h_{t}$ through **gates**. Each gate has sigmoid activation (output in the range $[0,1]$), which can be thought of as a bitwise mask when multiplied by the state vector. LSTMs have the following gates (from left to right on the picture above):\n",
+ "* **forget gate** which determines which components of the vector $c_{t-1}$ we need to forget, and which to pass through. \n",
+ "* **input gate** which determines how much information from the input vector and previous hidden vector should be incorporated into the state vector.\n",
+ "* **output gate** which takes the new state vector and decides which of its components will be used to produce the new hidden vector $h_t$.\n",
+ "\n",
+ "The components of the state $c$ can be thought of as flags that can be switched on and off. For example, when we encounter the name *Alice* in the sequence, we guess that it refers to a woman, and raise the flag in the state that says we have a female noun in the sentence. When we further encounter the words *and Tom*, we will raise the flag that says we have a plural noun. Thus by manipulating state we can keep track of the grammatical properties of the sentence.\n",
+ "\n",
+ "> **Note**: Here's a great resource for understanding the internals of LSTMs: [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Christopher Olah.\n",
+ "\n",
+ "While the internal structure of an LSTM cell may look complex, Keras hides this implementation inside the `LSTM` layer, so the only thing we need to do in the example above is to replace the recurrent layer:"
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = keras.models.Sequential([\n",
+ " vectorizer,\n",
+ " keras.layers.Embedding(vocab_size, embed_size),\n",
+ " keras.layers.LSTM(8),\n",
+ " keras.layers.Dense(4,activation='softmax')\n",
+ "])\n",
+ "\n",
+ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
+ "model.fit(ds_train.map(tupelize).batch(8),validation_data=ds_test.map(tupelize).batch(8))"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "15000/15000 [==============================] - 188s 13ms/step - loss: 0.5692 - acc: 0.7916 - val_loss: 0.3441 - val_acc: 0.8870\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\n"
+ ]
+ },
+ {
+ "output_type": "execute_result",
+ "execution_count": 8,
+ "data": {
+ "text/plain": "<tensorflow.python.keras.callbacks.History at 0x7f3d6af5c350>"
+ },
+ "metadata": {}
+ }
+ ],
+ "execution_count": 8,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "> **Note** that training LSTMs is also quite slow, and you may not seem much increase in accuracy in the beginning of training. You may need to continue training for some time to achieve good accuracy."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Bidirectional and multilayer RNNs\n",
+ "\n",
+ "In our examples so far, the recurrent networks operate from the beginning of a sequence until the end. This feels natural to us because it follows the same direction in which we read or listen to speech. However, for scenarios which require random access of the input sequence, it makes more sense to run the recurrent computation in both directions. RNNs that allow computations in both directions are called **bidirectional** RNNs, and they can be created by wrapping the recurrent layer with a special `Bidirectonal` layer.\n",
+ "\n",
+ "> **Note**: The `Bidirectional` layer makes two copies of the layer within it, and sets the `go_backwards` property of one of those copies to `True`, making it go in the opposite direction along the sequence.\n",
+ "\n",
+ "Recurrent networks, unidirectional or bidirectional, capture patterns within a sequence, and store them into state vectors or return them as output. As with convolutional networks, we can build another recurrent layer following the first one to capture higher level patterns, built from lower level patterns extracted by the first layer. This leads us to the notion of a **multi-layer RNN**, which consists of two or more recurrent networks, where the output of the previous layer is passed to the next layer as input.\n",
+ "\n",
+ "\n",
+ "\n",
+ "*Picture from [this wonderful post](https://towardsdatascience.com/from-a-lstm-cell-to-a-multilayer-lstm-network-with-pytorch-2899eb5696f3) by Fernando López.*\n",
+ "\n",
+ "Keras makes constructing these networks an easy task, because you just need to add more recurrent layers to the model. For all layers except the last one, we need to specify `return_sequences=True` parameter, because we need the layer to return all intermediate states, and not just the final state of the recurrent computation.\n",
+ "\n",
+ "Let's build a two-layer bidirectional LSTM for our classification problem.\n",
+ "\n",
+ "> **Note** this code again takes quite a long time to complete, but it gives us highest accuracy we have seen so far. So maybe it is worth waiting and seeing the result."
+ ],
+ "metadata": {}
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "model = keras.models.Sequential([\n",
+ " vectorizer,\n",
+ " keras.layers.Embedding(vocab_size, 128, mask_zero=True),\n",
+ " keras.layers.Bidirectional(keras.layers.LSTM(64,return_sequences=True)),\n",
+ " keras.layers.Bidirectional(keras.layers.LSTM(64)), \n",
+ " keras.layers.Dense(4,activation='softmax')\n",
+ "])\n",
+ "\n",
+ "model.compile(loss='sparse_categorical_crossentropy',metrics=['acc'], optimizer='adam')\n",
+ "model.fit(ds_train.map(tupelize).batch(batch_size),\n",
+ " validation_data=ds_test.map(tupelize).batch(batch_size))"
+ ],
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "5044/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r5045/7500 [===================>..........] - ETA: 2:33 - loss: 0.3709 - acc: 0.8706"
+ ]
+ }
+ ],
+ "execution_count": 9,
+ "metadata": {}
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## RNNs for other tasks\n",
+ "\n",
+ "Up until now, we've focused on using RNNs to classify sequences of text. But they can handle many more tasks, such as text generation and machine translation — we'll consider those tasks in the next unit."
+ ],
+ "metadata": {}
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "name": "conda-env-py37_tensorflow-py",
+ "language": "python",
+ "display_name": "py37_tensorflow"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.7.9",
+ "mimetype": "text/x-python",
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "pygments_lexer": "ipython3",
+ "nbconvert_exporter": "python",
+ "file_extension": ".py"
+ },
+ "kernel_info": {
+ "name": "conda-env-py37_tensorflow-py"
+ },
+ "nteract": {
+ "version": "nteract-front-end@1.0.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
\ No newline at end of file
diff --git a/5-NLP/16-RNN/images/long-short-term-memory-cell.svg b/5-NLP/16-RNN/images/long-short-term-memory-cell.svg
new file mode 100644
index 0000000..7b66a2c
--- /dev/null
+++ b/5-NLP/16-RNN/images/long-short-term-memory-cell.svg
@@ -0,0 +1,1334 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!-- Created with Inkscape (http://www.inkscape.org/) -->
+
+<svg
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:cc="http://creativecommons.org/ns#"
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns="http://www.w3.org/2000/svg"
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+ width="190mm"
+ height="130mm"
+ viewBox="0 0 190 130"
+ version="1.1"
+ id="svg8"
+ sodipodi:docname="lstm-cell.svg"
+ inkscape:version="0.92.3 (2405546, 2018-03-11)"
+ inkscape:export-filename="/home/users_home/Documents/GIT/FINAL_DL/Linear-Attention-Recurrent-Neural-Network/inkscape_drawings/png_exported/lstm-cell.png"
+ inkscape:export-xdpi="299.91537"
+ inkscape:export-ydpi="299.91537">
+ <title
+ id="title1438">LSTM Cell</title>
+ <defs
+ id="defs2">
+ <inkscape:perspective
+ sodipodi:type="inkscape:persp3d"
+ inkscape:vp_x="-405.59802 : 195.57742 : 1"
+ inkscape:vp_y="0 : 999.99994 : 0"
+ inkscape:vp_z="303.85994 : 427.17668 : 1"
+ inkscape:persp3d-origin="105 : 98.999994 : 1"
+ id="perspective4559" />
+ <inkscape:perspective
+ sodipodi:type="inkscape:persp3d"
+ inkscape:vp_x="19.240475 : 198.74312 : 1"
+ inkscape:vp_y="0 : 129.29024 : 0"
+ inkscape:vp_z="110.96647 : 228.68665 : 1"
+ inkscape:persp3d-origin="85.255818 : 186.2566 : 1"
+ id="perspective4559-3" />
+ <filter
+ style="color-interpolation-filters:sRGB"
+ inkscape:label="Blur"
+ id="filter6288">
+ <feGaussianBlur
+ stdDeviation="67.2 10"
+ result="blur"
+ id="feGaussianBlur6286" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter6777"
+ x="-0.02178747"
+ width="1.0435749"
+ y="-0.04814931"
+ height="1.0962986">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.90655222"
+ id="feGaussianBlur6779" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter8042"
+ x="-0.42381817"
+ width="1.8476363"
+ y="-0.71723074"
+ height="2.4344616">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.25697656"
+ id="feGaussianBlur8044" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter10264"
+ x="-0.0079611773"
+ width="1.0159224"
+ y="-0.024356319"
+ height="1.0487126">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.20177668"
+ id="feGaussianBlur10266" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter10272"
+ x="-0.009238895"
+ width="1.0184778"
+ y="-0.017114902"
+ height="1.0342298">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.28481399"
+ id="feGaussianBlur10274" />
+ </filter>
+ <clipPath
+ clipPathUnits="userSpaceOnUse"
+ id="clipPath10338">
+ <rect
+ style="opacity:1;fill:#1d635e;fill-opacity:0.06880733;stroke:#ffffff;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.96788988"
+ id="rect10340"
+ width="46.299076"
+ height="115.36229"
+ x="270.74799"
+ y="29.903158" />
+ </clipPath>
+ <clipPath
+ clipPathUnits="userSpaceOnUse"
+ id="clipPath10376">
+ <rect
+ style="opacity:1;fill:#1d635e;fill-opacity:0.06880733;stroke:#ffffff;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.96788988"
+ id="rect10378"
+ width="25.940634"
+ height="17.601774"
+ x="3.1655979"
+ y="71.54734"
+ transform="scale(-1,1)" />
+ </clipPath>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter8042-1"
+ x="-0.42381817"
+ width="1.8476363"
+ y="-0.71723074"
+ height="2.4344616">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.25697656"
+ id="feGaussianBlur8044-9" />
+ </filter>
+ <clipPath
+ clipPathUnits="userSpaceOnUse"
+ id="clipPath10679">
+ <rect
+ style="opacity:1;fill:#808080;fill-opacity:0.06880733;stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.96788988"
+ id="rect10681"
+ width="25.406197"
+ height="17.531759"
+ x="3.4345946"
+ y="71.547333"
+ transform="scale(-1,1)" />
+ </clipPath>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter12229"
+ x="-0.12156857"
+ width="1.2431371"
+ y="-0.12156857"
+ height="1.2431371">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.43953438"
+ id="feGaussianBlur12231" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter12233"
+ x="-0.052935258"
+ width="1.1058705"
+ y="-0.10762672"
+ height="1.2152534">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.43953438"
+ id="feGaussianBlur12235" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter12541"
+ x="-0.0144"
+ width="1.0288"
+ y="-0.0144"
+ height="1.0288">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.05206358"
+ id="feGaussianBlur12543" />
+ </filter>
+ <filter
+ inkscape:collect="always"
+ style="color-interpolation-filters:sRGB"
+ id="filter12787"
+ x="-0.0082712928"
+ width="1.0165426"
+ y="-0.055590123"
+ height="1.1111802">
+ <feGaussianBlur
+ inkscape:collect="always"
+ stdDeviation="0.67542246"
+ id="feGaussianBlur12789" />
+ </filter>
+ </defs>
+ <sodipodi:namedview
+ id="base"
+ pagecolor="#ffffff"
+ bordercolor="#666666"
+ borderopacity="1.0"
+ inkscape:pageopacity="0.0"
+ inkscape:pageshadow="2"
+ inkscape:zoom="1"
+ inkscape:cx="433.45381"
+ inkscape:cy="188.63838"
+ inkscape:document-units="mm"
+ inkscape:current-layer="layer1"
+ showgrid="false"
+ inkscape:window-width="1920"
+ inkscape:window-height="1023"
+ inkscape:window-x="1920"
+ inkscape:window-y="0"
+ inkscape:window-maximized="1" />
+ <metadata
+ id="metadata5">
+ <rdf:RDF>
+ <cc:Work
+ rdf:about="">
+ <dc:format>image/svg+xml</dc:format>
+ <dc:type
+ rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
+ <dc:title>LSTM Cell</dc:title>
+ <cc:license
+ rdf:resource="http://creativecommons.org/licenses/by/4.0/" />
+ <dc:creator>
+ <cc:Agent>
+ <dc:title>Guillaume Chevalier</dc:title>
+ </cc:Agent>
+ </dc:creator>
+ <dc:date>13 May 2017</dc:date>
+ </cc:Work>
+ <cc:License
+ rdf:about="http://creativecommons.org/licenses/by/4.0/">
+ <cc:permits
+ rdf:resource="http://creativecommons.org/ns#Reproduction" />
+ <cc:permits
+ rdf:resource="http://creativecommons.org/ns#Distribution" />
+ <cc:requires
+ rdf:resource="http://creativecommons.org/ns#Notice" />
+ <cc:requires
+ rdf:resource="http://creativecommons.org/ns#Attribution" />
+ <cc:permits
+ rdf:resource="http://creativecommons.org/ns#DerivativeWorks" />
+ </cc:License>
+ </rdf:RDF>
+ </metadata>
+ <g
+ inkscape:label="Layer 1"
+ inkscape:groupmode="layer"
+ id="layer1"
+ transform="translate(0,-167)">
+ <g
+ id="g1436"
+ transform="matrix(0.89305455,0,0,1.0058054,1.0658078,-1.5875497)">
+ <g
+ id="g1431">
+ <rect
+ ry="2.9579358"
+ y="261.14444"
+ x="6.6429443"
+ height="29.160107"
+ width="195.98071"
+ id="rect12545"
+ style="fill:#f2f2f2;fill-opacity:1;stroke-width:0.23194653;filter:url(#filter12787)"
+ transform="matrix(0.95880034,0,0,0.95880034,4.3481892,9.0950066)" />
+ </g>
+ <rect
+ ry="2.8360698"
+ y="259.48038"
+ x="10.717446"
+ height="27.958721"
+ width="187.90637"
+ id="rect5514-0"
+ style="fill:#f2f2f2;fill-opacity:1;stroke-width:0.22239041" />
+ </g>
+ <rect
+ style="opacity:1;fill:#ffffff;fill-opacity:0.98165134;stroke:#ffffff;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.96788988;filter:url(#filter8042-1)"
+ id="rect6284-0"
+ width="1.4552083"
+ height="0.85989583"
+ x="53.280468"
+ y="68.763802"
+ transform="matrix(0,1.1584917,-1.1584917,0,127.6196,134.4837)" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ x="186.17693"
+ y="176.74184"
+ id="text6336"><tspan
+ sodipodi:role="line"
+ id="tspan6334"
+ x="186.17693"
+ y="187.58969"
+ style="stroke-width:0.30651757" /></text>
+ <g
+ id="g6473"
+ transform="matrix(1.1584917,0,0,1.1584917,14.071849,136.59698)" />
+ <rect
+ y="46.896858"
+ x="59.559067"
+ height="45.18705"
+ width="99.861305"
+ id="rect6623"
+ style="fill:#4ec2a7;fill-opacity:1;stroke-width:0.20610677;filter:url(#filter6777)"
+ ry="4.5836725"
+ transform="matrix(1.0031719,0,0,1.2674399,-15.227239,128.62865)" />
+ <rect
+ style="fill:#4ec2a7;fill-opacity:1;stroke-width:0.23240399"
+ id="rect5514"
+ width="100.17805"
+ height="57.271866"
+ x="44.520744"
+ y="188.06775"
+ ry="5.8095293" />
+ <g
+ id="g11423"
+ transform="matrix(1.1584917,0,0,1.1584917,-81.653426,135.33973)">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path11415"
+ d="m 101.69547,53.23758 c 1.67044,0.0668 16.88275,0 16.88275,0"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ id="g11421">
+ <path
+ sodipodi:nodetypes="ccc"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.58225,53.22346 -1.14593,-1.14593 z"
+ id="path11417"
+ inkscape:connector-curvature="0" />
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path11419"
+ d="m 117.58225,53.22346 -1.14593,1.14594 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ id="g7124"
+ style="filter:url(#filter10272)"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <g
+ id="g7090"
+ transform="translate(-6.7154443,17.341029)">
+ <rect
+ y="46.023808"
+ x="150.56232"
+ height="8.0319939"
+ width="12.904612"
+ id="rect7084"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.25856197"
+ ry="4.0159969" />
+ <text
+ id="text7088"
+ y="51.551083"
+ x="151.8725"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="51.551083"
+ x="151.8725"
+ id="tspan7086"
+ sodipodi:role="line">tanh</tspan></text>
+ </g>
+ <g
+ id="g7098"
+ transform="translate(-0.57877604,9.4479778)">
+ <circle
+ r="3.5907738"
+ cy="43.566963"
+ cx="86.934525"
+ id="circle7092"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332" />
+ <text
+ id="text7096"
+ y="44.898705"
+ x="85.632172"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="44.898705"
+ x="85.632172"
+ id="tspan7094"
+ sodipodi:role="line">x</tspan></text>
+ </g>
+ <g
+ id="g7106"
+ transform="translate(0,9.4479778)">
+ <circle
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332"
+ id="circle7100"
+ cx="121.17175"
+ cy="43.566963"
+ r="3.5907738" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="119.8694"
+ y="44.898705"
+ id="text7104"><tspan
+ sodipodi:role="line"
+ id="tspan7102"
+ x="119.8694"
+ y="44.898705"
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332">x</tspan></text>
+ </g>
+ <g
+ id="g7114"
+ transform="translate(63.364357,42.205518)">
+ <circle
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332"
+ id="circle7108"
+ cx="86.934525"
+ cy="43.566963"
+ r="3.5907738" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="85.632172"
+ y="44.898705"
+ id="text7112"><tspan
+ sodipodi:role="line"
+ id="tspan7110"
+ x="85.632172"
+ y="44.898705"
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332">x</tspan></text>
+ </g>
+ <g
+ transform="translate(34.253762,19.725045)"
+ id="g7122">
+ <circle
+ r="3.5907738"
+ cy="43.566963"
+ cx="86.934525"
+ id="circle7116"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332" />
+ <text
+ id="text7120"
+ y="44.898705"
+ x="85.632172"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="44.898705"
+ x="85.632172"
+ id="tspan7118"
+ sodipodi:role="line">x</tspan></text>
+ </g>
+ </g>
+ <g
+ id="g7039"
+ style="filter:url(#filter10264)"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <g
+ id="g7013">
+ <rect
+ y="69.578537"
+ x="97.360367"
+ height="8.0023499"
+ width="12.879682"
+ id="rect7007"
+ style="fill:#ff9955;stroke-width:0.22517994" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="100.35973"
+ y="76.502632"
+ id="text7011"><tspan
+ sodipodi:role="line"
+ id="tspan7009"
+ x="100.35973"
+ y="76.502632"
+ style="fill:#4d4d4d;stroke-width:0.26458332">σ</tspan></text>
+ </g>
+ <g
+ id="g7021"
+ transform="translate(-4.1691348,11.88013)">
+ <rect
+ y="69.578537"
+ x="132.10547"
+ height="8.0023499"
+ width="12.879682"
+ id="rect7015"
+ style="fill:#ff9955;stroke-width:0.22517994" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="135.10483"
+ y="76.502632"
+ id="text7019"><tspan
+ sodipodi:role="line"
+ id="tspan7017"
+ x="135.10483"
+ y="76.502632"
+ style="fill:#4d4d4d;stroke-width:0.26458332">σ</tspan></text>
+ </g>
+ <g
+ id="g7029">
+ <rect
+ style="fill:#ff9955;stroke-width:0.22517994"
+ id="rect7023"
+ width="12.879682"
+ height="8.0023499"
+ x="79.987823"
+ y="69.578537" />
+ <text
+ id="text7027"
+ y="76.502632"
+ x="82.98719"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="fill:#4d4d4d;stroke-width:0.26458332"
+ y="76.502632"
+ x="82.98719"
+ id="tspan7025"
+ sodipodi:role="line">σ</tspan></text>
+ </g>
+ <g
+ id="g7037">
+ <rect
+ style="fill:#ff9955;stroke-width:0.22517994"
+ id="rect7031"
+ width="12.879682"
+ height="8.0023499"
+ x="114.73292"
+ y="69.578537" />
+ <text
+ id="text7035"
+ y="75.166283"
+ x="116.39593"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="75.166283"
+ x="116.39593"
+ id="tspan7033"
+ sodipodi:role="line">tanh</tspan></text>
+ </g>
+ </g>
+ <g
+ transform="matrix(1.1584917,0,0,1.1584917,-81.704958,111.03958)"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6126">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path6118"
+ d="m 121.20363,100.28111 c 0.0668,-1.670436 0,-22.70473 0,-22.70473"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6124"
+ transform="translate(35.057279)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ id="path6120"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path6122"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ transform="matrix(-1.1584917,0,0,-1.1584917,273.16562,320.39542)"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6116">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path6108"
+ d="m 121.20363,89.600335 c 0.0668,-1.670433 0,-12.023955 0,-12.023955"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6114"
+ transform="translate(35.057279)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ id="path6110"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path6112"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ id="g6053"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ transform="matrix(-1.1584917,0,0,-1.1584917,273.16562,298.6098)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 123.40192,87.510461 c -2.4589,-0.01002 -2.19829,-10.082909 -2.19829,-10.082909"
+ id="path6045"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc" />
+ <g
+ transform="translate(35.057279)"
+ id="g6051"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1">
+ <path
+ inkscape:connector-curvature="0"
+ id="path6047"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ id="path6049"
+ inkscape:connector-curvature="0" />
+ </g>
+ </g>
+ <g
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,111.02632)"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6043">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path6035"
+ d="m 121.20363,87.448815 c 0.0668,-1.670433 0,-9.872435 0,-9.872435"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6041"
+ transform="translate(35.057279)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ id="path6037"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path6039"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.57924587;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="M 36.170736,234.63556 H 116.17405"
+ id="path5722"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g5802"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path5743"
+ d="m 82.538164,85.711397 c 0,0 3.54131,0.133636 3.608128,-1.536801 0.0668,-1.670433 0,-6.598216 0,-6.598216"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g5766">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ id="path5745"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path5762"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g5796"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 100.39753,85.711397 c 0,0 3.54132,0.133636 3.60814,-1.536801 0.0668,-1.670433 0,-6.598216 0,-6.598216"
+ id="path5741"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ transform="translate(17.859371)"
+ id="g5772">
+ <path
+ inkscape:connector-curvature="0"
+ id="path5768"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ id="path5770"
+ inkscape:connector-curvature="0" />
+ </g>
+ </g>
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g5790"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path5724"
+ d="m 117.59549,85.711397 c 0,0 3.54132,0.133636 3.60814,-1.536801 0.0668,-1.670433 0,-6.598216 0,-6.598216"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g5778"
+ transform="translate(35.057279)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ id="path5774"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path5776"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ transform="matrix(1.1584917,0,0,1.1584917,-7.5322135,166.16175)"
+ id="g5924-5"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path5724-1-1-7"
+ d="m 103.74646,58.938513 c 0,0 0.03,0.07902 13.88106,0.171766"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ sodipodi:nodetypes="ccc"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.62752,59.110339 -1.14593,-1.14593 z"
+ id="path5774-4-1-8"
+ inkscape:connector-curvature="0" />
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path5776-0-8-5"
+ d="m 117.62752,59.110339 -1.14593,1.14594 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ <g
+ id="g6186"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 89.559244,53.23758 c 1.670436,0.0668 29.018976,0 29.018976,0"
+ id="path6118-1"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc" />
+ <g
+ id="g6180">
+ <path
+ inkscape:connector-curvature="0"
+ id="path5774-4-1-6"
+ d="m 117.58225,53.22346 -1.14593,-1.14593 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ sodipodi:nodetypes="ccc" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.58225,53.22346 -1.14593,1.14594 z"
+ id="path5776-0-8-0"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ </g>
+ </g>
+ <g
+ id="g5924"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,134.11367)">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path5724-1-1"
+ d="m 103.62258,73.599644 c -0.17343,-9.264879 0.15387,-9.325178 14.00497,-9.232434"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ sodipodi:nodetypes="ccc"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.57735,64.36727 -1.14593,-1.14593 z"
+ id="path5774-4-1"
+ inkscape:connector-curvature="0" />
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path5776-0-8"
+ d="m 117.57735,64.36727 -1.14593,1.14594 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ <g
+ id="g5977"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,122.92152)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 121.20363,86.324336 c 0.0668,-1.670433 0,-8.747956 0,-8.747956"
+ id="path5969"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc" />
+ <g
+ transform="translate(35.057279)"
+ id="g5975"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1">
+ <path
+ inkscape:connector-curvature="0"
+ id="path5971"
+ d="m 86.14629,77.57638 1.145937,1.145937 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 86.14629,77.57638 -1.145937,1.145937 z"
+ id="path5973"
+ inkscape:connector-curvature="0" />
+ </g>
+ </g>
+ <g
+ id="g6839"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <g
+ id="g5833">
+ <rect
+ style="fill:#ff9955;stroke-width:0.22517994"
+ id="rect5541"
+ width="12.879682"
+ height="8.0023499"
+ x="97.360367"
+ y="69.578537" />
+ <text
+ id="text5577"
+ y="76.502632"
+ x="100.35973"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="fill:#4d4d4d;stroke-width:0.26458332"
+ y="76.502632"
+ x="100.35973"
+ id="tspan5575"
+ sodipodi:role="line">σ</tspan></text>
+ </g>
+ <g
+ transform="translate(-4.1691348,11.88013)"
+ id="g5843">
+ <rect
+ style="fill:#ff9955;stroke-width:0.22517994"
+ id="rect5545"
+ width="12.879682"
+ height="8.0023499"
+ x="132.10547"
+ y="69.578537" />
+ <text
+ id="text5585"
+ y="76.502632"
+ x="135.10483"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="fill:#4d4d4d;stroke-width:0.26458332"
+ y="76.502632"
+ x="135.10483"
+ id="tspan5583"
+ sodipodi:role="line">σ</tspan></text>
+ </g>
+ <g
+ id="g5828">
+ <rect
+ y="69.578537"
+ x="79.987823"
+ height="8.0023499"
+ width="12.879682"
+ id="rect5535"
+ style="fill:#ff9955;stroke-width:0.22517994" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="82.98719"
+ y="76.502632"
+ id="text5555"><tspan
+ sodipodi:role="line"
+ id="tspan5553"
+ x="82.98719"
+ y="76.502632"
+ style="fill:#4d4d4d;stroke-width:0.26458332">σ</tspan></text>
+ </g>
+ <g
+ id="g5838">
+ <rect
+ y="69.578537"
+ x="114.73292"
+ height="8.0023499"
+ width="12.879682"
+ id="rect5543"
+ style="fill:#ff9955;stroke-width:0.22517994" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="116.39593"
+ y="75.166283"
+ id="text5581"><tspan
+ sodipodi:role="line"
+ id="tspan5579"
+ x="116.39593"
+ y="75.166283"
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332">tanh</tspan></text>
+ </g>
+ </g>
+ <g
+ id="g6332"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <g
+ transform="translate(53.317378,26.652963)"
+ id="g6106"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1">
+ <g
+ id="g6215">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 100.00468,59.055443 c 0,0 3.77178,-0.03791 17.62284,0.05484"
+ id="path6100"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path6102"
+ d="m 117.62752,59.110339 -1.14593,-1.14593 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ sodipodi:nodetypes="ccc" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.80425,58.931539 -1.31336,1.323706 z"
+ id="path6104"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ </g>
+ <g
+ transform="translate(0,-32.279181)"
+ id="g6223">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path6217"
+ d="m 71.239797,59.055443 c 0,0 32.536663,-0.03791 46.387723,0.05484"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ sodipodi:nodetypes="ccc"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 117.62752,59.110339 -1.14593,-1.14593 z"
+ id="path6219"
+ inkscape:connector-curvature="0" />
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path6221"
+ d="m 117.80425,58.931539 -1.31336,1.323706 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ </g>
+ </g>
+ <g
+ id="g6313">
+ <rect
+ style="opacity:1;fill:#ffffff;fill-opacity:0.98165134;stroke:#ffffff;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:0.96788988;filter:url(#filter8042)"
+ id="rect6284"
+ width="1.4552083"
+ height="0.85989583"
+ x="164.43854"
+ y="53.021091" />
+ <g
+ id="g6246"
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ transform="translate(43.92084)">
+ <g
+ id="g6252"
+ transform="translate(0,-39.290625)">
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path6238"
+ d="m 117.59937,125.00205 c 0,0 3.57051,0.0262 3.60426,-3.88497 0.0668,-1.67044 0,-38.778197 0,-38.778197"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <g
+ style="stroke:#4d4d4d;stroke-width:0.5;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ id="g6244"
+ transform="translate(35.057279)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 85.972657,82.163183 1.31957,1.321637 z"
+ id="path6240"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path6242"
+ d="M 86.14629,82.338883 85.000353,83.48482 Z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ sodipodi:nodetypes="ccc" />
+ </g>
+ </g>
+ </g>
+ </g>
+ </g>
+ <text
+ id="text5581-6-3"
+ y="181.18173"
+ x="148.30687"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="181.18173"
+ x="148.30687"
+ id="tspan5579-7-2"
+ sodipodi:role="line">h<tspan
+ style="font-size:2.45214057px;stroke-width:0.30651757"
+ id="tspan10610">t</tspan></tspan></text>
+ <text
+ id="text5581-6-3-5"
+ y="198.77341"
+ x="157.97829"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="198.77341"
+ x="157.97829"
+ id="tspan5579-7-2-1"
+ sodipodi:role="line">c<tspan
+ style="font-size:2.45214057px;stroke-width:0.30651757"
+ id="tspan10606">t</tspan></tspan></text>
+ <text
+ id="text5581-6-3-3"
+ y="235.88847"
+ x="157.97829"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="235.88847"
+ x="157.97829"
+ id="tspan5579-7-2-2"
+ sodipodi:role="line">h<tspan
+ style="font-size:2.45214057px;stroke-width:0.30651757"
+ id="tspan10608">t</tspan></tspan></text>
+ <g
+ id="g1082"
+ transform="translate(-45.508335)">
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.5;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="M 57.617755,94.747095 C 57.397553,87.961141 61.279538,84.580018 69.003485,84.62599"
+ id="path5724-1-1-0"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="cc"
+ transform="matrix(1.1584917,0,0,1.1584917,14.071849,136.59698)" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ x="79.295021"
+ y="251.12712"
+ id="text5581-6-3-0"><tspan
+ sodipodi:role="line"
+ id="tspan5579-7-2-4"
+ x="79.295021"
+ y="251.12712"
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757">x<tspan
+ id="tspan10612"
+ style="font-size:2.45214057px;stroke-width:0.30651757">t</tspan></tspan></text>
+ </g>
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ x="28.036568"
+ y="198.30452"
+ id="text10321-2"><tspan
+ sodipodi:role="line"
+ id="tspan10319-8"
+ x="28.036568"
+ y="198.30452"
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757">c<tspan
+ style="font-size:2.45214057px;stroke-width:0.30651757"
+ id="tspan10323-6">t-1</tspan></tspan></text>
+ <g
+ id="g11344"
+ transform="matrix(1.1584917,0,0,1.1584917,-41.244631,135.33973)">
+ <g
+ id="g11310"
+ transform="translate(-6.7154443,17.341029)">
+ <rect
+ y="46.023808"
+ x="150.56232"
+ height="8.0319939"
+ width="12.904612"
+ id="rect11304"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.25856197"
+ ry="4.0159969" />
+ <text
+ id="text11308"
+ y="51.551083"
+ x="156.86485"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;text-align:center;letter-spacing:0px;word-spacing:0px;text-anchor:middle;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;text-align:center;text-anchor:middle;fill:#4d4d4d;stroke-width:0.26458332"
+ y="51.551083"
+ x="156.86485"
+ id="tspan11306"
+ sodipodi:role="line">tanh</tspan></text>
+ </g>
+ <g
+ id="g11318"
+ transform="translate(-0.57877604,9.4479778)">
+ <circle
+ r="3.5907738"
+ cy="43.566963"
+ cx="86.934525"
+ id="circle11312"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332" />
+ <text
+ id="text11316"
+ y="44.898705"
+ x="85.632172"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="44.898705"
+ x="85.632172"
+ id="tspan11314"
+ sodipodi:role="line">x</tspan></text>
+ </g>
+ <g
+ id="g11326"
+ transform="translate(0,9.4479778)">
+ <circle
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332"
+ id="circle11320"
+ cx="121.17175"
+ cy="43.566963"
+ r="3.5907738" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="119.3659"
+ y="45.019833"
+ id="text11324"><tspan
+ sodipodi:role="line"
+ id="tspan11322"
+ x="119.3659"
+ y="45.019833"
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332">+</tspan></text>
+ </g>
+ <g
+ id="g11334"
+ transform="translate(63.364357,42.205518)">
+ <circle
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332"
+ id="circle11328"
+ cx="86.934525"
+ cy="43.566963"
+ r="3.5907738" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ x="85.632172"
+ y="44.898705"
+ id="text11332"><tspan
+ sodipodi:role="line"
+ id="tspan11330"
+ x="85.632172"
+ y="44.898705"
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332">x</tspan></text>
+ </g>
+ <g
+ transform="translate(34.253762,19.725045)"
+ id="g11342">
+ <circle
+ r="3.5907738"
+ cy="43.566963"
+ cx="86.934525"
+ id="circle11336"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.26458332" />
+ <text
+ id="text11340"
+ y="44.898705"
+ x="85.632172"
+ style="font-style:normal;font-weight:normal;font-size:10.58333302px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.26458332"
+ xml:space="preserve"><tspan
+ style="font-size:4.58611107px;fill:#4d4d4d;stroke-width:0.26458332"
+ y="44.898705"
+ x="85.632172"
+ id="tspan11338"
+ sodipodi:role="line">x</tspan></text>
+ </g>
+ </g>
+ <text
+ id="text5581-6-3-0-0"
+ y="277.1235"
+ x="17.419685"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:12.25794315px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="277.1235"
+ x="17.419685"
+ sodipodi:role="line"
+ id="tspan11446">Legend:</tspan></text>
+ <rect
+ y="275.85126"
+ x="80.56076"
+ height="9.8013067"
+ width="19.927786"
+ id="rect5535-2"
+ style="fill:#ff9955;stroke-width:0.30998442;filter:url(#filter12233)"
+ transform="matrix(0.95880034,0,0,0.95880034,4.3481892,9.0950066)" />
+ <text
+ id="text5581-6-3-0-0-4"
+ y="268.51303"
+ x="84.162361"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="268.51303"
+ x="84.162361"
+ sodipodi:role="line"
+ id="tspan11446-8">Layer</tspan></text>
+ <circle
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.31968862;filter:url(#filter12229)"
+ id="circle11328-5"
+ cx="136.74632"
+ cy="280.75192"
+ r="4.3386316"
+ transform="matrix(0.95880034,0,0,0.95880034,-4.647644,9.0950066)" />
+ <text
+ id="text5581-6-3-0-0-9"
+ y="268.59476"
+ x="109.18328"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;letter-spacing:0px;word-spacing:0.08371533px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="268.59476"
+ x="109.18328"
+ sodipodi:role="line"
+ id="tspan11484">Pointwize op</tspan></text>
+ <g
+ id="g11554"
+ transform="matrix(0.95880034,0,0,0.95880034,-9.5377823,-2.9012372)">
+ <path
+ sodipodi:nodetypes="cc"
+ inkscape:connector-curvature="0"
+ id="path5722-6"
+ d="m 171.29205,295.17188 h 8.68051"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 171.29205,295.17188 c 0,0 4.27888,0.16147 4.35962,-1.85688 0.0807,-2.01833 0,-3.13152 0,-3.13152"
+ id="path5724-2"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ <path
+ sodipodi:nodetypes="ccc"
+ inkscape:connector-curvature="0"
+ id="path5774-1"
+ d="m 175.4398,289.97057 1.58408,1.58718 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 175.65167,290.18348 -1.3846,1.38461 z"
+ id="path5776-8"
+ inkscape:connector-curvature="0" />
+ <path
+ inkscape:connector-curvature="0"
+ id="path5774-4-1-8-1"
+ d="m 180.18547,295.38582 -1.5975,-1.59853 z"
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ sodipodi:nodetypes="ccc" />
+ <path
+ style="fill:none;stroke:#4d4d4d;stroke-width:0.60413605;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-dasharray:none;stroke-opacity:1"
+ d="m 179.97256,295.17188 -1.38459,1.38461 z"
+ id="path5776-0-8-5-3"
+ inkscape:connector-curvature="0"
+ sodipodi:nodetypes="ccc" />
+ </g>
+ <text
+ id="text5581-6-3-0-0-0"
+ y="268.54803"
+ x="152.44072"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ xml:space="preserve"><tspan
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757"
+ y="268.54803"
+ x="152.44072"
+ sodipodi:role="line"
+ id="tspan11446-82">Copy</tspan></text>
+ <rect
+ style="fill:#ff9955;stroke-width:0.29721317"
+ id="rect11641"
+ width="19.106768"
+ height="9.3974962"
+ x="81.589874"
+ y="273.58127" />
+ <circle
+ r="4.3386316"
+ cy="280.75192"
+ cx="136.74632"
+ id="circle12239"
+ style="opacity:0.34800002;fill:#000000;fill-opacity:0.94036698;stroke-width:0.31968862;filter:url(#filter12541)"
+ transform="matrix(0.96462141,0,0,0.96462141,-5.443653,7.4607325)" />
+ <circle
+ r="4.1598816"
+ cy="278.28003"
+ cx="126.46479"
+ id="circle11643"
+ style="fill:#e5ff5b;fill-opacity:1;stroke-width:0.30651757" />
+ <text
+ xml:space="preserve"
+ style="font-style:normal;font-weight:normal;font-size:12.26070404px;line-height:1.25;font-family:sans-serif;letter-spacing:0px;word-spacing:0px;fill:#4d4d4d;fill-opacity:1;stroke:none;stroke-width:0.30651757"
+ x="28.067818"
+ y="235.88847"
+ id="text1045"><tspan
+ sodipodi:role="line"
+ id="tspan1043"
+ x="28.067818"
+ y="235.88847"
+ style="font-size:5.31297159px;fill:#4d4d4d;stroke-width:0.30651757">h<tspan
+ id="tspan1041"
+ style="font-size:2.45214057px;stroke-width:0.30651757">t-1</tspan></tspan></text>
+ </g>
+</svg>
diff --git a/5-NLP/16-RNN/images/multi-layer-lstm.jpg b/5-NLP/16-RNN/images/multi-layer-lstm.jpg
new file mode 100644
index 0000000000000000000000000000000000000000..96e8f18548e37d4f415ff46db0b662d6a8a2f155
GIT binary patch
literal 29122
zcmex=<Ns}j76xV}CJ<s~W@ce#W?|)EWo2Pu<zi=N<KW@q;o;`u=H}%S65!<%<m2WR
z5EBp-77-N{<>42X5EGFQ5)l<)WM*MuWntxHW#ts%<>nP38T>!MAjrX3%H+VzsKme|
z$jB_n`2PswA_fLVRz@&jfC5G)W)@a9b`DN1ZUF`+MrLLv7G{uZ85kIA8JU<FSOi&x
z6b&8OgaZ@Vl?p|S8YeE~P<GmQP&DY`2NmO_q9#r*F>wh=DOELf4NWZ*Q!{f5ODks=
zS2uSLPp{yR(6I1`$f)F$)U@=B%&g*)(z5c3%Btp;*0%PJ&aO$5r%atTea6gLixw|g
zx@`H1m8&*w-m-Pu_8mKS9XfpE=&|D`PM*4S`O4L6*Kgds_3+W-Cr_U}fAR9w$4{TX
zeEs(Q$IoAk49sBPfCz|(ko+Uaz{JSJ!otkL&dS2Vz{ph2z{tcb$ik{<$R^|%$evgz
ztYp;4A>uS~;l_iU%Emz-M3agxa*3&!JXHM%@)Fob#CfQREFZx=1ozMXTMRtRj0{YI
z%z_N|41a?cb!jjRLd<M@Wd0}C9sd~`{xiIA+EMk+#`8aeLUjKR{(mjn?d$Vd|Cqo2
z=Pp{`p|Pmz3ts`m2M_8Lv$yNl%k^hk-*(y9xo%0v!#jqG$-GzVUaHGHX5P8ZCQfS}
zSI_FJH#v99uKBfg>XufuJx%Jep-)5YCz^_Xdp7$&gClqR??3w=$JxzFU6Li5n;0tF
zk~IB|*#6IN`Tu^<et2KP=~rNV{U65a<Lm!3<Yn&vBCz#8!~WC%OzM$1rcdM(w*F_B
zpHct5(fx*Q{qtM$zaIqsXNXb!&tND2kM%zTgYDz%{~5NM-2cT9^`GJX)Bg-o_9(nh
z{2QYFGo0UI|NX#XnDrW?CV-=1c!tA`#n()J#wf){Tg!EATB-hZb?GDB<xy4>auXFk
zIQE3QKB!Hrm5*rlt(+HY_Fj`GulQRf(<G*OyCawkZ#<SbxjtC`QIy52S1k)3-Me+S
zitBA&RjNo}d{RW9s|Z7&Yw<km>&q{iv__i?-j6-`#91-+kp7fOKi)U+vHMJ%r>(VB
z(M@QjdCJuLCwy<*-MMeu=Hs{Yl$vdAj&sjHw;Cm-n?C-}aPx!pmdWb3((B?X|5eKT
znr1UA9-ds+H_aEn6qF?r9k=Z->wCHVe|IkX&v3!%`oI4S`adqB@UBm&ZMy!icHw`9
zeXNKa6Zmh3^S>JVKivNqNY7=Xno%N)aIrK<0&D|svG4gmExPsZfBk3Zjqwx<;p>t0
zS8P0C`ludOmfl$T<*!>l=bi4?LKkl@)GtnJs4QOov1acaK1bPOe2(k(=!;%hA7=6@
zxX|DG-hYM#v$MmhO+`9hD`_vhvxbqSTx9Ea*H)202UaMSYhG4=!tC<D9n1eUX<ZGy
zZ2v?$^FKpO@P7tJ(X4%O^}@EUo47O<El6~&*xUbh@`v=GABXi1y{?zN;rTe@oz$5d
z5>EwZw^|tJJhfRiL4JBd&AXq46I{!#+l$#)XMW5&{cz9ZGd97MOYF9bFtHiV4w>R|
zCgt~n_X@eM&oboynEcOSyZi_FzZ~KGvBx{@pKurcXJGu#aG|x}>)QwW+y2&{c(nc_
zYx-+D#SIrse(L9^e|o#Pq<5y^62-ZPG!8XYrZIn9v%j+_d%_>vkJ3M~jP>?y6P6YU
zVm7(Wcft5r@nzeOaiwBYA8cNDJ0>zSYTwjt)fUSpWorgyx0@SUEt!{arhl)<v?q<b
z+2_bz*Um_o_^RrBcS$|p`DC%BE1c6+-z?%TRIXoB<ka}~w8rz@jQ6&#ettG}kM6^r
zZ|paH?e(h=dJ-2GbvAo%O2k}-<UNXqwshQ(Jjw8%fwz|3CAGRajO){WvH3sNY}Nnu
zoc~`__T_2c?!LQsF;B_dqsgto;@%2R#;&gi6?%S4eU=yd&v4@D%Kr@4E$Y7r#~U75
zYqPEF-F_pLSy|6YMFQh{ricW(G7!R|YV(inU0nR1L4W1nEUU@>{~0EH>;2CVzwtjq
z)Yqc<-+#$Jil1q8<C~BJ%j5>@>vLZ17r*eMI_S!ly=s5o3SE2TEgM$BxMfGhqw+Xr
z_8UhFBpRfZ*(YRQ$a_2K)3xkM<EN*3r4t@J=#ly?9nbk=`NQXjJ-5t`_0z56RrmQV
zH#=`u!Ecww5S6&JnAKAmN|;qruD=)B?{u|e>wkvcqKKE~hBt0Il<xLXUiw=iVP5?1
zr$^i#Mb78i`GU{i>uZFHv9nAItIom>c8e2V*K2(IdcSM=hkx5e@AW^AzBOgix#(!s
zd7g8;7Qf3cJ~izhgTr~RuN_yzOgVOk@USvV$jw+P_I`!o)k8IY_AjN>|CUVu&)~4`
z-46fG`yW3ps=w}8|Ap(*=lY-G*Z(tcD@Dt!6$y-QCN5Cp{xkfu$*X@oyIwg=Ywv%C
zKNWZWGh8?O=eA|}$KzN0lW&+9&lUFn&#=DYKf@QnfUJF*7q@87(ETbKbnaN?e}=2B
zg;tC0F9@IYUHG5j1VUk4Y4W11X?Erx8?NrrGyNT#^-ei+MSI7wY4cn%5++#QSs0s|
z!{B&--p=(erro+xc;mIsKjn<?2JyR3UAq(&6BAuKFKAijEb)|NDNWVJ)CBe*H^W&M
zKkV;ZYZJYs*XgW!m%Ukpoc4^KB6pR%z^hY!8BL6HGqq>)+qm-5JLN0OX1$5LEB(Yr
ze#;q~L;n(P&orO&*1Gm%?D>P0LH-Q4bd=35dg}yu=&?MCxO&2qIYGvNv90pwrs6qQ
zj`{8uxz=9nDiXE?RFV)|f}|gq&#^ys<)7FOyC10^m@ii$_5Sd#ZT+k7ypEDTGP|N#
z_s!)PPu;C|jCsuGO59r4(7b`w+H>`MhQ&Y5ud2IMC30VEk9gto_#f2=rq2?Yr?n|;
zy3e&aajr`KGL24)cgf5+$veUC>ba}G`Qv^peYl_JkJE?sO{LP$n~&6~cfE_%x)Ocj
zwZ%5q%1y79250TCH<Wx~RIzSTgE4ZW`pnncwsD^{+9t1zez|VROsPj_?y$)=Fu5Fm
z9XIol|4*hn^#|&|D8|@UOL(ljSXle}_V(mSYv!me*?<1CrTv?FDZY8?m;8U<QvT22
zJuzf!JjajqQGNHU?^ZkR@UHH7x0rvS@23+zlG5D=FEnKDT(rw>wf~;|A2jP{)TY%g
z{?E{G{7=8mjvw~*#kzZX^;<&U-aq3k-mc!S{&(&6#RVtW*_9P!3bdF1JOA;-e});~
z+h_lKu(|*5?sc2G;zO5h%m2^7H2r7$wY2Zv{}~wjf69ygsl2eKe&wICO}>&#_I>1u
zD$$v4B_b59;}z$Vr{;Fcc+R8);VXV5T|XvP6m$G>_G<6F8SZ?~x5hcWwcvB$ioUYq
z`1Q55?2U82uh;&V|EDi%{@>#E{|t_%A0p2M%E(Czc803lc`|qXve(PQpPl!Xp6xB(
zwzRF}(G=13xpn`#PfA`1bge6>NOTnmU?hYCJg&c7nW^sEbuWAWw&Pp%mS#M*t(qEU
zztle{^FPBUxr_T>f3trbw=8UZ-=g*Tm32w`^|tN2)qP%;|7S}@<^Fe?7XKM;C%fPN
zQ)2&&_2IMs3_Gqo{?CxhTRJ8Do!z1M&!6Jkf6CU~vj6<fzW?W(`QN7Z?SB#b;p2aX
z9e@5a{AqRnw!HUvJa0z)2idCpf6nVRb=mg(XL$Z*|EKtWF1B0jpI@o}l>g7;r}VcK
zAJ_NR*gmRfzw5C0@UJHK0)>C4uE;$;d)j@%fg>|t*ab;Qy|K}MG+*w=V)u*BKJIIl
zoA&Dbx9Y{eSLU&u^FDl)->>3aMb!6~U#hv+y50JhtzDVFcIqVVnM%PbjvIGq3r^r;
zD|{JKf8_m-9~bRkFWLW#wPu>V&RRE%Ju$0K9t-?Yvi;Dsy>?Ib`umH{ytuY;c}p+H
zO82+fweOe5@7?b_by1i0r{Y6g8jBhT;<YO``kK9+>b;@!L|}vesfuT7u2%nN`0;V)
ze};9re|lr>d#gI<uIqhtw$h|pz(#oGt`23!X`jxs<jQ<DSUvy4jMvTU8%r1edwaG0
zKZ8oxxAy-G3cmLL_CEg4aKSZh{R{nM`=7t{|4jeSFwtsiM)Z%s{|qPh|5Waf`g-f;
z-sOL^bCch!lQ7X%o4uh;WrOnDi0A5dJ56T!Z8vMo{1$8daNFkP(RsR0ZvAISS4#P6
z5xpjDcbCRD;r|Tt*T3uk&!A+V%m4Gseo&-*c;Ek@LAb_o;UB{*dSBiO$9%Z^q}XwD
zz|6O0%io79yqS64)^JPnyoxpcH(!72`zU`T`^A>p<inHiTm4zTHuK`!E<39;e4=%$
z>bm1zKU&Y3X=iqE$)&BkP4``XQZn&DS}IRyvtHSR!U-1d!%Oz6op#>Xb=q%B)3I}k
z9S^qmxOLe!JlfNlV7Ojxqu>!!UzJ&#%CCOe<bA#PdY?Vp27Z1Wt5WG)>#e!s(cAu-
z=?1!resMofbb)y?e*612ugD+F$B#djoB1LBpTO_iGap{l`!cb<(B!j{(s8cly=_OQ
z*my};B<}QZaQV-$X?f?L{73z=TV~yVE4rj6__DNPb4Wb@?TR>AwWmtEI91nPRNyq5
za`)2&!)JOA__XwIS0C7?xBH`cgH7eb{p=?{3SB<Dt?F#={r0u7b=L*=ZueNcw^90U
zvP(E?rA1*+(c$*LJvE$<*9-i3{BV8e*7xnR<{#dxf1sZK@*(4x*M)1ucU`^s$%w_i
z!cl(8nWY~*>(~Pr>~4Hi|I?Uf|G@s2z&g3z{X+F8#2)`+{?D+G>*4F$2kUu%{b!hX
zH2x8n#-hE<b2JupF=(Iq6aAk-dD`;-3^5P?GhFbkT3Q+3UH_!$%72E}bL@X{uAllS
z|EJjbt92=RFV1xp34g2#YtFKYginmPA2a{f<?zks!qTQ6VwW7cbE0O$vtJLyxGxtn
z@I1C)+R|eAbI;UcQ_sgP+L!j@?L*)2*dNnZyfV)GHNE$bW$F@F$xAhskCc<^7x_fq
zxYJ#z#2I;oX~~?qEx8KewjZSr%@hCVe|UY{Jflmc{X$tm*}tx}FZW#;`&vA}zclNU
zijmI7qe{m%iAtn}vMbD)R2;kEkeiq8>3Z#(fB!Q)eOkIQ%Q~yJ)U~GnW8Jy>ru||z
z{G|`Gd%bMg4@bE^+q&ZArTgbgzjeoNbre0Bti7>lVvMK49X~Ukq%B|KC11r^{9RXK
zn{_#8<{yQNS+n%!vINU3JzcBRxqQbhzH4W;^GoD=Wd8Ns@!R9HxB2&yuiN&2ICygD
z`IQ;#xjufIGhf&~qg$Wp_4Mr*ek7eftR5v*I`z%A`FT65C-bFL%#5678sPM#SV7%%
z`XhV!?Fan%GVe)!xVE?Oajtv7^<^6xzsg&OPWEsVocVB~?qPpT*Z8UY=i5&vtd7(%
zp48R-ecrb1{((hzeyMf+O!*V|k-x7d^ur<FaKC*Q^cG$F+AVqEc5I~c3b$D)o7|0D
zt=7!ubLZoel%Cv>eYHOI_w8QyZ#5r8uip7|H|F$GmruUN)60|ROgcU(GxX4@9?K>F
z8P?XH@G}dyb?eetXXMnSv4CmG<^K#%Y!}yGe^Y;L@k*2b3_tlZ?tlF)|8;%1=6{Cz
zh5uB4PS)*DT-2rgeNz)i14?7?-RqVgbN@5QOF`R;>*dY=Gfb?l|F9`_<Ee|TqCYGh
zq16^k@p;)bM!qJUJb^3IC%JL0Vh{3sp?UGdWVcPq?Y(VJQ!QEKTBIJ>XC(e-5aWO3
z{<nKp|G&weG5b5V>hovK-z;*g^glzF;)S*8AC46G*XP~KzO_1f)$~u7pDCnIuuh+N
zZC9fEJgwNfU0%;u-TtxKb=S5XJDbBd%({28>AXp~#z{eS219nP;^QV0er$K%IW?(0
z^77M(yOd%T)qCbKeqq~}@+0}7JYVL0;VtLceoQ}5$FqCKRU4bPY%3pnM!wNsn>0!J
z-^{dAB9m@QrsV$lz`yi;i#(5wZ03Ik?q&b@xiTi)&x}*KqWk6dj4wZ?d0QOqVNvlB
z*Vu3UVTZioSJSIHx)%5E)k<B=F3<aWs?hy%K$h6{u%vg=GnQ^u*f7tMS&cn-M`2^L
z|JL12zgXto%6O+xICYX5&l=l~2iFhVbA8PX_WWVF=#N-iWb5-@U6a>qm7Y4P*_wS5
zR$FrT@+rgE^8XCeK3?BkBem<Q$^9FXPCY;T;y=S}gNm$A-mD#|dp>NM@vZN9frRW?
z(G78@{(ZmYS!MnDM_t0h<a-I%a`e_e+tl{!R`v<YX}2t=F6saKlwJAb>ZzaQ58vmm
z5xw+>@x%NhiEnbY?tHfSH{bf@)Bau9cSR*Yr7yilQsvH@N3Jz1p2%>W{c!xaf2Y;^
zo?KJ^%~mhW<+LU{P4?Ysl_NIMVe=M?NT~-gTTgKBPxLW-7|tTB&-UcL^2PGY@w&g+
z)stU&m}W~)?^N8ged^OY$stoC_FZl}#&2OG@3l2vuI~JT`a`)NX8+s!C-FbSa@X}6
zUi0|!C)!2duRpl;m$2%oLtPr{g1~8c_5J@0$9vEJEzW<Ebu@hce};bP`M;jsKXGl!
zKZ{?TZ>s*GR<NtA<o`69{Ac*}y&u$5uiC%pV2((jt0AaC4k`<Yu47K`->y6T5O=Ne
zp<myR%$I*{&$q{M>Fo7C+`h*1#-C<e7W-$urDB(9vVp1s57z`E%h;Rid43fB4*hpA
zp36pgRmJh+a;#VP6n{wfnaJ@`ZD)G^!XF(`LgH_E(^M0+*13lUJ!rdQm%H!!@5GA3
zM{ngHk!N`EpMl@RpVRko$fd1w^K`3^hWQrVs$IJ&dS>z#?i16(G8z@W^v@4G@$YQB
zXr2Ba&mWBsd-)%I{oY*qp0(osLG~lJLbuIaHc788JNLsGl@q)YbMj(DtsB`D8W_K^
zZ7vo4-gfI5yO5)iyGovw(WE8vC$3AzZtqZz*rGX6L`lP8)nW$T!;LY%)fM^=_s4&-
zSpCP*RrKt9*@(HtN2EN3qGK|?eYoRYXw|QJWVNzJjF~5snM&d9YrEek&v~h_UiYG%
z$-4CmB5oh&vHvj5H&OJ=wC_wFI*ZFXQ)(|KZdkQs-t(T)b@vZO-F@C^qrL1^`8WBS
z)2AEePr71RFlT~+2TK!!vVx49Tc)P^hFcl8(OTB8rarCH{m(G*bnt(Mbp`(!Ua0Mh
z@7(_++CKE>gzs*s!vLbs_nG|=e5fz~V_}W!s-r92*KoUR{Ovb?b!_BhuH9A(1x_sJ
zJ$HP5)E6zShe0KGAG|qew#{aJ*x#9ZxYy{epPj$zsEyhDgzD9~S??s@t^F?cY5Bwy
zUZbb|_f;B?6~=~sKlGo0=arny1rzbhVYcc{JGSn+yf$It;|P`Q3Wti82#GWvJYoAe
zHdHikZ)DV`s>;4M&$reyWJG=MVB1=s`fc{dW#^Yn%al&sI&<pMo@Hforo<J_v$_`F
zZpT;QeaQP^=%td)8T-VWp568eo#&+J`R(UT)_p}s9Z#n`SKqnrTG#w+lbw}w_pkq4
zUlBW}zW<+aMfRhRS+&PDXBX^hn-g>Ml~wJNZRKIV+hkNW8f{=<oH$2@Ph02T`G0Kw
z><1sNFzM9QQP^uWCHQunuk5YZInN%%w12L-#L3z%<j&rweNkgo=qgu@mDc(OyDenp
zAH+9rF}=jQZN9bt(z`M7j>WcbE}i)<d?aaBWbB+RJtvN&b{>~F##(Yu;KRI`5Bzyv
z-jlzWd+WN_<%I&#PM1tt1xlo6OsW@-7701E=J}E(_rLyUU~Rq4rLkzU0BGPjWLEv3
z_AU9p{`UX6QmQupKZD|1>3{o^|80Hs)c*Y!`y=~}Qa8Ra35*fj`_WCKM)R@tTQg<3
zg%5wKZ}oUTC1=N6WjE=vZC=7kdKC(jkBk1koS7H)yxO&L+18Jf%r;z^eE8bO_*b`z
zHs5hwA*s9Y*7EG!f}Y6k+|U3%Z@#N9qGGRXzm>Z5%(rKkDi?aHY)|7!N?I^grS#xq
zyDO9Z6E>#duM^dA)rqXJ;oh6CeL20>I$&$puS4(DnjYOM;g{LC;jp&mgC<TEsV7_f
z8~-!#+Zb0}ukvp);kA0Xs3Iym*G%K);!>>_9&w?K3Vkh7Z(pqLc`ll<TVc0lJo_K5
z5BZ00&z?P>=SRHfg*-jCo69c0dE9!pc<z!F=k9#&o4P*rVXv}ZtZ#0~Y46f2&#Jp0
zifAo*<`%H#Q$o#|5C+dz3~Q}4il@xIy(4zUwK(nzCbxU_s(r-{-OAOvaWleGbdz?}
z0S7jX6Y^S#;ZG%GxVA6Kh$?T}vbxsTbK~~QC9hL&@-AJIWU*NCoTxkVq^z$ZbC!Hh
zES_}9PVk4a<42WwikEAY7e!=G5=*<H`$oxg)BTDz?g@T=9<S$Rx&8=$@Z@9V$Dkka
z9&#$`g;(p2%il6jmfrr@dgHNoJ06=}xfuF3?d!AJ?Jul1bNhTyZ#RkV*6!xz_S-b?
zgrDw`rkjEbh0T683E0KH{cL76&o}R)tMlY(YeAEH4xnK_Q9`3o*T2~RWXk-{@b$g?
zzb37(ffxQWJmFmapW*tR`d=KYtj_m**(>jG-L%l^e)Nlz%5lfjUaL&plDT(6Pme_q
z-zCQ<RoB17zTr+T>UzG%{qbJ)L%+PU#QkkPE<Eg-%zWp}9wASU!vYK}lY4l=uFbDy
zTbY<!o%Jtw>zQnmAjaMk2SjatHEHa+k@6+b)d1A@v}O1C&oCi(`QJkSzbv7vRwIux
z*X;SHR@I#%66or1PWyN~hmG+^;UoJ5KC*4hRDNwIvZwm4-aF1&<~_S|CYc?-*Ly&E
z!iTbcub-<QsvnxqQg>c{+m$+=565P%z9Socu-t0t++)kvY(2WIH}~69{?zom;O~W-
zTZ+6VAG0mqy0XIj+o@Www`=MYY7F|0E-TKP{IXnE>G+N}JWdG{3}=-e+}~g?ty`bz
zo!=ENW)u8Sw)o9HsriRLI6u5y`Jj})U^1K53&yAMbDmE;%1~>LJO)%Qr}Cd+g6{D@
z?Ee{@M4#SM{P6eQwhPkv0<WqhdIU}{U%OK%U1s}ik-3JqWwY;|zBEND@7BM0?!5IK
zH`lX%3_mp2>-O>O|7`bc+j>>^!QP$wr=5$u-5v4eaXyz)!pbyVZL`Af?Ka^{{xck6
zU)?ufz{bn|?X<c78BR=*XMD|){7+20v+zCJtom=~_C^LQ*Z)}o>I}O)3(j8neAoQm
zwX+Y@C++@_y8Fbf5W9*`qH7s4CKntJ>RR&g#g6r^KY~lP*Kuw$yzQ9F5-uL#q#LrP
z$X)hG$0Q9~X05hAS;h}^Qx*n)<Ui8taqM#L(#LZ1KR$P=5RU2<4Spov5$bS2KJ{{e
z{imQwhYwo+4*l-^pJ5`q`k(z1SM9U;@VR1LJkNiI>|4wB3+XKPFTbrBCduk}?4AAY
zIpyn=kM93unN|Pwocv#w>*xM6NZO~{%l~ou@wB4kq`v;!AnBqS=hYl<AG0XDNa|%4
zwW;ayc>g+X>%;XNS^wCpd$prKs-ExCU(_dieY<xXbN(InX$n;v)n8uPbI;_!!-AV8
zS(nx=)hyaBx9Z#LU)IL6MAa>C1#a>$F6+`*|9tZrP}h<W*7CLg=X}fjZ{6{~SyoLM
z`OEiTT6{Tg!R9^Y6W`2xcIRC8^;PUn_3szP?F*QEMnmoGd;K$=In}8(H&@l@U#(%b
z+x(c<+5hs_mwLLUZ1cEh&b<ECy!_&pEP-`4TU<Ki+Mnj_T%Yvs_{SG7AI)X{_~Fa_
zKeIh<+4s*7sPTVzOs@Cg&GY~6UH`(e;mgzi4Eo{!eE&0?==EJUfBMfk`bXvq-OSp@
z_c5lZBJbACFDIm4pSz-a%_F7aiNpbklXsi=*YQ^CwgfmP=gY2I6tyI4qgL&Oh646&
z+LI@{eSC8w$X}=;&BZo!k9mRPe}<F40+;^y>%e@ZPU^q|#)q#P|1)gAq9x$>@0xwd
z{+|LxwfmJ<?Yy#9S5J5C^IdJ6?;_R+dMboXX`Cb?JWZkaXyJU}{f{5J{AXA<=|95@
z(a-naPF_%PJ!s2^+$EcDWlx`U&-Yt|iLG8n8lR|N<-DMJ2E(qKKMEJrgk1P?R<=$d
zE6_is=3r)K(Y?|)?is&7HTNWM)TzIFOyj)Ku98nuGmb`?7f#>3_U^hZul{9L&AE4T
z=dPN)TW#68G}b?BCN3J0N9osu=DI8U@4Cfk7n~s6cMLRWf0gr1H5*Ecy5^kf)wk>S
zgqwG!UD<M;^R;2ttTh`VZ+trYZAZe^@ChsbK6d3i^l_rXgRko|FK*?R=aoC2*7$In
zzoJ2O^bVHZ#sB7<HILc$ae2p<lUv?8AG#;CWAlua73Vwh!mT3=o<ydvVB5cBY3rP6
zz4Z?Q|1%i6W$OQ6mwwdCeJE^J*<SPZym$W@wyN{wA8kI8@@f0eNvnS@PH+m(JRkF*
zDEmf@rNyof-=-UUE>vCPXY=DoQNjXyBOX?}#WLZuvg25;m%gs<$lD|QaKUyVPSuPW
z|8i+d9+{MGz1a;G-_}h!_WJlVv4Y#~5BvEg9y)YIAKA`#gIQ*B@}$}qePWBs_tbyT
z_|GuIa=rb|xqC$))^AbU{5a%?@uSI=3p2in9lDXfbMtkzX%YV!D)y|J9dhzNLnGH`
z{+rQ{{EyjIJ-T!)*Yc0+LxY3z!B=M-xv;Fjr&w^N$G>+nUnB$1R-0_9np#kNbmh^a
z-H}^w{j$FGFG|8|SI-rluXcwQb!i{4@^?g?Z(Q|q{wLlo{~6Y4{%2UwwKZPfzH{E9
zo2(*%0S1e@G{{&FgVDZzRxhCIH~Ue<dzC}C<Bv_$X<sn2yk^O-smgM@qxNNYtUR|a
zI?yrhT79wN7sC~cy8Z-gfGn%Qzd#6i>|w>L=dEw~);`*8vFPWdqpepD^L3bQaGW1>
z=kZ0~>PTJ7t7kM;Xy07^$;CVQTE?QTzc;`(tP_4z{--8m{?|+Kf0=ggdc05VruECP
zT}_%P8>XBzU{-MnY0E$GR{e(5r_WwRtDDwxXm1z)DXiP4*Ole5Y0)l&lEkU0mMm3W
zrFHDvdKM7l=2hSSGkknp{GTD__D`veq3^D~+q0XyaNnm7GZOmOd9thgW}M`opuL#a
zuXMH39MRhuGc)?*QJNI1uZphSbt~GLsc55ugZ~1FmD#^!vJ)0{Y4)sL@p4<$(J9-?
zzio)S<z=Y4_e_$<opcK=H-3eXkH;n)n|Q2SN`Y(DVg|+wjDCye-d%ga(N!c!@`Y9U
zpXOZq*Io55l-7pE8vk?LHvj8U`!9lFtDipqXnZAFVNsVB8>lUmRzcjvu}ENC<2jH9
zQQo&x_bts<-Fii{Fkj*Daiyl^A6BkdqCC02t;#X~?#WuN{|tqCGaCOh)TaE~dhW-6
z2FD9E%n$BH$M4>?-!S69B^Gmw&+?1Z{ni|He;hYAUjN_CS=;um?v*|?%Wdk4xTuNW
zey_}3JI|%;WW;4H>%>mE*DvGb?E1ucok~L`c9wh(d;g^RCtoDbb;G5tv$sg^nfvzL
z+W7mMKBvmxEIw-XpFwDH$i=OHoj!hDU6i+=bbkJ9PJ4sns|@9v`MrKI{}b(-{-41<
zs{J2J{nynG>f6Nj8GqQ{lk@d1`{7$#zn+^~x~byro27N`LQcO!-l}@C$hl>IX*%`%
z?Y4rL${@#Q$2L_4>s|NhdDxz0B^a0H?a$w_@MKumv;J_=RloaJJf1U~e|j3%KCwGf
zO*d`db?w#ZlrPy==j{KZEvx^pZY%#Ej{gj=1OGE{9rymvFn_J{e+DM|UstaO#yfxI
zQUCL+{~y!-6MybcmFXv>!TpNC{LioKe{lV0P<ZODjAr7M`a>Hp{bzXoI^aLUf&PD6
zORoeX%(o0Y@t*-zUjD<HlKnq_U9tb5{GVZMbp9h>D|;d51zVnEZMtRq%JJzoDbZ55
z+#9zKTlJ*dd|6lQ^kd`VR}(qjEpIv<T)nzGLx<(zlliCTd4x_@NU!Wye(n9r!v5!1
z`JMF>ij<Dy_sYr#?}gQSYOJ$YyJzc8I(JEP`<#^}n#zhSE0rySFU-2N{40<0e}?IS
z{}~R1|76#)l>ZTC760dNmi!OF{|wg_qsHgDMg1!hK{HGrME^OS>T^etUGblxHF@cO
zhV$1J{%2^E|C{xC#o|{I{y)FU|6#44P_%d=y1f0vh?n)Bzpki%;QpT>YS$}`Uv0ww
zYTf@cFvdTTS@C50!)cSGpUwN{`fl0FwtB(lSh49#B412vx^?WX?n8DJ{V>~;>XT~E
zo91=hn<m#Q|IpF>q6(*qNT6$p2q+tYaIttjpGox0D38UnPHa3G9XV6j?(}!}2dp(;
ze@DD3+t2^S*WB&mSMDEMYaRc*l!=*hlu2$+Letp>{zY4BBjcI>RDM|BUz7Z}U$A1{
z@554a_f5?^v(ZyWJtgd73iEs)rsFcHy*1m)y{2fGb9tVMtt^eb`=<6i%hMk@ahXy(
zR=PUP+m@?+I%Bo%*XvK8)k=SGEy?ei&;Q1>xx8vuQ~l!mF5BgcN}0<g9vCRLZ%*D?
zz8@u-#9P#ztdsuuyxAsL=JJbh<wLvfZphS&ynH%Fvtp^dx#hWg*7h6Lug|X8&fk6{
zak17Gp-C6-X~)~{n&&T^x}#>h@k(!QQ?2~aw7KiQ#cs>I8(kgoq#h;6`%&V0-L9@L
z{}#6VXP6I34Dz~EkKvTh$Fo@``zjsf$%=ha#I&k$lBX?$NUbyHe+Ik7{}~$Af8q-%
znvW6+?kLGStoM7*m)mL=B2#~dZriHab?w#NsSoXURyI#Ne^*mlz^%IK@;jU4rMF-2
zN}Kq6%l4}NnNK7Z>r38PIzyZ}!$Owx<d>yiW#s?->c7)GRk9wBMd7io_Vi2Y1fs<b
zDOT9EECoLt_dfrp=9T;pf%-$+O#HcC&XZ5CwyyDe%x|+UWn%lXXDW?vEWWMH(vGT#
z5{Wmp%CuLRP_$rCm-Y(e+Lrl3$@!mbx9YEpfTlsCR_#Kn(ZWxkf0Vvro1?2p2oI<m
zat38utX^o6iT50yzxV%gg!As)zoXvjn?Z3;=9c@x)7Dy^3h&zSN^|$@-|e3l7uOg3
zE4}+x@z06lH?I5(v+Bxw_V3}J7kBR8v7ak<N&ZYTsiGBk-#+>y`{QhBZH0@S$j9l^
z<es)EZ@D6nz-BArvFFM6Z^3bABkNQ?yqo*zt<Y=Ft8apS`|ph1bXcvRmP2`h<3W}p
zjz@d8T|RtY$WDYY_u)&WkN4XaKaw>Mo_%89_P7(1<)>b$-};6@^tsk*ZuM)t_kE-C
zYpqtlUbfHu(y6YqD-Uj47^ms_Ia6}UTJbYqg1k<Cjx?FP_g#P5qCHdUE+qRimfmNo
z*zGo@_L)`YESdido0a71wd@}*`=elyom(iqYui*a_kY$=YmRgNXRsIgdik^EeEzh(
z2A}F?t=}6fx7D>${g27}=Kl=O+qV8^Xj-}d$1MAY%WLHSGuW7}|0C-ApW(>me-{qb
zpTF_<(SL^U&H2Ar=l^3mAH8VL<bP7-2mdpCJ{<L*;b6%9KfdxGF7M<2&oEEw`aiB^
z{}~=-K=nQRw?6Iu7jFAMtTW?<YeXOIt$tJ@b#2b}*UN70o0_sQ;*O45gzM#)*b`?w
zteuv4gxfs%P@r|ZsdPWf-RO*cZtb>fzg+hGCU0TV!+tA8aAVD!l&3pi&M8~HAI-k3
zsq5b|e)x7$$-7oNur={NLzuAZgq$by?BXm{{+Uj{UNC+C*8Y24{|=n}_G`_5h7V$8
z_G~rle>g1olwfvBUq9sVx3)Rjx>r`+D0|Z6Xy04*^|6%uTlNqC85}qN)|<QR)wZzT
zxBRtul~QBR6dh-*npBka96ji^&PWv6zP10J`r)5j|1;c8|Ig5{`Pz&1D*qYU=6<OE
z^Wob5525D&8HC-@HLiclqyFb_|2_S$)n@-07DWDMc%bujm-fl}hs*a!Se9H<j(c+}
z=7;=`x3QP)GyU76mfZ9D(I@S>xoP@^)A{{Jukc@+b^Kw<n^ni(yZ9@fP1w1#b?*K{
z{~1nvxuvyt+CsLa*|KNnY`h!oVKmw4;l=h9%FV%f`S0Y<?b^G>ex{uB!cc3A{X0rU
z?w<X7_~%7%?y_BCKa-o(tb!x6Ex4kSa8*YyXI-+;?U*AMPD{QxaiGawz}FjWlezhK
z{^!mseP7(t3|=iZ%kTMw8p-RaAKVreemERI<@k!srSsi)&Tgzv^3(|VqBo~$@dDXz
zlj9D@Ebsp(^Wwgc$%(h6(Un3K@@Ka0-&U{p?COu7k827lWJ)X~Ww;(K?0@vU!F{q&
z=6%7EBNjbp^>38&?&~@JJ-0vj<a#sFeJl3P+519w*43pq*M51o{#mVVjfrukjjN?b
z*z}{4)hqXH>Y2Z5{qCTw^J&-a9dxbK_;GiARL!pbL+3@0YWzNQN@Z94!6`i#|1)s^
ztN*1IXZN4sp!t7>^r-&~84s(@f2e;_^q*n%&G}y8U#7%AIUluX&-8!dfBrLk@!3=R
zpP?!L-?{!D@t^-QTr~Rl{U7K4-&XcpHq1Z&v;EiJS@q}VAG3c}b+zce$h-GBb<#g(
z`I_B*_f>iMS=lv$>%7mL2s8Z8P;2l<yYD|k>wgCOzry<R^$%wMXV|<L&5Frq3f9&O
zf9C&MZStRCzI{*qnTLP0mmkjl7SI3VdB8rym$zDvO<#6kTYdVr-#SqxnaWM!YPN@b
z%sfvw_-DOZt6O8a;PpO%{|qhVZ!c#~%<}Kd`Ln&{yh)nU`rXeIG)~^I3q1K~;njaS
zx3agFO*j9~z-3cisJrUaM6pX8j41)@3{;*ggn9os>|(WU?uz3dW<L_$d$uvH`Q+uf
zqTYhLwNwtCyTDh?6BZwTXzjB7@}Kno#-I7m@cjH^`OkJO1jFn0Z*R^&{~2b=zxtN>
zpTVwP;y=U8*CFAP{xcl<&oKXA=k)&lAAJ8a+{B0>_ixjLfBt7M02OKeKR^FC{s$C0
zGwLqX#T);z{P>3Z@3V_W_U@~z{NKtxsCD|Pv+gVB7VYaV**52VTT&a@cJV($ZK%7!
zp_4a5;-^1z`NVTcu;|yGS+P_9GicA7>*~sSDdxzeLtzn10!}KKg*Lu$ly$WS*G$*n
zCjL3`ac{5dxA;34*`@VOR=u{Yt2szz2{?oHXYxODq>iX9KkA6{($CAHU={KD+4GN_
z`??{}Ra6-?IBi_x{+~f1yC1xhrgPoxBlVxy;QMT5Jzo1!-f{YW2F_pqK%?&mz8nIr
z9RTA)yFQ+4OgBENwmSEj;;qxt`L}BM?mYM%V*g>oAM5`Njvo8pPy5fn<-2sZQEBGd
zC;L7xUbOT4kBB<i{|pP3#D70s-ztx)GUnnt{apW_)9;m(gj@b+;O0NP|BFcXzn}RZ
z7XK5IZ?Au$DgO7<{fF)*)$QAs|2h5M%1V3Ae}>lfhxdPRx&CLU*$!G*g{*6q?cv+O
z|5Crtl@&c#f9UXs_kUTH{xj^0{-c4W?X29xw=4fM82vui*Y(`~Vd9VSe@zqX@9X{(
zLDlxGzu?=I{|qy0pS!yj&qsAVine0+9o1R&X9)Y*a`7G8t@h92D=%J^4R^X+`ns5H
z>GE5z1SU4iUeQ*X_$47*8^!N`bnN;5Gh7JL|NV6Tp>~bEHv^A2>3y8#f8}$($RCYe
zKT^NV7x`n7e))Ox_wV~;?nRyZXDXSnqIi=JAJ<<7$@Tsp!n<?!lt1(}j=$V#w*5k9
zsOL%NixYmYW?FtHv>qjLD$C#eek?2Odf`v_vXd)R%&yg`?(aIwwOt@DXzyO{aI2@w
zO%{Dg-)a+5xWn_xT9xO%$M5Z1r}ygY*UesMU+;QTy;SsEe@C3Gy8Nw){3{awYG)k}
z*m_lW+M{!iv{y(k*|4M3zGK~z%QrqITX^2mH99I1=(qqhK?uf?Rm;0f=86T1zFU}|
zZz#QIS<OD%vcAF)lrWy{U9289Cw2~@^x{3s_UM4e?!}nrC{fps64o*o-^tyY|5<;}
zix=yPamROG#&>?wGRWdJFQW20*33E|x?$Uu+~_stfjS-aOuVV6$r?2`1UwaAoa!#f
zVXARf{Z(b{x0(8Ho<EYc`t7}A=0Dk2)4emRmz`Jk*cG#`&U2B<BxUtB_OB|*cU~W_
zv#VI`_>ukKdy$Bv71M)l*Zp|pk~2L#ET-_r3j5YFIk_Lbi|iVgZN0Z*ewNuq$#U)V
zx`VrIO3!6;SRBgZe7W=WqmL#ZEwU<Bo-Dq5bobhv$-X~dU9QdPJB6AhMVBvq{VQ+#
z{x{+mVvTCQ_8v_A)wlTLx|Yi~K8bpG-t;tf>e5)20qUK@aQQDt&tp=(Vp!I;ng1CS
zZu|dwGXF&AtS{4!A3y)NZlSA4umY$Xa`vPApT>2N)p=^W^9BDioRE49S)TVOyKnoE
zTDJH<>aTw(ESO+5CvC@nhOT`Evk%4z{m$L(Dth+p+ja9rkG!6-_SU147Oy}3Q_J7I
zGVV!9+qt*=(5$;>&rRh2qbl&~eSXb$fzwqJ-Bpacvl|w$#0W%AY&hoN_N42i^2MTU
z?W+RgA~SQ>WY$@J*j&*ZvSsJqqj!9gcYHdmysFZ{&4J}zq~)>N*H>PeCip#!(LZw5
zFHLNgjggJvoz=cYAGcl0xY8hO*W-E4T2uD*uJvX=I{z~$`_BJcDgT#m&z)myUOdza
zY2NJpkMU%8ZeW?^i^mZYlr&tJ8AB4dN=3q-oV_1bp$J{*g>#*<ug4{wH?F=%Hr_Iv
z{3`QG0AtXL@9!Vp&Aa!HEu}9~NP0zcqKaFBx!~oZ=L%mBC@);GYU_W7Kh^!eUdsP#
z)4q86)ww>c+09#T#jAaj%G34C>n~i?^?5J%(S0f(r{6N2D_e5$;fwrdo4qqd&z?Qx
zZ0u~(c#x6rN$u+9IXn~R*+*^ss(a&$m-ewY@yCCaH}OtABD5y(CC~FfamM$o$0x;0
zrdj@de)`1ApVwDa)I~14P;yhx{O`iGi;XJhSC?I!^ZVKCFrji2ZT@4k&fF4jf4*be
zfn8-0i7F}D)~t8?ez43bKbjY=_|g8KgV_DAzr{bBwr1`sS!%k=_a9s6lgD1ZWft7o
zp<DYI=eZ|OpB8pkrT6a4U6X3FvNYBoSGNqAJm>ChyBv*0T@yexD_-0dF~?i<?4Gr`
z)y(;A73U=V7k&-Cu<}9ItgcH_BQMTc>3eg5ZSvt?%fH?0Os?O2I=dtECgZtl$`gOc
zbX`zkoRmHNgX@7UA9$xcjXV-yJ=JkiLBxaV>+>S+ysj6l6Z_AQ|DWMdaMsC3yh@q>
z88*c#MOP?IW4pER-?tavHPezDZ{6X$zxnvRaJy;3d4l<6{n_WMw2#%KFM0jTy!E~P
zmeOv)#m7F>HW^wrpW@UqxN}32NwucQ<6r5Ui6=H4Th}8g!6YHmq>*vax^GdJCU`iE
zh)gE>%zn%HBfZ5B`499<R-8T@X7+2^?!`B27hSsaPExfi{NCbhm7FzhoG%W|WM3W_
zFQ2(j`|oyBU#n}M^m+eC2*$WfR;cdvFyLX7l>T9Jeb*n3y+5{BKfLR@&mf{A-2b}R
zhqbdRJfByWZM!i;FD>cz_xCRw8#I`AA@cNk)9ZVJmsG@AEV=qsI@eNFH95PJiA{N(
zaop5L_CJ|pAUhg-3&Z#Ry?cJ)>9ud}{{GkTs8qjGp8spN<eK+?oEP4`<+D4b>DePQ
zcexhf=9qP>i$5IuSh6E$Pw2zDN~g|wF3qm4?k-SCZ+*mj@__b!g$G~OCoI}=Y0bV%
zzvus)xBHv_oBjL$F+P}8*Z<+M??<uuD-M2CcdyXBrx|xE*6D(H#mreoTvKluy`5+I
zc$wu_)gP-LOndp5m*KCeh5e6JuRK^1K4}@V_+B%y4;8t!<{!w=&T7>y@%R5TFnq|D
z{dcX(zOhcxd1<BDujlPgw)BNvF<Cuz>Z1vp*(BJr8JIp!aI5~ZHrDz0F8(97UZ;aT
z{97!xIQyH^`pt@ubAR4?RT=!oHR9v*H<uUu@%Ygy-|=<z&RgfKBOXYlZt7^Bs^gi-
zc#iK~1M~5y@(vsAB^CBz8!m70wYu-K$=k>!?1YH${*~|icb;57>)4vC!;fq?$IP{U
zx1T@Ob(W3tA@`f==a>GjKm02rbWxY~=g&{TEB(L?NsR?exabwH!wYU?o-ZuaI?R&S
z^U3l+;GUwauVJ7ifrbAWVwL|hTsT^qsQ<OA>|g57wz>W;{~21hKiKzA<5K&zm+W8H
zIdR`_{pEZ0>$j<o=4u^!_sBEP#X0hJj-I^g(ofsp@<cqDVD;qd`j|!gqqfFBi!c7q
zVEjgU$-mRT|1&(0W8}TpoWDSK;*Z%6Pfv+ef8)FS{OW$ir=QG(Jg4;T{+GYQPy2$6
z`bXtMR_U!@x~2C^{K&a`JIa~op4(b(wXY6*mm-9W_ye@}FEcUR9uoXj<Ud0VYJ`2f
z-y^@Jt^dPk&r9#Oc<<Fd@+|ux@7c9$f0=ZP9aC*|Dkx$&$x<9@|FEjU|IzVw-Q-L9
zK30?Nx_rvqo}BW^sL0{P;k%ySkCw(u?Bl*!lW^H&t=+S)UruLx@85Fj+J}{sLT{Yj
zB7A86Y)geNFC%WXAByMtWAtI`mMFH#qFc_E3!N-iKVPV>{@(Jr++pThd#XMj`y86P
zKK;~x2EF+od^h@YeifTkcRBv$e})6vfv%!ICw_#j0mfN1K&uY+>JQ%ow#2h%cH|}U
zC@ZX-kp5n3`I;K@4|o5yuetHE>{nfX@ybIVPF7~JR5v6~vv@!6-`3y*_xbne);`p=
z%AU`g@m<(T=#qD$u(9m%mD*=>w;y`XW**P+!u|EK!~YqMTz?Z-8WiUJzWzyVsK%v|
z#bw`O#edE@w*Nztu9vR=Z|x{Ep;tO#g}O_E-JeXJ-oYT+p#1dQo3|?EXRB<d@aCrH
zb8j*}F16SsbE7A-B=?Q#vUNFs)IM03en@XWf8D$JKSRf|{|tu~e=yHyd%^wh-As=E
z3^%iWuD0K_??1y2wpAk2&i=js!Pj2$$5b2Xhqm#tlh}6pT3z#=%(d;(?~u7W-_8kF
z|8V~C^A<ZzcbTZX1-hk1cDZKX1<%j>?N;#k@1@$0J|C|i_|L$bJ&*5N$zS=yvvk{5
z7H6x3U6)VsI8r3|Y1`8vH~s?cjvukV<ty?JZjR2f`g{0CduV4%*PP?I!p#$$nx`Fn
zcbxtAqCLMRAJ=gSwOHq!z5aFf{U_&s6&&0pR6P0FzFGALw?FD%ps}dytG~hfFOZQ~
zm48lauI54}6HnStSZ(@Jx4(b>vA>`ulf#i|mQt?nVHxMHa}|X=kdb$8;(w(1^`m`<
zew}fR;NLa>bSmZ_*w3tQQ~OX~+<B?w`}*x~b9$zSU0Y~#qWsCK#4|ZP%b)dZNZ`Bm
z%lSe65q;?z*AL$oRzIHKc3qxltAySefz0hzYoZ!=bSG_}Bd5yf_AK@G$)Cmd8m!S8
zTH6pM_Byp4uWQ^dr2c0(RJVT-m&T$EwxHQ>2Go|#UbQ{9AF2Q8HLL&nr2dQ5r{etY
zQv2qwvMAht@aJNtE{*l4&U1j8cM!Zte1i1mEVZ6HEkY8`y$?N)y(v9bahAc`HSo&U
zyXI~Nz0Lt!HyE4UXg%{<enzc$p`^W6*XJX*?B*7I`yREmck7|#lHiy{R!>q=gy&85
zGOAWEd9g}t*T0!HAC_)goLTTW&u~rfYVq6Ow>|jW_hU7GeXi`Cvfe_WyB<p?u@%TR
zymI=gG<DVsmM@{P?!lWc=|(;K_l{e;R6%l`O2maamB!$sMlmjvj=WaMW7b}HLivi^
zmo;w-<XxRj-|pSA(cR?ZQI#bfSMM<WkhEZ5$8NIiy1_k%$+yo;KJ-{q#YZxts>#A6
zdWFzM@E(5=hS2Q)3_rfhf4yY?mnU@A+WP+ta*zM*a{te;u<P}~{|tLB|7Q^TbKtv_
z#-jc4cHq?{+7o`v|IeVno&W1^_s?0cU!!oJANkL~wERc>lj5ZMW_{<jMP1q-FF!|=
z<jdD&i^a)b$}hEbdt_~EcTRlAjvZ4TD)l<1B~>#pn%sZ8*Cylst+&-IRq;3fUCURQ
zpyMRFd(WZc4ZAB(zV)BFwVJW2>+@RkZ(TpqSHGOKY+?H2ShZ?jU9sqsVl|Z-=LBP#
zPX?V?Tp)2$=5t`i{w>oF{bvx&D$M+q-utEd_08=P587(a-45BDkh~$Np6kW0wYBwS
zUp^&Ng^IQ=sd#_*)_;bU{SP%RZ|(Np&-XLP|L;<fhQm*fMR?raE3!KMGTYi8<wv5@
zyGx>dlil`q2VT~@Tq7P}_GDs_o8zzMh#g+$yKei>PcT`rPTBd#-3R<aKj!|iT>4|b
z$Np<qXR=Na_d54x=be@3WO`&;&lwcQ?iF5cm3P^<sv$7uS^r1r!&ZN<&Cgw)9X5UP
zE2f8QgVayXaQGMVw6@ItR4s2w$x=zjTV4l50$n{oE7n0+uKUC1e=cYB|LvUrm*cv7
z%=5$Ne+rxI|Mjf?i*l@N-9`CUd2jb0g<l+9MM9-P8bDa~A9D9*Eo7nLPW4ZsX?s;S
ze*FC7{)#|XkrVd%{~5CPOT4&HcXj%t8n=(U_omiA+%Hz)_4<%h?b|aGUY=YsGpYLb
z5gE7l&)E6Sp6d8uBr@^K_cw}bd-Gj?%=;y_az*xh-o+nlFML?GaHYqmEz_P^+$t2_
zf9Av5!(M`Y#h;6>ZK%&$&EF}<@lkC3<GF6fAI?8=ztv#Ivi?K+Ec@;qdhMglldZ6I
z&kTvq^4|I1_dMO!cwpV+Z{dG;{S*0jVgJo}k{|wr%kh5{j@o>yB0p$b1j|M@j`~fz
z(kug=wsScrM!aQ^VW<^-WRGaHmEBYR&oIFhx&-mab=mxmUp)JN_+S6x=;~frD7r+*
z-=veraAo-<C(%%clPdEXxQ_1oxNLUHEoPnlS=+8X{j@gk?aesh-_vJU=KQ@haU)Cm
z_O(e>;pQvWc=t<Y7G&Q_oA}%WWyVil13crmb{B*8*?mg&CoZS|XNYtE&)~e?6SaKW
z({uapvGvPe_37k)sLMD2+L*W*u`%(Z=@ng*_qTqBSMQ#o8@h)-joDYh!T<I;Ti@!`
zqB!jz?hpE9UwN+h(SIn)Y<Jy64zcHf8~Z9*5+;clGZk<f9^bicmZ&1Ts`bs)8SG7e
zQa>D<{;0oepTcz=>4W>sXI(%1;pmEr{G&M+P5175DdU@7xOJlUp92%@!UCtoT=VbS
zqxwK!;*abP=SSW0tdUW&(QzOCZnT=a>(1TT!CLN5JZ&2GtlYr*vrp3Du}t`dUCR%g
zZ=ZE_+U)fD2fb|XuKPNR&Ix+?(dhQFM@I#cTTiOz`^=kqDZV1w?aG@RopoPUTX$P?
zCe6Ea&qL&RzDoS7nc@;bGo(FU-rB3$-y<Rt{k<+K`sD2om$Ubt-1zDE@p?|j2l_p7
zd{=A4mqceeACi;b_mQP4ZF=4WcN5R@ob7=v4s3Cp=`So*g8#^`k?+{QHNDkN@vyP?
zqO1$ArQ$j#IPIM}r%-c7`j*M33*6;P!}hJ%`Qh92>Q!n>3ztWoU!l7DmhP2_+8&BU
z+{btHyiHo=@uEO$+589n{F(0VW>KoG%fHzKIWDg>J25}G_W<9<gqp@}>l|J;dL}=9
zGbL5}^weYj{Qfhnoqx|-&iwIRW9N_lZ@m}3*k`tV^ZpCTANEX-GHqMc^=)#z^rX!j
z+@<3}JKP(J9Fp9ut{Xn$mw9Q=Yh$_WNA`m&*M*|8>U4Ho)?2zs@XW57L&7H8zBRqP
z#Nf%yT<GBaK+IzOBAfUJ`n-P}HuRl7vP*3K`Y5;CkGli5c{Z8*p57t5T=uAA@rTQY
z!siLabDw><R`rj?j@wJ6Esk!QP{h$M!EW<}=lQI261%btf_;}R-FH4V_j>J?>+2RK
zKU*Cy{$u*ldj>y}Y(hnn125OOu8UB46dg7#<GR<>DQBi#Iw=#>`!tgApFn|m_{M*y
z>ObYh^Vh!-=Ks3xyw8v9hqh9ey|-@oaBq5g?oyNfWD&8BMK_lOZ8A!zSvs{trO458
zZLPTLh7a$i<}S-D3&^}xx~j4u>eeykF4ZrgGK^nV9(nXIYX0Vy-0jOg)&DMt*zGD3
zJ`G%<ue$!9;p02|*GuYuNvt-F`p@vs_W0kD{{IZlqSu@MGgL1Br~0$;ok^f;-M(7z
z2{tI5^y}A=yX@yX|FQai*ni^Lk^c;AaqYaIO_9sbwZabhsQv9c!H)5d$I0V=g28*!
z;!^F8zHgH=|G{eC`t+kvcHccI!Fw;OSXS#TQ1%c%Yq+d3g<<1)dChZG;TQY5l|)l{
zmL#-EvV;WQj6U@6ag$2n#i<OYx4rgg9&`&V_j`5j{$jz;>IdzmU02m4T->7VTYtD-
zWR21Vm5M!{iOW>?{+lecB~fAKg{ilG%4=}k4D#wJzEq>}>D=6pKOWY_i)PvUs$X#X
zKf~A8_jKR9eQW+xz?JXYoqJKNcWV8tW>p>OlIB=ocsilf?(0^cl(hRgxh4DA>pJy!
zDs23){m^gTpz8<r3H`9mm3h1PkHdybUzhCmdAn_s*{fS?Tz89QgfRYR@Nd7gckzR}
zealzZ9oZHZ#VvX*ug1KRx3lA|Qdy)+wZDXQ$HBMp4r>Az#;#bsC9%MY-_cu2$3s*^
zsjq7>L)g^M?2(z$)u-Z3>(55N`^+tGQe*#P;fGoO84k?nwlS8QFShxSv}MlQw>9>e
zSyI2f_x3tW%1l=EaWdmOulU{GwBVlDkI9d-|1)%b+n@6>c52_ge>&3+JwJTC>gi$o
z8!cz{dA;n+I<@*q!|TFl$%~C2Ot3ra9g#funMZQu5(b+_hcxD8{#A~d47<$jT<SBl
zH=ny<{`SlHnCgIyTNXY2z5Cwp-M7Afx}j6k{M+8@Kf|Hi4>MBI-~2Q9&v1Fs^&3;A
zZXan8?G;&9*vc&VIw)LCTkz}5&$n*>)7^hXAkbCx7&tM4)>mGvldC^5bLoGE>&E{X
z7D~N%J?BT|e+H$r>HisG&i`x4UcCO6@`uGQGyXI1*@0$87AHU#8Zz&Pt>c{hlWTM6
zmh<2>5TKU#^(EWv?CPbza%n7@$^Jb4t@<PXmRs&`J=eXeQ~VMCcz^G&w{xG(Ud|n!
zXxF&-o`1eT)1`at3B5WCbb8iE<V|@z=kdLN92L!n{xfX7^`rW@xg5uJ8}=1{Os}t7
zWAfHK;M3bDUHO}Jg55r@IJF|u>}Pr=tDBo)z#qTAlWJTa%@?fkx>95PSi_`Zk7xPP
zipkzxkItXTHICnqQM2<q|6Y;Ke5p<I9CaL4$8U6s-D5A7SL67|*7!)A_3UHD>pSf?
zym`KD!{*&xw>Vq`I;VO!OjOh}o^;pG$ic1|xw(!yX!==R;Ky-?4_yaJ?-u)BbUm2J
zc}4GW^BHBy+k1K*D_eXOJ#XQ+)Kx-z--+)Vi~<+knLG7A!-=y$R+d*Mf4F_*Qs%bz
z<tt_FGTtWi7R9AGt+0Q;Oun^ZeQU{&+`7{b-^;u*e`aePu;bE~8)BE=e7m38?boYP
z;PvGAL^WxZrw7#Mgm3EasbjSNY*SJ2*EGBSpWbHEdg(ug*Wazx%ik0E@bt{wu#k?q
zaWbqE%XS>7D`L}jZ9c01vs8cQe};+IwWiG8Z}t0}o8%tZhkMHpYc9BZ*K}>~^INjH
zW;TiIB6S%jOx(>nhkyDqqwG7{r{=wXD6;)pc=scDz9rl8RQgX}Q{wU{ExluN#!ObU
ztx7O9;>X$tlP@2$V*JmrV~+fvQ13%Y_0R9@duDNU*}hA^&#gk~!F>MD&|$w#t@Fe0
zM?O}@(FfQ4DD(Dsn(OVc(=(;Z>70ZB-!%r-#|k~)&uN#vek9+%Pvw<a)be&yk5X;_
zO``SsijP2}IRzoV1!F4z1@|3%T683A$rY8!{>#?p-~4H9eC|lni?E)RhdfW(c4~lD
z_I?I$3C1-F0~22z|DWObj{gknCjMu5k+kM&u0vz)qym=C>wiDSR@i@^RsSe{hVh;3
z&L3+ZU;k0N^u_i}-8D|1Bergwa93OKskN}$-6gUQMLS-fc#*wPJYe1=&^kV@XM3_A
z#M&P(%3NDntWtmTS*3!Y=+D$i>@A+>KWd$LTFbfqMo{^cU(?IBSVY(<yxw(gYxN_3
zuFUXR_quDBzkHu5dvTIP*RHe5J`H`Gyh0QEA4;y9eW2>x+S<qVLeW`8u{mPb!Udzx
z^4(Z__u`)$k82kgu;>+eu94M0WM}$e&h|1X)sH(by?L9XWAeEsS>dpVj_igVhs6|x
z8z-<YEs5j5Tsqrc_VstG9&!2fT^lysw)T<|o9nW#ptoFh@$<L?GQnkv@!oT<7Rd|$
zXL!Px3p!t@-Z-?*@K5O0AC?bAKC-^;xz)P;)}w6|3kudTr|TFWJZ6=WekEM9@MHg>
zc+owYx#sPq73<QYcSJ>N+LsrKtG{=vJf73j?;PHFAZcEFu*AykewSkI>P?TkZ#-Ez
zZt7Cklf}_VRvL@;^V>K;%N#-{HZr4?W)J^(eq5yg!@Z_7KfgGCY4)v<>0!63I4?!2
zf10NF{EhaZPPLE($75EXV~->CI^qIX{E@%<Pwx22AM>4dZr^`hZR?)Ckl%unJlEyD
z4+@_l^Tm~eX+lL$$@*I>Z=HI6(dpkQv$wyb>$HCyUlH{}>i5mePnqIYK54<Vt6X@D
zStkZIH5{IH&e7JE-|Op@(uty1o3(k`WSX>3{3(7^FXFj4YeLLR@of{_ZtFg~tnhj1
z-0it>k`Y>;B)0J}&5!zWo_EiuifW(inLO9kZrPhT3wBq(mF_$Iy{M@0*2={hdwsPu
z*4uyJKVHvP<FRkP|D~-d{h>|nonKxbeCp>li8ECFxlH;~b=i;6mk)=n<-D?-?Xm2w
z@U6eUiL7!G3Y@&Tx^iBlt4O%v!uzmw_?~}SBUjx@2QSM8O(kVtik6qR@BP(UY%f$3
zTlv7A>9y3q`48VZA1$9I@N;9vt)RSJyR92;O1!H)|1j26<b<8O{C57n{|qm#+iBH5
z5DNXzaKq^Ek+<x};st)pKX~hWlj=wFzPzooea&OG8=O;^CQ(o+VWHUOF3GlU<)Q|Z
zdHN1z<$2lLkqchraT2K{=c^+BL=NW)%OyE~4Gg*%ad+NC#!cI86FUAg9AGthxvhQe
z8ufoS`#-EU34S)cV3UpiBk?!uAHMIrFO>ArY<s2s4W%u8cQV5|6!}h_0gw7VJk(I}
zI_&Ftju-nluj-iG<@VWjX-&|XiM(499j$+De0BWJlj|F<&#rO#&oFV>`G0%*|FuUv
z-`S@xI?by!tM?G6$}Zh-zkk*9{M32frl<x9dilQkHA5uOwa%!Ls6%l=w$Ap*-||!U
zU1Hy(dtok<G((JbO_OAi<5;~FrLlOkDK}}Q>7Ub|ovP(J>I8lWeEksK?tk42Jg7In
zc^{)~4gbT#UoYkUXZX?e`o;QN|8{r%XPCd~_x+#qecxo(YyF;hT)x?!|9m{tkBt8e
zca-k`XXse+pP^O%!+(bDjejicW`!{SXGmFhs;fTkX#M${^$-029DSo%uk?H3BYP?P
zv|Qoib^QI_ANKVx_>muAUg+`t;!Pc&jXdud7Co8gF3IqC#kz8@ztjJyf2eogcqQTU
z*4o;<#lb6M=j5Do6g*}TQ2VL&^-HcNcIy8bPB7j6w?FA$+P0(VmCdJDCG6e&BrE)S
zc<I&m`MUk~f=!3lW&P=%bsW^beV{%KHrq*v&i~>4@LK=cilSrRp2fzdPc=GpJ9=-U
zg-XH1GpqLS9SGb1M=O_m>-i3cZy)uR*nfMp{*TRnhKa`B{~6Y$|7UPAy?FiIl#erJ
z9CR~HnG}2f=e)mKCUY&ei*9;Rxc|e-FZMtGd{p>Vu{c=%ccJ?qtzMCmeIGUcGwfU*
z{GZ`T*P9h<?e>Dh8`N6-IqNr9{E2-}3ip3lRZ@4WztjEinP2_?8RXYbw_*Ry``T*T
z{=$_%wue~hcI>QF+OlbiZ`tEYg_)KiwtRKTubbnuF2=3n?l6|wC#vYQ{CM{E)nWUt
zUyS?P=k>ErT2N!>@rOTapZY5M_PbqrtD`ija?!TVq)5Hg%+v%nqjN{5*)6%cOJn`K
zDRX8|zB{kWQ(GB#yKTWXJ11zn4K(!4mf!K8;R$ac^mx8P)7usO{~WGE&h8VADcz_1
z!stK4L;LuPtRjICax*|n8p63hn)^(3UA9pqUnlI;#h5i&ku5on{Ea8M+%4W5`L^y=
z*Rib{;mfsxZ_c>7Y}LVo^R#8Uwb<SGT2hSWH2bah-u1z*vE0~OZEEf_%R6T}Hy3`j
zIOf0C^*;mG_rv?YaCiUvmH%;le~sDurSXUVaXz&X^?Uv6r(CgX`K;y5e^eJ`POs|y
zJ9q2S-X;2lg@xK1JQlm^oH=g8z-ZrES6p}UKf~5_(?3doJNVIU>bl!1al#*a?x$bS
zeU`*JWkQqg_4RBaEaq!k3w;aDESViz&;LWs`0%MYizQ9hf4%X$tm{kukMMn<RXs8B
z-!Iqq%70j2WBs4OVaNXW%l<P+&HtJg|M~le{|rspU$ypL|N6=1837&kS9jK*zkm22
zN3GZT(!VwRmW%dE{}X-RUjIT|{O_0hkKO-h-v=#%(bNC^a{sY@xcTxytMAswE1%0)
z@AjXewff=xU#v&}Gt`>@xQG%=i~mMX|MT~c{e$)^tG>ryKk<2%_Qm}V<9?L?Yb>n4
z@B7cB{?OhJ@BcDA`p>X$_MeRZ3}H|IGuYSj|4<yP?w{^{`Th~Br(0XN&WjYOTn}1t
z%BAbTTlL>(dl;`v+OEFPe)v9L6aTjm^;Z%98KQ1(y!^Igck1@d+q<r@z7)|Abbp||
zopZu*{wUKd|5>+|W@mq&<;<#=dhUx(psVOt_YI&DMYR2~uT*aF;mZee7<aivE@XLN
ztj)aLb`zuI_qC@qO_t1wJJWg8Pfvg4BeUzCB`qsTN>$@DLslO<ylT@tM^V$NWh)=n
zsz=T6u3mny&_KH8n&W=C+3gmq>`~Lok6Uvr=FD!5);Imn!1D1P+tM!?=&@t^Vfr5t
z>u-IVeh5CcXZjI;NNhrVc66-kx@^Pt9M7CJE@gd11*$(!mndineqWV*IB4>w)j#SF
z8m(-btMS=$b$y)sl`rAJ54`1BYTQ4hbw279Iwd_<J$y^wr(3>f=k_gVaH~k1BY309
zI_y14GFuKyHM45H7GE_TO_tQpRtu+v8y(Hh6^^-g=C<PY{%J|o^Ic^<UofccHAc?<
zQBmRTU+xOm-_S|elK%Ux5X0H8w*HT!gC5>EmU(q$>8)$~&pp-Qx&3|ChKhQxe@8Cr
z<e2~L%@<m1XSlQB$v<_qEgFmV9r-2{=&7)%OG5)PC)CA2)D&5us|e$4<Wc7;-nWb5
zqxJ}iar`(Lw@2nbgV(A53@u)l|5cs;&rr+%VO`aPqH_1OEk`adW|#bKX{EgeMSAf+
zL2dt^ukHVSu|KrwTab0#UiY0RFUF?-Q3{*?`C9$&7oatgol*7YBmXnJU-(a?T9f_y
z3jD_Orz-sVD)(I7HTcK*a{I&gnU}3xd{p+{^1{_Gt$YOUTShp?y)wI2eLLfvNQ~Hl
z*W0%5QGBQy|68*Dq2{#HzoV=3{+-Z&lJifdvbn4_<698_`oMPyucb`eJ4#mXH4fdl
z?aG(iJ8nsLTb~GvNOLj>yx7RIq?9FT5C6ma2khIz_DlU!m{BM5!%ukW$)#r#k}tds
ze>16)_50D5pV9@qQxBeC+qVTJ1lH^RsC`%+{^RM#RG!I+_L~Y%ywACJ)oad0p<@A)
zegC=px|YuWfCvH6C-b)~oBSxBwMOq_eXot~qlru9d4E&~T(#f)uH?nLuvZbfPo^>!
zIR;K)co6biQ~c{IyYzTeBSh*Csb>CXuucEZu&@0`;F~4#af|Ofd9iLYQe5=^a9L~r
ze9eD`?-%|vaM@02eto5S&Ro%Js9OIsa7<nP=j;4`zvLgfZ<<hB)n7b$(XR6-dhH(y
zZTZhIZ}WeK`_6wfAmLe55*~@77b&{sPC0&kWqZ!A>ouxQlt5d2)5DI6DFr2j=^H$s
z_1mQGcl=X&Rn>cEp46o;_c?B7?b<e3_ScUl&-G8LwO_vZE*-y7Cu`lcH+THJ+GL?A
zvc&YW|6BRr=6{So`giUzd?Xuv@K)Wzwb^p&8@BG>JIC0wP%tdEr@Z7=mw~m$`VIS$
zBSHIgoz0K9Yx+OJD<<X7K4w)kXX5=c-&Tb@nJIiMVAke8oqu>A*fKv{-<Dfr`><C}
zR@!x|Uc#pB>ZkjQ#c~%+?%6E-Rc-rg)O@^3>OVuXUY<$zjxWAW+PsmI#jabmZrs&m
z{<UJ_qjb50*I#?z^w`?FU`zbOR@cNoGFJCH^#9Z^{;}@6|Gq0VhU?vzEPd9S@-5)J
z^vh}HvwMPk4a0aVUVlHK7~W7EHxW6lcn_PXM@`>DHD_E#9_$S~$g#c2V#=fy3mt6>
zSUqFAlb<hnuXZ8ORpcbLwSYFOzfJqkFsXD&{f*}j?f-g|ygKsndh7H@>TiXU7iP_R
z-l=P1?ay_xT5kTril*-#ZacEKI_vai>i8>iEp#Z;GP=PqDLWtC2>+kE3jZ@CrT@Dh
z!1SNtu*nbie@*`x1nfUN{?Y!QVe+b_+utmIU~T>T^b5T|g{*I%o!@fj_($#|+$z7>
zgsZvVY`b{iyyK+>KQbTfmwi27@SpO_eM(ohuzmP8dHuTS`J$UnieAb3l{)oEM(+eZ
z--Ahh(Zyzb7Zy}LKV07zXYxV)uywBU(MOk$T<d;N8-CP`>4NXK@3EE(!nz(s%C>Cs
z+HNRts(VWti#d;jZTLUQ<%jIQi2rE(&(JP^#Qx9RINwKs#p%Cg|6Pi5xixR;s#`C<
z&C}YjX?pm+X}_N89$nJ6am#08Plcd2jAcn|Yz3_4{~4J6X;u6_7N=azFZN>Eg_qy0
zg0jkHO<g)UQfX&tScCE%pUdlHBhzCW9yE1NJ<tBw|EKBvTkHQl+y7TFi{aNg*tjM!
z^uEMx*SE&={yQaSzyG4C<d)vRn&?Lz3-@ggF*^9HV%|5SmwO}28J@E=H#c|vs@fN_
z??1z2=a1k2_SGr>XSlNNU4w1-KN0@FA@v>dKWzUqywNs~`8)l~e+I$Y=C}XsKYsbo
zkhr_=*Zr~`d?zPcN2J*;R@=L1Py69}rdLbqf5eLaY7w{_S5ftIX&3X#T~Q~cUQYXb
zY{8kbXeJ5vW5R{yw!RDz{~21D|1*TjKVtvKU;mryQ1{_k^AGFy)-i3``f+~e*0tub
z%|T6{j>qhaUeP~Q$Xj|{fM}^gPxFa17CFQ1AHpAP+y0R`>Z9Du^^wspr|(J;E7#2Y
zX1&{cs^^)<dXpD~`Cc$uGtqi=Zv8Gd&HIA(Pj=TR{b$&*|KFsw4EhK6|J2woyZ=q`
zw*L%Qs@>LyuGjiu{+~fr{PFg`eP#E5^=DjtwS@uH52gZUd>X%H`=7~cmH!>L=lsv`
z%6WOX>ErdCbwWSlfBW9JS8@8tFZ-4^>-Q+Cm~D@G`X#L3yi(lC68EW7BO6Lp9nbL<
zcD=06%>St#<^H#?PWV5=m3F@sy<613Z4j3Ws$Ec_a!mD>IqUBV>*DK@438QtZ@=$}
z-}G<!qb(n8pZD1b#r~M6{Aqf`{P0Wn-l$#L?Q-<{<4k4A7al9V<af(&I^XvEa{8s3
z)DQLEvwgNtyL8N3H`uFnt>VP)lOnD$Iqqp`sWD8RDni-!mrPXbT{3a1pkrE^1q0ub
zS!<VOU9);OpY`AD>iiWEF?v7p)}}@#O{@)_GQoYv0)D;^%a7^btUpjbeR`eL$K!|m
z`D5>WEVi4z`sK9q2Im~g+*#X-3bN0Cl^4vMr~kL#KJ#z3%f9x|i|fOWPKwGBTf6Vx
zOV-&dE0)OXbnc8`3ksP~P_cf!_EUWcw~2M9<+modgs|?3IHGaPb5oN4BTo~?ues3a
zQqZarg1Bb!>KNC|-?D#Pcg%BXyT>nHF)cH7V$+h0eVTjH($pIkFnQWAaIRmy`YCVP
zHR<;^o<Gvvdc@Oq&p-W-Izfk(+B^mKgjPsK$Z;r4aD340Ab0;i126Z7_5Zr!oBuO>
zsLlS*a9q3fvi|-wS;6=B{dWBDT)y}3o@wR1*=o}TcQm=)RtXY4_9XS;lqZ4**S-EH
z`Tk+L<A>`9|I}Y$yZ)b{_kw-;`PTO3hvV7j|8BVTpW)*!h65j_AIWdn$GCkSKhKZl
z53jc`nA0}RRy%ORwnz?%*n<-33=h&5JP4E9`qBL0KdZkB>_l!?WFM_k@5oE=jf-4(
zX{$uuJd@u>y=Rn@98NN;_e_kv|Ht_Lq4U4gK)YfO@~;2SAiYjH>PPNFdB#8WKSB#1
zuIJ9!C%a>3e|TGE$yzJv@Tqp^dN=lDaqBr)*rYH!$f^%;9tx^Id%6F&u>H4R)6uJx
zg^!nhbZ^;be&fe>uMcMHAIB;^KD;S2XkB}GqR6+)Yfqh8dhOjahQKM?xQqn?p71m|
z*nB_utG!FlKBKVE;^WkRQ{}d=cD2~q9ky}%_U$e^1&t3YR7=mUS(43tAjs{$@Ps3t
zk^dPse{U~+srSCbUwnD5$&9Ucjoy}UmK?dVMyKSr+0MB$(;Y-}*MB<XsIog^vD*rf
z)okDRf5gnw`_HgQ_#^Xw2If7s{~1n4tzG#i=%e1|<*^ZQf0#dhJv#S_FV98O{|xOL
zUL5A(GtGN&f!+GI;mse@qj&DT_Q$n)b-Ke=IbM%vSJ&z3Zq;6Gxx{;~hw_P2K9jT-
zZLa%RvyxeOo@KmL{mC`|uGW7y{m<}{JG}BCzhqV&(}hPD_t^fn{_))N(xdW{t=sp0
z_~4qJbf*4^?%huHJJ$st>q;My%70V-+b#Y_zti-?=2h3`Kk5zQ)qC|!@}1)LNtayv
z&t}Lz;&aNo)%So+-mHj${mK<0F0}xS#elHrr={Qj%#!cl|EBn({J*8;rC(!fj^-}R
zs#Ezp`Jcj#7q@C|ty@wP`Do_CjoY-J-I4g?TdU}D$3f3YfKgecaTY_r-xZtShxhyB
z?7I(tsF-w3*Zr7O$%dEdhxKOJ{#_mU^27bRdtW;qR`5Kb6w3POVXAJ|g)Mt_U%vnN
z=>7&1-|WNpl=ga>>PnS-IPz^mY~0c=&C<%ZM}&;e?XWyqGs*bc?u{RRE8BlqXQxpA
zp#ML^w!r@k+~5DLJbr}#kKliXZ>Ja@Y`xzVXMbhOe+HS@ilFmHHb)*^mDzSKHvaJE
z8)tJ<wfeV8r(}y&mKjZOb~ax3=tkmZ!wUQ8k8*n--){+*Q=jDdzV+Fz31@AWT)T9V
zVS7-U>$F=prb%k=aOyhf$UNuz!umsh|1+#A1f>m@`oEhtEdI#1`lIn<|AzdxHlmMm
z#eFV3y0UEVzUw-duBm=b^my!gdRkeUdMgXVl1UaP*uK*@Gajx<`(U^9Blm%qOJ969
zn>K6izV%C9UGS_~lc<}z;&xn4?w*IOJ7fB0udY5M|6$^PhGhLes`fYQZ{5CK{>^?S
zxVJvvtTsEZivcv6p?$W_?2pcMJG~#_;UCT}?>^sdqMPo$>(W-~iZ3_brOdK?oN;sN
z%|<8Da|XOsp`g2<h{xfRYvev0@3&ztUHKwv*~Eu8Z4AwJu6=dk)wfx9-dBZv4&J$^
zQ|jWHtBl+QtB&{;giM_Eb(RKrc5=<%I@z$tZoB(kc5mKrb*5o%@rtOuR|NO3J~GMe
zoNe>^Yg;rH@G(EMKm0cPpzx2=0U!BxCa!xer*<tkd|s@vZ#8$twi}T*JFc0}OZ&Yo
cu~Yx*i33N=zaZ}JAy`L|QlHULt^EI+0I$RPGynhq
literal 0
HcmV?d00001
diff --git a/5-NLP/16-RNN/images/rnn-anatomy.png b/5-NLP/16-RNN/images/rnn-anatomy.png
new file mode 100644
index 0000000000000000000000000000000000000000..937ccdd65423b9d4202a44d0e995a539aa8f6c74
GIT binary patch
literal 16735
zcmeAS@N?(olHy`uVBq!ia0y~yV12~Ez<Pj%nSp`fgsGwp0|Ns~x}&cn1H;CC?mvmF
z3=9m6#X;^)4C~IxykuZt;4JWnEM{OZIsn3q8Yxk?85kI(1AIbU-3xpeVBr7%{|pQa
zot>Q?9v+7eANE)*KLKQLNswPKL-ZE$3k;qc7#I{7JY5_^DsH{Gd%Eb+4cP;UDqXf0
zmfEIyRo_`z*SvD3POsH+HT?<h9;*BkyyC1AlnYhf9b?{Ab3f}???;}0AKxo4Je|P;
zvl1t5<1KdR(W!Pd#XS`VU(|hktt~&RO<+$#$^IYn+vDf98|>ZwLq{Yo?p9h&&9V06
z_`Pd?oRxa<>Z_1V`7!nQfIlgpUzo4gpF88@`mXc><9$Dx9$k-%KU0zMw5NXe<{xht
zFOQ42GJNd*Il$+=PTiI*D;8=S|NC;~ThqnUlIttZR2)q{)-1`NX0s*d*!fQZbF6;Y
z&a9V_sNeYGEpzwHQ2TFx7xry?9%)>6IsR?(wI3M|Z~mSydb#_k`ofwg@Aig-)SFIO
zw)MHqw#)Kz29Hl`h(8VgEO76YZ2qUc0X07l#J=#9tPm63Z}BUj;!Te~Us^#(P29hm
zr&}M@Uzwj8^Y3tf^P}(f=Z|l)Nj{c-R#HY(ey!)T^lpEt?CFb*|K(o!bL!Ok2KVzn
zlA`Y!^@mITeq{eDW?JLC+Se^#-;{3VyffqD<Yztd`}x0Lzg}5Tp?ZGC$K4xk`n%S3
z`ImY3tL$I&qo4QU9>d3%pZfg~i|7wq&Ud)ozVg^~)n~t3YeH+H+-AmT)$Lb1_1bvf
z6gT_y?UU7tT@3waetfNUa=Y8dNYSFIeLt2+zCS3w>BZj7zYUYq>pxGC+%IJsv_@3^
z{T9pUn!hupkA|mrFF*dB^WfpMKUMed?+Q%%vi#-JN8Ov9?R&r74f(C2{>}eWi{$=W
z*Y9iWEBh5+x9-RFw2x;c<Mpz#jxINgt^Cu|ubx*J@yF!f>j%F=eT$EU*GC`8Uv%VP
zrcK?k|2DCIRL+_0HF*45dSS2Re7PE<(7Nka@*A(6`Qa&3S(W)dr7-PSc)jD5ee*Qt
zsh|7tWx)@fb3bZ6-VXhD`lV#NZq}q97n(Lcs(jRAuk+7+kG|^OiYbnBYHDlLxBmTg
z-^=*u<z<rb_Rd;+A9-w=E^J)4|EjdSUc7aj@I3p+`5JZQ5ru^{$KIEJ^%wD9o&M*1
zc6a_9CeysUg?b|X>(l<cyK?Z8CSTc8e^LMSX*E;-oPOAIUq6elt9%M$?A(uEox2`O
z%HLN^f0P^+W&1c^v#$KniA9!=)qCr|?Yi7Ax!=3)?wpU6f7>H=U4A4Pe<$S6Kc9UW
z$FBDs@R@tgcXj%o-lO63O;x|%HNN~qH}TlzbCUI^C#IYY^)`8&{eE_M`mc-kB<IJ~
z*3{%iZeDfv)3LjWv74H|o%>-Tb>eYB>Bp~<_ak;b|90^5ElK(OV!w6Tbu|W$H~*UW
zd8YB-jJG}VSJwxZ`F>n~R`ULyr+<$u(_TCG<HgX~n?7G(eNrs&!pB$764+~^s@YW@
zE%!Cvx9j?!ABRj&ZhF;~a?#wUKkvgeeU*IeX=Z1Cl$2-O__q1VqneNM?h{hk=kLtB
z{`Too_0{b)GQo3F96xT!I#$2tUibHGZvAs2|Lg9~PBJ^qzg}KmUL!K({h|$X&c?*d
zxOVo(i>UWUUmWVOFYKRiYvaM+<$-@EeY!A7QvSaElbDa0pWPH5Up~6rk7MCh<GL@k
z_0NtSU0r@;d4<YH*IloVPG8<<vaXJSU*P`L{LPiS^zO6&zWpIJ^!U%6{NAUlwbm@Z
z8l7O9yQX~A`}LFN|Kz=}&)l^xcIM`%SGO0-s@_(QeI*MioFwMx$P}$I`&+6~u{>*W
z#f!p^VWwZg*Hu7d(Z~k-NTCI+_vAurUTsYaI6C_y&#E6cFJ+&d(DmJGaYca5mt{Z9
zcV|?t+xRGO4=S?JkQ3qWVn5pa$nm2_90o#2E~pS$kAYx17gPk;bcJIeh_N6^4Y{M*
z7zku1L{g;xkthao{0Nog?{fc$hTt+FD?onw^SOII%Z>-WNIv*?^5}oAbgVWbyQLX5
zK+wVm!nGl2iniQQni}*n_2c@@H{Z;DB<b;SvrUeu@loT3N5?-0M6DO`f8TxnP~65K
z8(aN<)g5a^{r7kKJKYWW_+{6RHGzL-KYCgG@vNYHoz^@%ar>Vy{-l1JelzCdm;FEb
zZ9m>O<G*+2cg)8xdwzVZ{;+?_|7qIu>cr>od*5=u-aYU1)Hw0~`+DMC?oZMe@$c_G
zKWq1c<93aU&L7>Mzr!kOhNEb_#ym&S{b4rEns%TNninUYFWavhe)RI=y_2<%8n1XH
z-j%L#G(CRn%T*Otfq(ixxSp0edb#t_<qxjTaUV+-{dltO@=j6z>s|NfGKa1aId9N)
z{@q@QnU7wEeDwS=>xrnpdDng2?00cDH0Je*%x}~8jQd!!<VV(t`_uP4dg=8s^GDYc
zQUCC+{&hB$?w`7r8y|H)xnA@<LzjHG_s>_aQo5EOKYIM6zo`HJ?*2*T{Q_+PHB<hC
zeB@d1<L|TAQ6G60{m}Y3^^v5@$J9^UM~!<PP5#q&`;o@HbE5mL?=1+hVb-iG519E#
z((B{P=imLc=JAQy|NFCS^`n;wAIoDF`Ft$d_(OjFKL5RsUM77M2Kn~#svkdl+od(;
z<%!q7wv#@3`R1eT(d$mGjo#3;{PNNLTJsLRS#9`O(&yt<-`B#gE7}77{M!}~8F4Gz
z`F)pv{Hx>K)@76G#OGhtzdo_<-xm9$ms=lw{*`50?#8iK->zo$zudaxM|nj2!@KyO
zM*aIOyWBx@-a8TbuKd*(UzEOkCCig=`s;x$HTo8ldK5+W-}A3OdU^Mw%fZTVAHQt+
z(X9CYw8zIUJAb^ey7l#J*wM?UAGtrSIOSe`G|<LYx9<DS;62H4AHQt>@%C`q>9ZB_
z`_{O1rE?se|0`$J^rP%~-_4I|$9?Py-}6XxeFcR1M>&44`u|_McC2<ml|WL1!CYK%
zbTL#vh8)O(`h<@+d$%7C-1BI6?4PwF>no1(Fa7bqt$zEDmm=$btgk=!VgBO!57*b%
zS{>E?w{ZWy|9@ZqsDGKSzCQ8G`SU)n@6W6ER5_|$_gY^4pLhNJKl}gxb@^C-=U=j)
z{alF4|2;f9Kl*CTqKeg0N4M|!;d?{f%+x;g&&%b(dmh~m{PTW!nBsixf8Wb=vqaYa
zkY4sxet+3~h)#(2|DV5iH+B1|Xhw;=_79~jv2z`tCp72X7VTkYC}T@dO=mcEa)bT<
z56T+^CJDOTc)!oul11jHV@_wUgUuts{)!{YavwX~Z5}ZT*59g?W8ZF@pQ8G2%k95*
zKMIe$mFRH)v*b~$&y}pWz<vJ<FCUA@Gg0~Xd{^wH16SYXt=*Frw9~Y7*B|S%KD+g2
z#cQ}-d8P1g?a})^%hEnRbvw@Z(O&e4{kun}@B8{rpS|Mef@Cdmf&b?obx*o?P-Wk(
zjBGJ^(@Fn#bcxT3i#PAnuwRiR|FNT7Al~`5M8`F0)wGX0oo<HiSyP|#YS;Cv`69_z
zl>YS|6`y`4@(J(7tM7zz7fomSzFzC1S&xnW?74gYJv+5NyG;D;sR?hrPe0qywPfey
z^}D|XKAq7~pZw8t($sr>Z|COS+IeKzsVJUDkAKzIXPw-2cV+sCjm849r@~VHHkzNg
zs_;+nW9g&`UxnAEin;Z_UUpiqPwY|hQu}cJi8C+H>i@P-pua-=<4f6V)r)-Z>-IeQ
z9Ji+G<J%=^(eh`$e4jF3LH=tG=cD@(HILWC|Jh}EWWtZd%@tq21fJW`So3t&fqw5S
zkoWt=9({ghztsKXE88WDeqL?V{Qp{AcKQ{CdAp=GSKLg|cI$Ip@4U7mb$eFpd54YX
zzn(JCTj^-?=<K7ZE4?iZihSQ5J^t$2DYn;*&#dQb>{kwFcC>lKeq?#rqs6!9>{?p!
zwM*gFqkiVhv)^|tVf)x&-X*^6k?ZoVGa7Lvr><}7dXb~B*3)g9!6U(ZA$iUJ2K?Ij
zPt9lL#VOmplTlimcjOuKqp7<&I^6&4dE^=?^Ep#7Zk2n8*M(KL4;<T)mcV^P`Jd&Z
z>`%8HKfbMw^n0{8B}J;g;>d5I`qiGnuOI(DvR(c~fl&N}s|t1!AGaSmxqH(6>W`bA
z-tx3j4nNYavQJ0r-mIg?->&?9V&V0^X#NcfavwK$?@#-@GP36Rw>8~AKCl1RqrKYE
z=F#6X$G7i#xY)C1yWIVSMsIelTyl!x<K!;?J4e<FCGV_FH~P1&@on(x>;6#zo^3x~
z9$CHq)n<|T{2kL{H-6K$kL^D3DCLS*V2AskfVWE)|2VW|_qR~z!koERdTQg<9tpl*
zYIiT{&)-*G*I3^_ZCgCe$N?0-++PDV<s*Cd%r5HBRnI>4`s|J^Y#%%N-_9$Z`fMwY
z(e8<>+g4juPUclN{qF8y^XPA<_*L2YH}+Juhs;y-;raY=*OVoZ%a`4_wZwbPg_a8c
zN3J`UT-sV?p1tdv@w3yPJlQ@LhMG#Q^qzC~{kvTgf<vYM+<fwUyMKU0^m;j$<7*DJ
zSJ*#_R@J+`E8Ta}-G4ng<`%p2w%&ZZ@2VEV*>@%GJ8YkSthF@vNsf5>?fndp%gnpi
zU!Q9u!uD}dq}8UISss7fU)M-#sRTc3C<?pm_9x|$cHhfY5>@B6c3+FC*?i@d{N?*m
z{rSh2n0C1TY4}le^}(AtdAHXrTf665g2$hS?}QR7d_7iMB)TxFKDr;er?#WG>96e^
z^X|*36Mlbv;P~y+HnWe}CX$chzrBu*Q-G8jNCn7pvHj-+`hRQ}T7PfaoX4+)cl#;G
zeZ1Tie&YPcucg<fOMahY`baQeai3y;#gYB3KW5oH68tasNc&TV`=8kz`X`OUn}3uG
z)+>W#dw*zG9#Q^RdnEpZvfRgeNAwkpj@UEhmB<L#OIIE_uW0A|r>C*vSB3n?>-z=z
zf0Vy^{Qsl>5#@h#fBk6fQ14&f|6TNd>N>WM`B#tIOPx8s|3^*DmsPcYn2w}tA33i)
z&u-o>sb3XG{B`Z373bOg>phabdU@CV{x{c`-IcBV^Sb+dn7^p}*L6>;zwEaDRdFQy
zX!`2RBk!N4{g{!}p<>(fNBG5KLHiS`(Y`AC-q(I~jX$m?S5@k0xNH8B{(9|?H@QB3
z`IxU_7i}B6M{#wf%b!b|rGEWb-u3<G*PC79wa#acAL(xGk9qkxKJJetzh#BUeXuR;
zO7|{A=ZBx>`c-j6d7hoz$ByOw?lzBJJ{I(s@2@y=`M9#&$1fi{_`~;DytQjzJHKpZ
z?~l)AyB{~pyT2*h|K(%H{HQ;rwLhNC%UBbuw&YGmz@LA=c4wyDiCtHp_3PK}*~+yx
zkJLr=^FD2J`}ovtw&KU1cbmk&Jvu!<59FJb6*ntCw};hiy(qu=$Nb->achnque)|?
zLBIQ+$eKH`Z+zUh+|w}a4p+&SIokb8T@z%Oh`#>JVioB{*ROAXcbk2e$d3AHo!cD$
zxPE{BGpcaX+4Zl>%2(avxxI5s&D{T|tJVGG`+t<ppCgxl@6x&Ne=eST^gkrB+p#9_
zfBwBX+o&b~@4O03)%<(>`=93>YQoF^epj9Ey??*EkCLLD!N+#482gLQPUd8OnPDFL
z?fWUGnu}kauXSd*n3DAUZd8~4-(N0&v^(zKOWdn|>&xAZfi;(|OH_SaH1~?ktLD_t
z<}+_(Uz_h^FSx(vWAEm=kCnX!QUddLb%guRf7a8z=YOw!-5&caxsQ&W{^4SeX77vm
zA)-Ikan+jotL2J^nSHmJuQ8sJ))D`s=Ff(Tiz(V^eVg9b-i!L^<l8jG+J9cYlco90
z38wEqH+=p6NBr-<ci+zP|5x~T^{D%)D^D++D9zU27qBhN{KRw7`d5-G&6FdLJ8>M*
z5Bzh@>(4K<<$0QMdzFspetg;E8<uLaYyabLyGb_{>IxqvM}I7oXP^6HNASL~fM5T9
z&RVla_RrPJz4Mn?S02&#`?Krk@dF!zzLoS_eRO>;F#DC;nmK_rb~Cp9vDh!g^-*8P
z?$yp$*+*7S{`hKj{W`<&$xp?cFBc`uR+Rh5`!g@I$F1gfT$se8;@v%|@7?4+uGje&
zHM#7)qjKCfzix#i|F5LF?K5-v^SXOITlvd|ii>5kO+Rkj``GUtYWyfZrl#_${fwjH
zZ)MjmkIuWj?WEH5zTd|mX6hc<zvqYcytR=v&%eF-^JqPD(C=z*_20YKchukgS+SMf
zzxUDZuRk_Ew?DJ@WYP31dtN;?KC*w+kKWloGz_QSE?0jX?rb{KKVy?cVeK`|?Y>V1
z_gj2Cwfa{Mw>0~a?dp%iC$IaHay8{qqx-4Ur0)wsvWH^#n25@6Unlid-Sc_Qraz6d
zPGmouulSGkX!P~t(cRO_WwY*uXZ)F>)-4~UuD<DXw!nUyk3qE|r<}u--z~q>eNru3
zXO@r3NoVz##HrU6>M9>)KR@(G^uNqow{^2VyUpAqcs$+w=HdMNQ_Rx29!*!bTemNw
zc)8k@bPjX2kDu)Ri}G4kRuoNk<zN3d{p;)w{+OB&y_F?v@7Z3HF~6ESeQlwb-Rftn
zR-a#a^X^lYRbI{IGmXk@?)kN!ojhB|Urx?NM65ad2)o|C&__iX>+ZdL_bFh$=bx<$
zp1qEGa8KaU8^7|F+but?=gn7{cYH&$ub8cY|KfAka|HfZJ$hZL-hWBKE~swfy)2KK
zYgX}-j(ohfyj^W_a+=%Yhl&i3#5MLk+7avd_(W*)wWvKi{$C89r6aff($fz=n|`Qv
z?YI3=Ts9|q?XtBYDZHKbW%`1d-|nbQejLUv8CNawk+HM<N7DDd{no21-=@Y+|8kI3
zN?*HiX}o{lri<C^&z;kG9*Jw#?KHfQ^797kLN<TNBg%FPA0JO|m$yk5=>O4hMEvpf
zKizE4f(8436dd7}4|e`}VRe&0|BrxKId2zxS(@2*_f;HeSB|UFylL{D_2YU4yIWfy
z7teg8Y|p>w$gzVAkK!Hw?D{R&nA)ZP$0kRkCpr9xvYo`oeXFi<eRdS+|6wpo=i65+
zq2#ibiX-hxcHh6PSZLx?q%?2mEQcf9_1ZD(c$t}wDBB5q+_>MhsO6zid&LoUCA;qt
zUHLb}9|_8LRLnozf1^a=h_YRaT6gWbKj|LXAXAV0julz;{q5Nf_Zo#qfA<A%sAsJ?
zqHM?UG4N4o;+%hX73DrUc6|REHR+x=+ap2wmWunUCB<t$PwsH95qKnge)V71zwAyn
zkDliny?_5uuy(n?BSHD5ivRiIZ@ixf_Wuw#^1EKsGhV5`;z+wfT-Ex5zUp;S9qu&@
zkCbnJ|1y<3v(E#hjN!=fvdlf*KlYslS<>+1kX~=O(EZK+N0jXtKJI+`dz<W$<G&Va
zKN6I0sPI1G_h|7xXGOV>#pW|N<anrm)c%_x@v&om{GO)=t_JR@d(riyFkb$ljMT^M
z)4%=m9&zik9Z{~U`?&Mz^=E2zYh`yS$$k9XJ^k9uo!vh?W@{^gc-wX7lnCor^|e+U
zIj?iCY}K*AJ^x&sYz(Viwk<u>_h{=$p+}uJmAA$Ao(bIZ%}?)9=gq@qMVCVNlvN3R
zJn|;+MqPH_qvX3YTPl*ioye7EFBG4zIFCbr$=l$&eUDo6TzV>!zIiMY{@+-kv_s(9
zYYB_4>v1*${%2$_wmdTLP<IlU@2hO*ZX=j*M7Yae(7#3FpGEHvrjITS8@K(t6m#_T
zx1NfmcN)t?>RoIE=l!@6xaaLR!$+MAheJ{nbhzH#J$9r+{h9dwnI1JQ6-oyt1b5jh
z%;WI?Q>OHh=_AV{L4O6gk4IK_+<#*=QRV;d(?$aRK0+s_JaT7y)Tw!--bbabwSwtm
zhq^#Nj8gh?^JK~WPL-ek>)$-S-=RKD^<TmLx9s=x!9AxxvpcRYf4c3wgW^1n;6G_f
z9|a$EGAPdvs%Zo3e<nKLPsPs7MleAnxXWH?9*2L;jMrBc=FL!Zik-~!@ra4wt6h^j
z)YXJxes0-SXY;7@<PrI0g8G8~ED{U%v2%Sq;xbW1esYKW&vo`kgoRs6HE$N*f8JG*
zw20%&+`Ap>YF$Z6A9qEs>rj8zzxA!IhWp1`pIDCwZ*KnzX6JBzJn|;dR`I<?oVAjh
zjbSy%?U$bJA8)C~2>PFq?|i!XZr-D>>Ozk?e>z9`UE9&w502oU%G>|kx^raxR_3H5
z!l!4>D-m8_zm4moi@@c{?EfEq-6`~_^Qq6Z9i7+v+Xeku3?nyx+;`riUP^HuN79Q~
zRrjCYe{(lGX|WbV+rzV}r<lCXFfBUElw{0(VcF?<700EM6m&cF>(9;U3w4+O$SjyX
z?}^g9M~)|Z<KJzIQ-7qXf2AZgIL+#(p#R4OESD<W-ioe$$oxoi-?S^06Xx&ERhak4
z{qoh)3Fp(>6qM&Za=vNJ|LG+U>yhAlMkmf6tvC|=M`PW5$Fnx|PBu?{oBzc0xZ6Cn
zh0sT5E7bp(dR1}WBX`fa^LG7=W_zSrH|0*|8qf6EtR355CoEW_cR5z$sBuvF-^otr
z><+uyJo+oJe=;AtRLicj3UVLgI{JI<MU$>vuXM6`^jGY?!L<x0-baG(yMI^*hpXqW
z-&rA$ul}#uPb|G%r|eqv<SZs3u<V7a&MO?QbGX?&suhfHIdUfO`akwZg75eLxZ`^}
zG-JKY5#@W*9~)=uRn}TL**w}S7{A3rc1!HnY*1%NaDRE=$@X7yen*t=$$xZZvcH+9
zcx^kIKz~L35%Jvc+kB@I79MB+*zw(?CL#Xhac*&tgC_oPUGEn9hx21cz39hlQ~x|t
zZLT=-TV>yis4Xj>bZb9rsyMR0??*t?*OSY;|IY4k-xL0b_sp$W{SNm%@sGM=KD|FU
zmAl!erQ*nM)qNS&c2+G$<{96W<UY>p<bSfbH+=DZ7n?`6!tvLx@SDa3Pu|V^Nbvpa
zA4w%=m3J=N5!7dWMET#<Bhzj5UKjgL%v>b<NbrBvqtnM;-}*Oe(fngi3UwdV3jhCG
zaQ{yJns?=Vku$Ty+<u14+H2fVasSBqw1|ggrFPx=#b<YYG~H-YQ`GvyUZ{TeshcLh
z41+su1ESe4ZGMuv!&>!$==4`-e+W7Lk?-tJ|7J30uciH$la~W_{WtrNFSE&Ls(jDM
zdhf?iKwVUgeS0h=uYXGAt-qo8PxR>PptH=6;!Uq!3B1=+!)bR*@t^9^^YwQhZRFp%
zT>oa#TbXn|ahE;QA06H)@@dA83+n?mKbvrL>64?+UM!Efy0N+9{*mmg*BRe`rJhyP
zKB;fwaa!%&ta(01l<!4<G_8!Ce%5y#OMk_YYMp(4I-jzS{V5MU=)N2j2x5<<KN-z@
z)xK)adS;LU-F;QtLN@%f`u@4PXzl-Fe=J48iDFr1)RgQ`pNpUW{XDs*%>K{zDt3YX
ziv9dImp?PkdM(8J=zrXw@}jtXn)k(xg#OP}vFCr+w_@AF=ZeC`&lZ1Hwr=>bUpW5m
z#^_IrJeLV?|C$r{XYH=b<s8exwukSLVb6?=kJp;NX5-JQv&ro_mugDta@Ym_mp^)a
z&UEjOs)ad@e{w}~&K|wKe|2rp`kGD0(w-bz+`cAzrQh}x!T(&3KL2_XTk>djYhTsJ
zMwzWodyaN@M%Ay94Sc`i>hoDTd^3Oi+aor;zGSc6oO=TQr$36$VCTJ9rnpjEUVF;*
zuJ=}UC86cN72n)>G(Y$Etr>q=K6m`L{#ZG2j_aD5Z5o<h^D_)HzOClGHS5><I-{S9
z&#BpN+@-M6`_}%gGMhd%p1Y`WErR#B%b&KR;m`BFe7$IQF-`4g>rTx>-}3XD!xt@&
zpBk|JmWj*sOJ29$eMoGtIAX7`kC$_K<)XD`VqO>C54~BLreGKHs<>2Gy>`vd7x`Qt
zJK~k?q!ML?-@eLQC@O#6X@=N+e$(S+$-C<P_p)!iYOU6Bf5VS+H~0Lg-QKAR>fgP3
zx;UngyK*Vdhq8riAEU+QOLp|{-tQyZ6+U0_>D~2uB|oQEXNhMCoZlg}_s16NgHe4y
z*Btizcq;3vpdEK}?J4W#*cFa8kG?*dy6$wz!lzO*yZNs!oyz)+yZwFL!aot`*qSPi
z=y&w@9AV$ge02Gd)#WS~#pmlDnY9|MZR?}fl@*2(uFFLA*$!Xt>w0n7Y0boG?h+q6
zu6OcJ`G3(|`}5B4jcP~Qn|)o*Hs7e|49>AVD)~t8y<5%aRTpkL)U3)ge8GS4YB85=
zgR{+}*pB>@^5RFo&re+|B=2!rqQBzEZlU;IPwD+XEXD5EFHD~LV@^nOMgEa+W_5Sx
zy5o;7%gqXXl(|#sp8ChH3mz?wtGV%h7k}85TMMU5y61VMT4kTl=ZSHD9(;Tw_3?ZB
z-kZFj;hei)j(=bE&_}(_LazSki48>;rv$Qf)UT@8>G#JktNk{=)#KHyKKG+^zni41
zNIsgc9``q7{=y#>jx|;y*><0UEjx6kgc-Xk%6;@sRuAtwU%LABeN&O+#}*#@Rz0)B
zeb3~NACEmczFAw6zhb@kmRYA5K6YI1K7TcQY0Q#YB`RtrZ`Z7RZ(g>M?PCZ3*?Yyl
zUyCK@oQYZ|9CP*D?8`+fB=Z?Rc9g5fnfTW@{e2_5@)O^}D_@TLyH3iRH|tM^(DWG#
zIY0It>8|oxTJ+`J%GI|v|EiXlxJ~@uosNu)?ke)IwE)X4q}Q*Tb7G~Wbj`a@m0Q_&
zetWUC_r-_)w1>`rZg;-tEwtQbx_ZjV=dzJ&3d8hDJyzZH(BFOYs&DSXs`HPdVx)JO
zt@-rhx|{4H!S&+z&skTVG<`IcKeNPJxAXstSbcBilT+*#{J5Te+3io+5p$WvvHQNv
zdo8eb<Bz;mSL=7xXW!npwG%Y5)AXb2+Jnosd9%+fTU+xj!ShdQN%sYxllP?OBp5ab
zeT){(k2_+U5I_C=CJnnhsao3|6CVX9703I2eK|p~j``93)tEy&pi!RvQ!6I&a6J<I
zf9+B8!{w`uVdKUiIndC=RU~b}0{s=I9!!v}*EwbWNbrBwqqW?~L(Nbvpi$?!vYTFH
zMGEx)m@f4H<8|u>_;B=nw?E5MAM=0g`2SRCO`Y|I-*&2x1pl8_^?3X0#okhW`A35P
zXQ$?KnoTd8f7cB%ko{YGUP$wg+)nW=F@pOwKYouWcC>k9uJSMcpd!X#70e0WzkK<t
z;(6`A=ilZhb5=e&T3!?vce{4w!&g1s9+TvcDF55J-2ceY-KXR4via@)?Z3$SR>-Ml
zr8tq(uHG~3PlqpD{BD}SwS4#0zh$-aedF2oSDz5*|Dit5ZerMnf2EJS7auSB5&m|g
zK)n7AgD3^Lz5nv{_WgUw__3q@(aY29^8a>3`)c3h+V50jw{G(Br)Kj$9*U^Z@ZHu>
zaU^=`w3~Z_)|*cV`ClKb*RuOZ@#dH%_xEm-3F{KSl)>HMzNb{Qew|RswON0&gnaLH
z`gcEV3bB8){X*=Pvi+Cy>`$*hSoiJw@7+I*)_e+FxgpE9#(Mn|!Tt)>qvlT@9g7UT
z&HlFDZ|jd&{!Urt3i<h>h5tWi9Q?Ar)vI;hwd#WG>x~shtnF)mw@uGT@+jrJ67CeB
zeP{bh_3PVo9<MvK?;X#P=drc-_Wqe`Ww)rqy{13rT6x&P)%pwzJI+hZ2$Fuh{C`r^
zo)_)2ym$MngNBTkulE)`n|yVZk@(dzk&nmuzMf1db$jHjdZ1)``+=YP1wVGY*Sh(5
zLEl{A`d5#(8Gl@P$;0<{yl7vu%_Bj3){Q?OPx=vYed?rhwdbpY_5?UjxMWxGM}X^N
zN4WI5<p*a?Kk~k`THt)>`N@xpQ!`5&B|di4UwV1E`ceP2N8ev@%ys?vN>=erXTL!I
zkMz#-R`u+{9~bJSwl@c+M19Ok-*YK6_?_q@!T(b(m-{Qm-+k;~{c(5JmYdTj{)qW|
zvzEI(%|Gaf^1sSQ?aGPMj<DY<OI3K9Vrs7#_qJ#K{8weHA3N%KKl1)etUuTC<KJ1P
zj~(^CAFn2CzL>m<^JC`wO%67X;zMdo3yOYdWa-3xiB*>SxL?IiI`_b`r%_YS2d<Os
zKL7jmP5b;^xkr@$-F!6r;4!O@&Br%)xc|}rdVGFZ*L~^Us*~&+(^TwabOh|qt3DE3
zzdCa5yq({)D%bv~`OtjeYqdg$`yUt1>)q$iK04m?qb~i=xxWpbN0jT9Y1yq4k)OSH
zy<x{o=Z_Cp9DiOJFX(9VsPx>lBd2#V9$bB7`C@C=oNaMOl<%$TVc&ATbH5M!m*YXb
z66;m#qtbaF3EJP(mV0s3ee&HG%|)lrM6DCf4z$X5X{$J5|ME*~u7}{;8q>><R@--k
zY-e8NYCY#uf<oP+tBx0Kw|MDyxc}Mg7jG=*V4^)wG`_F>%vM9Yd^g{FQ!;a#s@j$n
z85CTLOFw(zflH=P_-a4d^ERCoN4`%y^+LPz_C$ltkGdCr*;uD^WlNNn=hRA_-^yuv
z#rCW2tNrw{TUv6b&DG`+d*UMVz$bHl%syzdqLFFm*_Ymr{P(_gmGZsWTX7^>wQ}ms
zIX~Ezm=)epyCBsq{#ag7?&CZE__bSp1{_@T<G_+%xhJ&e+eQe@=gk+qm<ehoeEh<C
zefOr#H;dxEE1heWrDf0l_jFO9W4+CzTC3Xi;R1-cGFa*1Zu97VT+M_przJmj)MtNe
z{+T-O8<Q|vQTHngyG|rSz|46~6-Ph?Cl{K2w1O5?YJ$}B_sgZ0Izmc5WkmTFtkJPN
zK)~P8=8<NHyAA8dj^zq+A4QHRBbg48?CpLapimc_dS21t&oZx?M-mEk!Oy=oCpY}y
zo$4TG!O?L)sOPzbLS3|BoL7_0BlFTI&QBfpLwcsOR36FKK31$Cuz#(D{{;6x+mdV?
z<v!l-`UsNHIaaKKOF{#egbqmJy~+u0P30rX|5TIqOjd|bS;F(NqrO$*y~>H`j{W~%
zv3~6Me^lcAqlAc#{=Gad9rceT-YcDm?)bj>ZO;$$zK?oMpxFeaH~%-6=Vz{Z6(!NO
zzU;ifjyvK?CJtMq61GS&oV)o(^|_(bf)$z-kACM*^Y*hdjd$Ykym4;xa_gh%9qxZt
z9SJ_BTxa{}the1)SDQ!Eh2&?1zd7AjaYSGBU+8}mSg<1nwX)ntedT{yi(XvHHsQIf
zVP|c({n6pY*_-3~w(pf+Hdo>3s^>f(JK{rXBEB2{&fBkH-@Yh+qiJirs-4Nx?aMzd
zzLsse_JeQ!!+?@6ncp7FPUp1)xnj9(rE1>gx$M2SCJW8?d)1veYj3z~dE9)9MuGV+
ziXU^DsvTMHS#$5=>6RZcH8*Semj1Ax_;syuW9X#*j{Jl6``B$Cg{#=TxvYJp-6<>M
zx;@v&`rlh8T~yj<x6S*{xy}16L5-+R{nPK)?+?HHHB0A+c>M8YpET^ZKRRu7_GpK2
z>WYqj{`&PRb??2On{jmAquczI$3&Dn;wS#tW;S_S>TBQRn5z#?7j4&gwBP3ajZpKf
zb-UN><CL&_wru6|=hJ?C2wMI#v~Kh7O;+<2<vyP7;J@WIYlCj?-*r!Pzes+p)DP>p
z?RD|P-t1d(j772i)2FTqE?#?cyZy_SiX-~UcJI4h?9n{@_nn!UK=~x;O2G-e`p#$n
zT#YQ+)W3bRK2zl0XX|cjINCg#E~sC^%3Bv7a_(sFlKbxtT;IM^`RcoStOvyMk8FKp
zvVrfBp#AJ00h4m<elJ>jspFWFuHEJ{e6q&U!uOr{kGyU8|6yWdM|^L^Rp(_Zn|>_%
zt122T8F|tF3G<qIF?kd1oFkROc7Lvkeq1fMe_~AV#hghuzrM+;y#DgR6U%EqO4lx%
zyFqj6efvGj86Sl!$1&zeh3S0U_*`iI$)qa#qBAKjH8=jLDc9dQES)W|U+UvZ5zYC0
zJC8i~`}l9hk1P8aGCtnDxoE=djnCwdDA##@JgD-qS+yu{`;&~1cXg(8xc_lJB5uC8
ze)gZ)KMH(m6241bcy!y#|7Xnhzlt61f0`bJ{tNt5_GUq7Im`4P>w<)%_A`I%h<B?g
z+5h9?#g?5{AIEl{e>=~r!~M_h?(J<3nIC_h)O{rQzvAP`JCCAIN5<DIsJ^rMf>q~_
z_Ky4K_4d7ux+AV%wejYo$Dp+8@#k7*iQPxvqiOer^{Y1iJYPH8;*p?zc(m_Mw_9Jg
ziae^#USsh{@c)c&KO~R*4|)C0&5QHnq9Ef;^$zzx-W~IUY|52?7MXrX29>8_HLs*<
z4EO(d7a6>+t$n`QeI5n5kHww+??v{{yfizOFa7S$ZQXKqm-s)f7TI6O`+SB<-TRDW
z8T%coas6|>=KreGII`Yt&*tLaDj!e2NIg`1#~|Zl*jd~9SJn)I`=vgfTeolW5%cVp
z%W5I7I=5>%&tC3#q4&qtj_)yX%RBvdtv*tA>EPCiY=<5HuSAC*S?|7Q9>bg0;*XBj
zm^S;GI=-KoWvxE*Yqpc=!K@wbpcJ)Vu(f7}+kV0Or)1al<=m1y`9?ipuXIPqnS}S!
z&sIDBaeCyv=jv_ybwzE<I3IPNJ>~9w*v9;8#)JjB2WQucq(>QleX{eX(yDca?r9lL
zf07=p4h;DCv*3lA_&lY$aKF1HKNqIwpZrm<<4gKwzf}K|Yj#Gse0*47BKSyf|C}EV
zUoXwov)f=MCHB6?-Q(8l#dE&9EmM&D$lVd{99w?*X!-40etDxedwUZjYJTid4CsiT
z`s2cl#|Fzfzc1f&NNsz2g~{&|)sLR85V$v^;m6Y>+t=U7UM{m^F0*x^oysDnDJJoC
z+l_Pk?af!6{#0@Q?c>wyd`zF8+4}R5Pqob>agq0xN7ttQ`Tc#xui$(8s%+gpg>G2P
z*?s%@>7SDIQb*Q%+0;wnsD_Zr!~Ju9_+IhvsW?)<=EoJqf1S=-1^q!)pT6$ADBef*
zVKsiXav!%K_Y;sbBbZIlKENO8F7*_f4)qRq8%LW*oR0+k1^nX_{_GYyBHUErB;e2S
zh||$Vu|u8ZV~fH(0fZn}AudU<j*jQ+9dPT1D8VomER3)Vw-f3n|95TqGvUATJO#Or
zEFU2r_%r!vbxRKmoDcO-e??Qp5#b{!N<n@?@-tQ?(BR;J(0G(UrG%R>Eczq;+<(hy
zf&L#?ReBbI%GCK%{|h?b->eg!zn|^jUw!t~4(Ib9fvV({o{L}DeKbFE^T7W7u?uEr
z6r>(e{ug}oJ*X9;EcbD~hMlX;qvzjO3+<os!)Dj#gbw#V{Ey1@Rp(vvJhI+<PkFDN
z+#|vL3w}J=dZfNW;D61>=p)Mi9v_W&@#}E^v%I^1`aLG_3M8m`<xg8Gj;#0ClP`1x
zR3G;T%Pm)w`zWvY@BKrGy@wNoj_jXSasSF<pWFOJ-HSVujwdYp@U6C_Y(n0?_nqJM
z1Gg@}-&rod{6^$-o{t^#RsJpe^rUs!yPYpq`%V2Sw(M2g<ge!^8SO1TXu_8J(ed_P
z$4-BCIp*J2A0E!CFTA$;)QLBO{XbqlI_<b@`~PbZ{#o5u?tVTaf9Tp8`SfW&%DUgj
zcbcDjJUjmTOb44s*&lCjcxfR2V&3PKKm6u!gic+iyxjYK-rpla_j4!uiprl{`sv5c
zyEo?>$$h+h)V#)MvG&oYV#jP-RODZ^YI5C=>Eu7ccKIm#xx+1wUQ9pz^uyOr^|eR#
zPx|3h9I3qhmF7{gr52a9Q?iSemEV?F!CDj;EBpSptZ4p`dsWY8&%AbC{$oe@qsO0~
zZQUqW_Nhy^D{YeWhbvks-bLG4?_ISL_1UZ7qx;b@_~(vWkN!`Iwd@V9QTV<s{CV`o
z^&igf{u1u+hr3fhPc?hh*LD}NeY^cOwf&8)3_ZF=ek*s8u3c{SpM}K^UN1!QXYE}n
zoOJoF?UQ_?v%HVuJ!|e<=zRTk$&aY81za~BLSFZpY~LtW;(cVR!VJBAcW=AtnLXN-
zbZ_wl_dnX5_st&7=C@XiJMrqo(<Da)|JEx}7rMSbUAz8g>yI_R=ex|Y-S|gS@{!<w
z>5nrfEk1g@=%~wtUe(adK=Iczvp=6L`Pdn;=d;YOzu#v(XZyIm<oPQ5jE`Q|{360~
zb*;la*LB55{;*#DN<u*2fBxJZ6+Q2`K~Y|}+<uCe@m0G=O5xLPBpuvV|96G9(Wm?M
zi+<cF?zq1q>94JnhTwmZk4Y)tzngsA7<B8$RsP`Gy$_y<$lv@uOYNUZbbHyn*B{tE
zcF2p|pFcly*N+g}SzTMM*5=FVUcRpSvG!4M>5nhE%icTx`Q3T`>*0vy6|v=lk4j6Q
z#~)d)49YTfbsrZUOA1x7THWFPC%yB&)xlM)8-w?(dvG3H6p9Pn7rkue_c`I?UhD89
z%603mos3}ix#d6Y$CB?eJWikaTD<7@(aTnyKaO|xALW^r)fio4wY;MA*iV~Ar$5R^
ztd~Eln^5s}hG|8!VtvQ|l#eGD{K(P!xN3Kr-+rkh^?QEodE2ty%3eY4qr8^gn<|?n
zf`1<i{%84kyS;1uvGCf|xQaKvn;I&P=-;zi8CH2<?&^7`AK1oqb^iD+Z2yPF%dy6x
zdQW)oy3%uto578;%5@ci8-INh=>MU5q`Qx)xxBmoW$J^Il?%?vJrexS@Nu`|$BPSo
zysdeZe|`3ks@>VW=d1ZaO_Duwf{(=h&$xFeRW<YV-cW)6^B%dbt$87_NU+G*;ODWw
zkIqKhq`Us9?iPRg{lRJbPYVO<AKqR3Z%M^}+j*On{;fTFdL}EI^pnz3+w01^m6!gO
z`^fl6aQ}oK9;SZp?yQ}#Zh6SrP{rb_<xP@DKn-}IN8dLcusdlsGiQOL-6XwG@#AZ4
zMBj<lJJ~#%F0lUD_256&^BkAWQup6<)<*d2+=zKx->*Few|X3E7TW~yPhaC1vVGM?
zHNn>_W`@oFZ+KCrR{izQW#{5M-2cpZbac-9&UJgQWdC$~Q+|1#%Gd5iMv=e5qyKa+
zl8jP6s_>QPV~2c(_WlzolaEY)vf4ayUWsvv>syJhY0GuuDk5_PPpiHaJO0X*WA2Z<
zEjeq{IAjF(7oI!aP;un;qu=Wt{zlgPmRojXpX=Lu7pB};n*Vct_WQLKU#k~P3du|Q
zzwSr+$AyJG?v6H(-iyzl(HOV$=YikNPj~-xIzIdSs#Sh<%PXn_yxJ;`oLBz0|JytD
zorRBi4)1Uf`_8g_!}O5Wy-&369oV>6#ME&TvyZCyhQ-`RCd*hKwrAMIIHASmk{rYS
z^yHb$K{}rtUmtB>zx)5M&l5YvFKS4ey<ywE?O#z!`RV#U6&pXc-}FB$`9#uQ<nil+
zf4qOE1&6o)(*HWY<;IU!Q~&*YEZVca?(X$Oo=-jU9Uu3HozK4e@#o7EWjEtr@B12Y
z^l;ns+mHVjnLg%!|2X`Z@xQE^yYF89TAE$)_Zxry{~Y^&x&?b`EkAkH@BhcscfVaA
zTcUsU$I}8_Q+kfCeZSUl-bZD#eF-*?-B0`H=jX3gIA3wB|N4(*4}y3fyVo!MC4bDg
zF6dZ&XKv8y#6L4Kzli=nQ*q3HmDJDK)$4lX&G)<SwfTQ3{P2&7n`iTX+W!A--M?SA
z4aBab+34*qx$9pdd#>qL&+$~peG)h3arPWHW&3&1r!mP!@6rB?9RHuXA2Y6dQnR+T
zUb^W+&+(-PpS@nE_4C_%{lgXedya2Ou#q@lVKDDwg3aTD6~~%C_OL&e<UeLC*OLyH
zI6y!iqLBeYVbPB)!Hg#V@4She1uu#U_)ZK7#m;_)$w?2D4=gz`&5qfp^U=}t2F;^C
zk{&xBn37Zz_(Q;8sYJX;lg%=b$J~x`le<2)D)eU_t#FjP-1V`^W|`<?)=D4kW89x+
zMD&P1F<dL5e<C@`FmA%}ElGP6`aAI`tbeR9e|Nxv@1l?MWuENc`){wm^!>n|{e=_$
z@A&BXPKbGX$H(owpXAN!78Z5QKJw#s)z`wu#>XD38UEYm`0wzs^OF72H-3C#|I^2P
ztU`S6jGm9jDtrohzRM<B9aFFT_`v#5Ne{aOzkytW&4G%Bj|qPYFNFr~vD10XKKXb)
z|FVPsmsbYbEYp1)URiQOaNGKAp1=3be)~T17hmDZkDFIi?|65^chkCUo3AdsUvbmo
z#n!}{YpZoDMN9R=_glZH(vIK$$Cc~bj;4=|SGv76MeY9m<H3O=D%s!fyq{MP@Uv&Z
z<{u|Ff0~zlFYogit9v{j1Rqa6@ng-DO3|Lpr$5{XxKz5}S#3hirS<xj{paWVJ)S>n
z@xFxS)4lbas&`E8=yjbx_)|bKzQkwEEvfn2=T~KQ*mu;`^!V?($Ma#M_H~W?>F#gb
z-(NqTaqfyu%?2Su-|&Zy_v>!SewcCm=f3KWdXM>+r+tjP{NwKKpI<c3u07Ily7xTO
z<H<(zLZ&ZYt+4S7AD3FY-to7R6YiaTemnczs|8k^YtOYzSoE)2NP2(OoypVr-^69q
zEpL?zH`>S1?Dq4%bItiz`x&?9ul}1j^}^;~Q=1oB+DG!P)mND0)|Vd_a%NR_LiP5C
zMUM}2UQJ!8CgDBV%9BM?_4eNh`m=xjPxw>2^ux1|p8DLUzs}h+9kkqOCp)F#^Rf2n
z>rxM$Wd6D7!>^4t2A=({iw*Y9@44Ik?BxZMy0CdxUteCa({AXO*uS#<)%3KA6-TVM
z?sAJg&aXXH{rSharu%qI1OJB93Qd=ikJ}}e(<`&O?b{odMIAl$Z@GM8{eRY6`mrHM
z>X`cUbvthSP}(=6$M@{(6`L&uPjy?=xG$aG-}BM)$9C14&HDR1Vp3|}mcNX#IlfEg
z_GjTuuE*||$12U)ll0EWj{DT-t1Au9i)0t9{8*aB;bl5|Th-Csz4kw+EU9GPnppEy
zUA4-*=XhvOeP>D6)~GAx5%2Hk|9Lj$<3$B|>G=P<bZm^T^&DS%?0Vd^ovQtA_nrS(
zWYv87^JmXHGdugM?32x>^!KiJK3}$>UeEB@<D$pf-A!`M({CJEo;EpkY4xp$jXN^`
z6xx43ExX*kXZ`$H;)XrPpB@trT#zrl-zKT%R@7o^qkm62=9sYb_4v=4Yt?i7YL7ni
z5uF!zvid)smp!I_)bsKAkg!LZ0UJL$s<waJvFW3uWL@$1d4_f;ySb13SfyKWIk86W
zyOx1mxY@s_dOLPsF7JsK7w`Qz_0#ET9}`vgXa2b)-g&x5zuWYx?4NzffBs$i{PJB!
z>8G63)T^fFzv|7a&Ayx(bNfg0jH#)%$A2uzu6Y-~np3ZKy6@w|a&pHWvmFz!Zrt6+
z{mwx*+JE={-DiIM&HQ}Hz%I-Ggll=s%^!Qud==^4v-<RcT6V3B{XZS1J4)o3@jg!P
zE%Us*vyl0*_(R+Iz9qH0Y=xh#O)o!prDFTXtEX9xx%ZyT+x+O(vG@NM<{jy6S<R)l
z_E5ryqT|)-@6YdBU0{`6qZ#|R&iHRZ?BnvF>yvdK<vo0~_t^fD(3rl>8Itzh=ccsI
ztlQD~aqGP9<l+TU#_Qkj?s;#{{z}h6U+UQL#lL)b<*u96txgk)x7fF3OK;j*yCeG3
zTaSf5cf6Nfk|!<y-_%nls5x7D|I{qeHBY4Zu1ww5t#R~D>*PhxF4|uIw9__xdgjtu
zk<Nv4S$DL4Ol(x^*M2+uS>=Q`Zoh4!c552gMXY~Tv2t0|8@23fS)0CzyjQrkWS@IX
z&Y#%im9N%tWPg4-`IYmhyRkQP;y6t!mj10u`+3qLXsyS!UCdYaJl(><_EV);v|C$Z
z{!!bg&6{qg?sxiQWTCN^{o&EXWv>z?e}1@gQ)hAG{}Z!47k^gD39GZ!vE6xFRAuG#
zMgCWF3tS&RRtOa7@4v`&ZS%UVt@3tDZ(s6~lbC-sn>*ic-^`@R>poq}y5+grv|_){
zN9U}<#F*1RT$5kEj!du4yO;XNj=%Mo`d+K*XxaD2d*_#ys4`ENm~X=VWKVW~Q)Ke{
zzBS*BOzdj@SVR^pPq_5s`L{3qcUlBuuK(D*=wrU;^2D=mXMD=i=<o6WBsbao+131t
z9oELj+9$7#o4(II=HQRe%S(ec`Toc#KE3y_<LR}(Ll=J9IO*G#IiAVYTbBGW%+J>k
z)!eW6$7JU8(w+(OR&uYiYp#1v5PxeMxm~=U|MZImdYs#rAG>)seDe1<7iZa(z0vDF
zY4G|+il*i4qO)p;0z9pP4!?O5R?%~Of^y~cDN8QvhW8v#IN7zX?5nYOchB(+&3_h7
zkY$;B-TGMalrzyA(`xJ7vyUb3crv%#b9&Elk4SSTpG%9Mw|tb)JNxU&8u#;@j}5;>
z1asHyGy9n?IbTFpr?#;|JyIq+YU}Lx$9}j@|M-2M!=A}I>~1bLJ6XXqYlq!Vy{Z>~
zWF_^lK1kk}CcNy|)AGzeJs)@6yYW+s=SO;YddTX76FjA?PtBRnf4qWc)&jekUw28R
z*s%E)+}`S+JL#i@MQN?kOY@WSKT4Q|))w7ef8=Zh&+JupGrw#9FqBK~Fs;p=+;jYb
z=IhDbpC>y?^cxne@K3XSpJXJ*{5)X(o81hWj}5ybK0DuM^^oijJThxf{%_sKh9#Sh
zbzSK>zGJ0TVfEXrvlTpZvjo41A4?WGmi~Tb;=u}@c}K0Ccpn?qO!}VFccg-6{+{}t
zbQ`zgmsvf>U1ps8I71}MbVgFo`%I(9h9yGZV+=K~y??dZ^K!{c<HHY}ee+%h`Dk9*
zqxb8&dZXj!)xP^wYM(nWGCwyscr|P947SgQAI$q2bDBZIamCM?AMaP6XP(Sq`u%By
z{<VZRPf``*YK4Lq+<WU&D-@g%XnN($qhC+D{z~MoxVij$#c_{?a>*%jJZ}PPjRapA
zx*fWEZqctN#*e|QL$&{VK1!$uPA=H+{{FcNOSxn|({{I?zNWwIf7KY6T?;fmyXN~A
zfrJ?*W!Bnv%8st^l)9Td^Y%Ik)3e8z{!W%Kk5oRY@=M(??b%xGt9v$BupjfbKC<Su
zdr_m@Vjk1^(a#UmT{(aD^RXKzqF2_&U00Xv-*|Crf_v=z6q{qmb}jxD=U#m*d5Yn`
zec5;FlWg3qzE^U(GB7YK1)U3UY`<M;%J1-F$vtN-f4Lm|`r>nw+DTakwi5iun33Bq
z438OM0-zQnyiJ27o?vr9WwW9Pyje#%<HPZf^(*#;98FyI)B<Fhr>mdKI;Vst0D@{!
A&j0`b
literal 0
HcmV?d00001
diff --git a/5-NLP/16-RNN/images/rnn.png b/5-NLP/16-RNN/images/rnn.png
new file mode 100644
index 0000000000000000000000000000000000000000..160b03036d0872cc48f35733708862a2226ffba4
GIT binary patch
literal 18140
zcmeAS@N?(olHy`uVBq!ia0y~y;7(&;V0q2L%)r2~Z(}S20|Ns~x}&cn1H;CC?mvmF
z3=9m6#X;^)4C~IxykuZt;4JWnEM{OZIsn3q8Yxk?85kI(1AIbU-3xp`y21GW|NjgO
z44s{w?(Xh~4<AlGrgaEpa7mC~FoXTR+7n@34Gatl44y8IAr-gY+;Pmi?Z({pkiqr4
zt(VzeM%Ti~1<PhIdld2|xY;%6{JybO<K!&G_HEsI7L)DHzxqEr<uQMy`(ck?pFqw7
zTS$ob@%`TiWd@C7@pUW=OMkr0w`V@UyGLaI_Byr(=?cw%Is1PpGUPpI&4@m(`G76%
zRN}2|eZdEK_slrFrF0Hc-h-nu%ho;?DPY!rW&q|tm01RsUvT(JA=i$^>BTdw{20wY
z6!8Uv^`A?8Wx;yq0JoKo?D7Wb3d{DDV9n2|uUT9u&z|)FZ_H*!^8yp;0JhT%8qKK;
zsRxW2j3U?^Bz2fIn5QvLIcU^i^g}a(Er3mj87`k7372O)&2S3D-N3Ve2c+sW!>I<i
zDPT=7Q(&s5F-`$#0;z&&B5d?DMvw!+MsMH&IS_1gb1DPafgq!y?gbeQ;=+ukk$aH5
zx}hhPAr)#}ThbO)1K|v_S-ctB(z4C|2z5?nJ8Q<1vCSmIY{AX{)6?#URz_!<fh8B*
zoVE7mhisGGlaQ3m%9Xf~lX)TMa6_7Ra~kukwKv-s&rYL>g0pNOgV{`?H}f*j(&e4C
z_U3|{mloV?NXy=q=&*UI<7QK7lV~Xu=?ggsAE3xHp1qcHm;vrYh)EgS(xllC`WD=T
z`N(0j>1_kB4G<qel!GjWdYP6A$nx7+gcCv0Db0o)60>q85DtchnJFwNA<99a0QE9d
z!BR(PSkqJidc18*Yer2>Nbv(rnV7y|IIFk$=G&wHes?E2Y&P9JA#MLH86Hr2TxzmA
zw_bwH<oA+&E(>qh+h>$*I~!+lYZmWL)89{)r={P1^DRN*!kZm8bGKi8GiTP_x8HyN
z+tr)su-W}=oYk#avgXysfBL82-Eg)UBKGa-oHX^>v7zU^{TJVy5WM-Y_7C&3=hol6
zdijw57E|e>tZzFdj@Nyxe|_oA%(U>on~twcbFMqPZq?mO`ddt^*WFy5D&Lo3_T<K!
zhgatuUuJ2xJMtUXlA9A!H_vXpKYzi^;<Iw^9Ln1m&-O)a_<Y0Ua@u;e(=y-JPhT{%
z-~HbHdo$|%;-+0+{{PNWyZ5)2{BKBees(slI%ihq=63CXDNm1Fcfb1l>#WcF)yfyM
z-9KN)x9^?(kA^*I&hyU7*%|D;`sUB9*j~Ae_xJXF&cF4>?xXj?`7>(#AH22w6@Azt
z^6b4{`3&>r+Zo<nuAe^pdU)R5FWz2#-w$uMQ~!6s`xk%4H}zS2yC?ms;r@_plP2-J
z;N<fUi-QBt&i?mp;;QDv1uN3lyVozA9sB6@;;`rLt4m*9y>olMM-AV<9~ar5m{#w*
z>7Up<`^$EB-m`J{>z>|zantKl_&NC-mH(APXYtN7+ubLhaX+rf)^FC`C3TuL7tQ|7
zJMg?e?ri2a>$7pK&U1}#_gY(<R7*VAbV%jQ-7l@*Qjh=JyPp3_&WSBIL;slDR<P~r
z{l}Jn$U^>-zpuOL`t)_rE`3;;w!YcQ{QB!t1uI`JlK=RtCSu-t>)S7yrySnxGwbn-
zl_xKMt*||}|8nqV#p`L`6PvHU%<q?)C3|Ou{G&}<PR6hNHS@n7@BFMXS(EC&Ws#D8
z=W>5k|9TyGv;Fk7gEy;A{Zzd7^y6ElErvWISKqvtzw6fryT1aayE*l)%=VnIKK#RO
z&$GY3@3RZo{CxIZmjC<rpB3EqXJ1U}t|N!1yx1PPd!5(5y_@G(TP(UcA$#-T`d3l$
z_r=4G#|M^VeY^bg?Vji5-;SMkGuNN~eTq-qIo^DRecx|<u{Ybz>A%<RM{enl6<Rfk
zb}WCsEm%_dqkZpg&#XIH%m!8qZtx42UD#%_MC(QxquB-qmc0zK&uwC6Tl0Hy_UW)7
z)urc*fBl&BuJK-hPP}LM)5i6@ZnGxc-25QCqC6o<?xf9ctBG;)Q{u{VQ@<SiskQUc
zR=bZ9W}8o^{m<gh6|H-HlXZT>XV*<HWOJY0`g?a7=b!c7<$VUxQmb!<{x063A6T;d
zrp}&N#VDW7e2;9oN0)cD_g!`|nUnZ!k?p_R1r4wJLSO&TzaetAPu!P_`_`FhY6)dk
z%MaD2d|UPSh|R*peed>di?n%}T&BNy_6uYE89p`3)ATp5{^E7?P{I<;nX+q|+V1aC
zR!{3r;r+5NyU?`mlA6uj)n4U&2D_)<{K$1|+22e#*&AwxyGu$9D>c%y4K5uy6DL=9
zK74L^!bP?}7vlr=MCx2yvSq>9H_B<|_QD=++p_=f-E?#JrQ<0zOmX6RPDasE`)`(h
z>$tlsJN`mln)~xN(|p-y|Gjkh%?FM1Ug6^A-Cw*6H!sbcHTQzRACcPm>x4J&UayyT
zsl6!d%)Tpcw5qjA3opH?{1%b*JuB>?w7}-28)x0Urzm#+((j+AYwIt)dE_i<=rGkg
zmibKI^aa0X#!mls-0$4}nYzBW-n`A+yg2@|Z{|&tWsNt*XVmUsn|Y@!_^aXX-;7tE
zZhL&IU{fh0)M}0Zv+A%L7W>bY^KZ$iT6uH!oi}eiH`kjrRo<C*EAQFCge9J{Wc3}t
z8}HV7(?54juW|K?oB6rE`kQa-?wWV`;F_lWIR|qxqtdotSD2l1{_o|q>vN^k)2E-k
zGp}Ig&DAx#Z@ca~>T&W-RbYj5q!ifB{pzWum);z{_N)KrWa-YE`WY6d-&8HQ$#~v7
z&HKOq=E?bIZO_=vIeU~_W%HlcHk;3zS6d!=^KEVZ$=vT}rm5w9JDm8}WL^8sgWmj;
zwohDY_}gal^X9X5dnOqFm_9?-_s*NF1vd}=-dk_{JKkisRob`pb!qkQv*x|AneZge
zJ~J!rKEL7b-=-WlXXnnaz4PX`@b6XgOnz^0*reNYzuRhQ=S*AaoNwP}XDnJfXQu6q
zH)RI971Qb${|%^E)^_vKkuXbZneiQ$BdrIb(!#Ur7%}p=+-SKkWkGySmY4v-*
zl~=wxYr>}Nd;HDUkG8-3X8+ynuu1FA>-M(KXY=AW*h!~-yPo>!P@OWHvhSHU-$E15
z3T)ngvF>HvGB)1lvt{>3<=7s06aR2i=+DVF6P9Ssxckq}d}(cH^~}Ge?uY)aG{3xH
z;qsfIzqh8^U3v3isi}0Dcd*&+JpborhtKfcyEX5vc~jfwv-7svPg^fNFY%kzj7bMQ
zCYxH<ZM~7yaq}T}>fIo-D{nqry!c3}#D8=7N$d7)Y|6e@-^9K?n6u#KMepnxwrAdG
z?YX+_M`l&y&A)cOGj)5fyz%{={Bu`g{bupj@Q8&EeKybjz+4bM>kRAbo2Bc!C!Cpf
zEVC^3&Ynk1n`a+*^XZX=RMm{w)bic-bJEN9>hh@V>O7LLq;uw8@4UOQ|Dw~r?{=Jh
z^KROE`R2~}gOX=HmGf_29s0es?StZ~zx%hFN@QB4UBCD;COFD$Z-VXaO*vKv-l*No
zU(Z>8N2hAWU6%Tq>FcL#zukXFy2nIzbNNL#g`?u1rk$Nvp7&$U+}nn~Wj5cR?YgO9
zBlkKFJ5}q|WwT;+AN`-J*RIRY@3fZ3ZC20C&o3%JzYzcXxcv9h_M5xgBfhowxv&2&
zz0GKM|IP0A-<_A-o_M`o-1^+&;CG8|{@$PR^Uk|tv+5Y$#Gm*V5_3Ckn~Brr*D3q=
zTK(CWH*NXZb%*WmrD#80UDp5Bbo1{SbKkF&u=xG4UhTKK*x7r3|LFSui))@A&TaZM
zT+Sd`YW~fy)oF+Sv2A{v$bSB<=jQT*KOgAb+q0hQoalY;^DPFuQ@&~0{atJQJz8gW
zZ2A7{YHWM=+*jJXG;rqLUsZRLziCIy#=ZF=H@9*6nSYZWe*Mh<XYb2j3^(>qsNLVV
zIrZ}9GjU6`#p<p8C;pU5`C~Wh{?BJ??*8w%c}e1|-aeiUo%>AJ`^`)L|L4>G(mmG~
zbnQR4f5F22H`m`ie>7#DeC4uddtdLd`Jew_J2&s8DQDu!-fWrk-%kAOy1ums&D{NE
zX2u%*ep?z_HN)`hpBu#!-bP=K@BS~i=33s1iOW*HZK^R7|9ALPt#W;zPVYJYb+e3q
z&-qvP%i&w;-T(Z1?BCim?y}Q;ef~G+eYv|Wn!cyrJb(8+AQf!w%M+Jg%nwd6u3lpp
zEj9OMe*awc1mE`O$LBTY+}|~?p6Snmy>}Kh-khXo`1_0btL^>kAN1e;SO4JSXRar=
zi=V3>xZM6?{rQZ9ZA<>2$T2aPeN$-j#REB>XVSJE^f=?jyjj&yn)l4KBROf(Jx6j(
z45EcL&&)NLG~;Ye^TZjwY-!#}W}A84z=B8`!GcL<QZsZrZ=Nxe=)8$!-0Yid${;gA
z!WjwMRML!ZC#a+?ZiEUWtbz(7>_T<{*fL}%fNaY(01G1=&+7&fhB+c5!ECcC$R>o5
z#?hz>wt-!OkSEG}2>VnG2e<Fy6*_OCggMBk&^SZ{1iF8qafr}p4D%&TBMY7YVYXqU
zm^0H5Nf6}xvpE>v>uuY6Q}~4bgd;g=#-Ox&Hs|I41T(1_@T78W>EbdTl{DjOt-E@8
zGAx}p&&)fMbb4;Hf@U|*;x}0vf1D1SFy*F0+r_%YMItIs%`DiG-+Vj#M!MZGD2-Y2
z*p26<jn)vh+l{NsZ%$jQ^3-fXVoU69M)#nDZ0?CQci)}eSE!(pYBnLUruOcOy5|la
zr*aeyzu5K5==P?mOYX24^{mVGpW>x;>w!hj+S~b?^JTa^XY!^<MB7_$zhfyf$w>Ny
z!KQU}XSU_2YHn}jQGR_(taDP|jRzh*t8X{oH0Ntua)!;QXLXtOCu??RkfYqLhGuU|
z)%M)Un<5c<{oYUImn(Stj?UP{duCsu1=t}zt?#exDTc6JVq4Rqw@sRI+o4Uc^sVZ*
zNq#pTWb_nR^ffI}n7(U<Zl=}FD@SuYC!buZm*zcd8kg9XnYx_3MiQPG1`?hb6ApSf
zHS@R$Nr7;R!K4cc%@bSLKv=Rz#pFbS%EE&QDhnOilnHD3!e3p!W$LzVt&BSke29Dd
z;NCmFcNh8YG34q;=Uidm@O)3hyuVD_p4)tz$aT`|&J~9+ChmRzS8ZF&98<hbKT(ZA
z$|cc-g?EvGQWIONprA>|p$-m+5SOx2h5%gX-U*e-=Eb?|I}B#uJR>HUW<2|*(wTeb
zdF*;S{*<N7KUUVwvAH~%b&1=)Te?bT(#$rywt$K+j?HTiEVR>2>t1k>%OKii#@U>-
zZh_Mv=7j{-8BmoSsUT*?1}2!nM>c_&CJ~Hj-5i@|p3M<ViUct?u3>-~i(o<=q&Q79
zVH<=AaT~-4urm*V5^+mJSi?;v2=gY`eGpq=OdDO8!(eW#h&m&NWUS&e(GOc@=sM|W
zAIK3*G}}BA#C(ux7|p_qFgYUZz~!WE98#zbovGWzhOFyAj-X>R+`SMxFv17oyw#Vj
zEiDtbEidgzJ3qbRd$Q!FEwlY+8_vFICcJs8<Yrr4kPmP5+&r^wCdl3I<@&1QBw=Fr
zik>dFo_9L$%)PIv9(QwckL9G@_BnGc4dmyUXJPy_v(4q_-_)+-0fjVFO_~{qqnf<U
zEE&v7%Qj9kzMYl?;b1pp=GmOtoi{;l0qI_Qb_SRawky}zFdF1&5NE68=9#(1hO@zu
zkgJ9;nQRG6qd|_?Hgg8p4Y0rf2N2YI*V4{}!9xiw0aE343=||&L2(P#IU7^~n{A%C
zHU$)BVKcv;v8(+5^3M@aFl?3FENt-mug2{3vOJJyZuQ*Uo%bzs+nKK$&mPGEdFuU^
zn}^q&Si`-gdLx@*b-*SGXHAeL?J3_BxQ(;-#x(C=#G5opKXT4Ok<ByDzF{`nZLT$Q
z?%b_0YcF>^|FUxW8>MprT{q99ePf(i8>MFb`<m#@^BMj8yHh56IBTY@H=kww^UKR8
zGkDJUU40X{JT08f@OXR7rcW0?&ANHFV{b*_=Z_${OK<)yYWDZ@({tl}yJF%g!AGwd
zr>_aOjOd$r*OASYect58o0_w1d2Vet&-`t(JHGJMuLf_kGfA@hXT<*3v5xu3pOR44
zl-BR{d4lKZwEh`;UzyCl8Jn`@<kzw^<LdS^bu;zbrysp#Qf+wlUxi$zz~rKnhZcq$
z=H@)JZuY6{*^-ho=FYkKca_-W&C|5E)_DH<9d+SNfwviF+l~G2x2KeCpFDAmne>^o
zZ+tUrue^D6U2VP3=i=9wo$b!Qo2tG_c=JTQpuOu&t<{(}2OC#M>@QB9`%I+2tnfe@
zzt;M*>Sx|%&9b$&w~XAwIQ8ZkzjJRqr<I1kbu4YM{aLj^^>w(O^*6WVwDO{J8!Aq}
ztWGnwj@+CcE4oVl_uh}3({m&WHIBY9)m#4L{bMPE*`QE(b?MEE!=GEc#Xht0ZGN2^
zWS1_ngY9Ha+JDQ7?aw)<>pVMTT-{q(cA=j6`!B=UH}hxz?Ynupb$?pf@|#~|8PBAB
zTNu|o$6)r&+ouyQm+KkMzPUT?+l-rYr|Tqt3p-<{%)I&gfj7o!v(t@d-#p`Y@J-@x
z`TbFq8w@wk+&lG)d9FP;K;Nay^0~LYJFhoW_O+Z;+I92Wm1diT&8sJrRxGmZ`C!Uk
zo%&4@lsD7Xr<Uoz{^0Uk<D2;I<^SwXWT%zqehW7;u+H5qF3)-EkM@P^c$Tip8jl?*
z?vZ;Qa%_IReSY=d8FP2tbdN00_@2a{6!gDmg-5)>!8gCs&Yw8&=E&K&I-WnJ;u-y$
zc7!WV{{QEVg}&8{y~i{UgA&s1H%AMW%r<_T_~d2fDOKsC(WPnE?=ea5-I8dwx%|`{
z$7`~;{+{TcQgpH{a*L<n=GUp49!fHAp1C(T?Ya7`*afmbCkE-=ROoIq+kCxpX1On0
zntI+hh1Yy<6Ysct?Mj)P5iT@i?=jaC54<f-9(ZGNmT%w9Eu~vN>+hS_%YW}p-Imbw
zZ&muA&z(H*X3N>S@`;P@{*qQ|54`BNJLO_q(3!Z+zadh3Uph(-e~|av;%<2U<z>A{
z!zo!7a*5xn4o@sSu6!o#+qs#t3vAzI-9F+_{dUq*{(s-=-yPn+f7buIxi60&-T!~a
z|I^O@^v~PhbJ4z;x#YU}YrVsv8+ra$t+KhPvs?SGgx;Ot9ewwOO?JQd%+G!$?aV#z
z&E08vwN^sKRr`LX&3Zrg_o^gq^UL3-nO#=lyES|E-#N3FcilACKD*EV-kV1~4-YeL
zmE8RO(wn4Zw(t1fzFs>^R_e>9nQ~#d%WiKoyR35i&DAQG&6{s8-*ofn%aW^ac0W0L
zFX!OoeB-qDOK(~(eOs@WU3l_y&h9kv-7iy(YvZ@h+_cRsSpQVOs+l(#Z%uz1x^h<S
z<b5}few|`^!EfK$8FQaqd3E;u!k9(1cVusE)}37&eb2{S+x&8EwCd}l5?-bzyDjSX
z>C6r?wf4UMCc!G`zsAhE*@o69S6;n6U)JWNQhm$i+m9!1)}8X^{c{^Vx&I48Hr<{&
z`{lyDlYD2*&&W^u_O?cH8voxAqu+T;Z~DKSExV|;#v%9E8<yo^rgraZuiTt|ZO$>v
zi}#kKa8InWZQHs-Yxb8c<GWiUHgzA{9+{W)&AYnpXXvl5dZ(A|G5Vdze}*rAqs<bt
z<g%*Z)lzBtyMD&DXY_AR{kBNr`_i*tHmpk7o;xk)Qi;aaoj1>C*|X_Z{`|FN_m$V>
zzZ1$DzWtbRtLNrP-?gu5)?Zq(xUsCT>}PIu+W%{>s^b>VKCrOpWF5=#n_*sNnH7Js
zlD>IwzxUk#n_9+&O*bE|pSR`ad$z@2{kD8xa(2etTUS=~-E7W#_l<kOtlaZ&?yKAt
zoBeX*3b&mp|M?`cm!@>jmfV+n^v!+W%IWqucORIt>83q%@Fla|4%IatxyRmISr)c+
z-~JMj-?K0NX9#WC9WpC-nf}p`2Rq%iPk+iKrdrL>k-a>nIqmrRw6uD?IFWN_{{_nB
zi*tY1e)00i*>m|nf0qU19(!|SS=d$sxw7<ckC>~E-8QcNH6uK2-%LAkwNSIm$shMe
zY293Lv9%>FjeXgfv;QJZGH$;4&-!1xbn|rMw>J~FN^ZV9E9&Z@9rqXB-21Hjx9J@p
zd#~A=6V_~+Stru7HDpuW;>Y2}o0q3<*5?jBm1fV{J$=v3)BcBK<wIs0TCZIhTDy6F
zX~_1#$L!fVX6k)@Ykc45@})rYJ3fuExl`9%{J47IP5Yvy&(q?k>Up-MvG1?`m8QRL
z`5f7>*@o6DSBC0tX183tT()-Fw{2(t&AXlvC6j(y_4mJ>Uy`*|XXo5}^ZU=fiTbUY
zshi)|Ppk8}`^NEa<F|iXBsWi<6}2|AjQxB6m+~`jPJfshv}5M7*23JWZ&sWP&{}o#
zvi!qbm4!FMUH#uT-qdeD`TB5{ahkg6)>Zl6t{;^@7q#y8g3tVle?O;b^=-a-Hbf@<
zwpL_)`58Sur?4~sg8po;HP{{hTj>eZrnmEM?wxdVZ})TAFNL$y;#);^a+f`_sAfou
zZ+raq!kdJz-wwWUd|O@mZTZ<5bFW-k_0ROg=G+;7>n8InbVhH=e0jXUShQkm=qAUb
zZ@Nq48jhI$*7~zCrR?MB_suuoZ%g@RcDv`Mr|;UVnYCL?c3(eevs?Vk=4$C$#ni)-
z{_4#AXd`<3X3@!a%Z?UqyZZUfBZbOc*LL?x&H5`bFXFymclU(U&GO5Rx}Ml;P#yB!
zHum@Bv@?ELS)r%jc-}j9_|5uq*;%%?UTpraoU`nAg7MSiH(!A~eRsy)^SjKdqh`d$
zw5*vG8`E-!d$YRiti8&)$KEVi7WO~m+k?Yx`r?auMfQt%tgT-p`*738;0-fBJ$|$L
z>=gO*=)L(*)|h0zJ@ZCq&x;CcT|K9;vwC_?@A_}HUrjrmZP*K23jlJ;*ZwsA)Au)+
zbbNejxK%i&e0Ki)&yS}s{8kvQs(nA`|DN8W({Z`W{#e`&JK6ToyC-mY+I0K;CuLD!
zm+8oFzMKP&yQsvnOfkckpK>QO2Jd}(_}lf}Uv92RVGQdI|Lffqm%A)x{WdSLb7AU{
z%hUer?CpGJ_Tz2*%(?%+{Wd;tx5g~FEGT$&y~*$IKXZQ`*}nI-Yn6Ps<<;|@QmMK3
z3<C9To`1C5wRhU_o2y^#wf$LW7Isq0d1c%Oru4eMh1V9|>_6(SpY}vtQGa!Q%^7{C
zcfB_^|IIT_Q@7n3m09*}qlsJYdu{%{Q_DXb{JZh+pEv(9*5344JSjGH{k_$@Y|ibw
z<pVad?WuF}{vB@%{~f=%!+P`o*?WH}?0@QdB0TN6xZ?MNZ{p0~H{Z#cZD{>+<<+ds
zn{WOJ*<F6|$M603=e5?Xu>1D)Q~zV8PkCP7mj0Uf{QJ?$bFa^Cu`%rm-Q-!RAzBl2
z?c)2!!1T2Gb#?Mjwph%qOD(Ho+h_iM<<D*RL2<e~Z1uC7S9_*+#cT%^fu}RSU4L3^
z{m;I-eA1n5=Z-MOY&u<SsrTMi<a<}_u0}RJS<Njq44Nky&seZe_*C>py@{EvY0HX6
z7xRBHJLf%mzxl#l`zGEAQ|g~>Xr1u)cVhLi%D%{*Gd_KOv(iZ{<NoJvqfg<BtglYG
zSzmuEj$x+0)3h^s$NHDt^luiD{{1(!<T6j%<abe#!e3Uod~;e<G|lw*@ihf{Q_l+4
zsx~-IZu{jK|8VCdv8mmf&tFeT6Ib6L{Z;IO_Q(0>^R`v(=`G#-d40T@_5WWYXWzMh
zc6=%IxvzZI!JJF6+N*E=U6uLm=9zO+hUZ_ceKp<7J?+YpUH!VtdLw;qR&JBo^v9<9
z(7leEQ%>n!)_+&6JA1E;blP*>37c+KUT;}+b9&YN^#|1|c4e|{_AFofM!YGl_(}Bp
znzZTaUvHW`_xkyNZ`Q>(_Q{)PZ2FvInewyp+m^KG^V9ZjI~zAM?KvnTes=X(l;+-@
z6mZ$x>D<ial4sQJZe5je?`6fA>U5Lei;i5`V^O$ABQRo1%D4H2D`%t`mADxmb$#Kt
zC4b2(&8xd^t}Z<JX6>8o*J<sW*85(1lVNw|)4$eAu-{g%;;9Ydm7R0>P2EwGXyf0D
zOgGo}o<2HdZu;RO!_?<D->&?Ye%*ibZR6^lQq#|#6H`ooDt<ZT^OPB@Zr*G<<(nY+
zMSbV3L#k&J%wDbweY)n+vI@J)i}!hyeR8N>IU}m?%W>toH{bO4_urVKE@n8X`c_ZU
zq}vbtgpD>^2GxByU3oU{UE9GKsotBXzMGf0?ae#EnC3ORw!LIOnZQ2PI+k;}<e6o6
zwyxSKw5Re$?T-cL4bLTRGg}^>_VrcT)6;6^S7mOUo0X-VpVf2e^O~hIHhq3`vTLRN
zs+T=mwp`C&bYE2M?71ifvCW&;ak_4<p6bmp{p6g7+kOeAwp`{(3w{@6TEl-T^31ig
zTbl%hx8!d7bRuK%j7#aaBD*xJm)(8YwQ~Ku17gj3XZU=ByV%OAR<Aqy=FPjfNooD7
zrU?jb?mjw=alJ|N&6eF^elzCgpYGdoE%A4nf6wY`i)LK<yhb(UX63gjikr=sMwvuc
zI`<})S<T%xWA51lp0i@J`glY(cSr0g@HU>|yDj@^+fD10^SpW`4!?QvCvA54TGtcH
zjy5kkn-ul@{4~476FI&s&CZ(5R&KpH`O!qnwwpIOCYjuh`LQ?n-|yxdL7$G^ySQv#
zn!)Uh>!(CFSATW*8f_RIvvr-0qIcmY(co8+(Tdr}bDk<%?z;K1;>H=j<IT%%o(hfo
zX~?xGv^@K)-M-9ct8cpO3Nwh-UADHaP^DYb_Iy;ArtNvZh?%*wf@aL!=CDj`^W}9X
zXDz#VO37(sptbpLz5kgH0%uiboAs^}JrmZQxh!7hWRlnU&C@0<eIA*XmVf$vl+<Zg
zkJ&e0Dyf8S7T#Ju^Y4cbSKrJ$(_JWEH6`udEZ@c1tedYU8KuqG1PQE{mD6V4bSrk5
zwfF4(%&b{+&mDL<EB5O1bF*$M-%eiqW7*8@X_sF_9n4uf<IC!6E5A)zSe|Xz`B}$!
zbMKY8EAkWi&CE7mR!m7R+ZX=w+PC&kzQ*-u)h=!=y8R~ccG~OkwWs8&1ihmTIWO#(
zxz^lW?N+sQSlH&H4I7NB727`@U2O9CdX7-6cKcaAD}(AacXzB6nQU?X*t8^}=Qkyd
zH!qJWcA2FcrLgd9&PCIkMaI8x&pcCRbM}wvY?td@9XH?J37&k{|I^(dpF=9qhMdQC
z%v?Krk<AHx&w$l8m6qR}Ra=;~`L3UF|Jh|1wl>|YGCZ5ppLE7|lj!7xp1C)pPgiGs
z`k!YSJ%ed;<&v4l(}G__?aci)cj@+uxXoK7mkAj$&e}R-(q|p(&0C*!B$a)7Y(M*E
z5>KL8zscttIb0w&=Y5S_<7&7ve680B7XR#ML)9MNoio=iU6-^?LU8rXlN~+@X5A*A
zZ{{$?YAdD1-;{5!WZRtn^!h>3=aJcls{21Q&%K#d>|z+Lvf}ESC*lbplfjDH&zjxS
zy*BA);*$84>qI86IXdOfA#Ll;=@ah>NNRItnO0Bh<3E!&A<b^K?}F^78*Z+;yxsTT
znZ-_KVR9!{h+X!n_FeR==W|5OfipQ*MXtWdwc<(LeE19VjI#=7_n6)OyKd_3H2s{~
zNyf9+#>n{g2Cv&Y<C@Ibu-VGO$+nlWzDcBQ_B399wroGhXz^43>|Q%polD+kbU4J>
zY@N<BPVdcApLJw?+j8lR>6vGf(&A>nt-F3Iar4ygGGESjO^x$86}mdir!ds;Y{aa!
zOV(wTMPz;xODjegY5TTx)#q(aY8_G2x-+9iH%(Z2GwI%+Fe7X4fbN^dGtR1>{bP3f
z=ent<)7X#RFT0&``^3hhCcRgdKU-HaSx3=ZGCBR5N>IF}*)`vdGh`3t=w&|Je)C-U
z+8DFnVYScx#Ac_R?q0mI(tpa{8P{~qPFrbhC@+y}TFtY0=aQNC)2#1rExP%pd(Nxd
zZeg;qK1V}Wi)|`QHS~>`r8`@B*3HXao39(p_MBCjx9xST_VlywmcI{qd*8+PU5vd#
z%3Jwf=WF5dy#IPn8~jK$Ycy%T$+9bq_w2shO=1;Y&phhxROWqa&H5&jb~tCo+r;88
z0-JB!{^Fh5bkjgO?QqVsSncI!rwPARJktgesODyn+&uYTWl_ynyObB{hjV7|ZsweE
zxBAw~zZMQBD$?{o%4W~L$sq}1NHZJ&DX`CXzdN<wW!6CuC(-N@56CLs%}|zz@R_!o
z9>z0xHwXF{NN2KbPBc67ZS{;cD7&lE%kb-S=NY{w%{L9CkGd?Kab@?PM6*k0<TtY$
zNT&&#Mla&sJTdLx3|px~IWv5(ykW|n(QESgL{3!r3<K$-OJ>Yf*FWPn>t~ADN^PGR
zy-7i5&rK3fQ$D-LX!a#hwKHypnQ89l4Wu)l&A-{Q&LGjO%hNo10m#I8Gku?P38yWd
z!Lxa>JIKiMZ$xru^d>0<`uu%;SMh%8=7yUd#{bWVWgN~CI(}ezl2^&TeU<lQ{rOJ{
z-t1#N({}UU%()AAH+#yQcb<22i}?Ae>}T3;UOBVR?v2KodkcM)jjp%L#hj7TZM)L_
z(qdks*_pJ&Zy%3Oa7*d4te7F|xutVXn(*BDt>+v|mNBOt&iQ|)&&^<V+fAX|f6H&y
z_Wjg7({?j%mh8bCr8M!3!*5#NFkVwgG&|FF({J|N#c!A2EMCiQC;#Ej>vF?4CTHcq
zGU_JQ&a?O4Y<k|lMD|~(P7GgHB8SRNpR;{duGSF;a-N@Ec0sh|=Fx9Hjb`ebThH}R
zdVb#LWL@rs<m|HL8SEFC(&oS4QDN~z;hN~2l%=%~3Xh-2X*3JmEd41a-E8XV-sUs!
zgx)Gl$~c@O&Zuxn^oxek{PrBjRPWkTpT8coxjw^p+1mOUH)R*ex(MZ-c%a1Kva9Kd
zva!|Es5?n73Qx4${BHX!t-QTVz;laYb937n_6{D~&5MtoxIdv+<>s8T?|49FJF|V~
z5lm4wdUD3@eAs-?r~h~Sd9a&1?f(3&_jl>-d$aoA!4ST~ul}T!w}Q-B|Hs!joY(VY
z+f9SnQ*M5@)j#u&17ynKoa;O%Pk=-x-JF#3EUmn`OhMkCZ*#1%in?^#WaHfzr~Q(b
z-+lkbkFAM8KCk$uu+5bI*ReR`=f~BxoBJzkZeLtnT6t!br?TXXr86hDiKINv@jClP
zE&A2+!hg~Gf3)WOPy8{vzVT_-gqz0~*Y@1>y{CE5>THam%Ou{-og43`c<)`a^!)0~
zZ;E{0Cob_P?MnMUb@2zyTlU?F24|-ozA){k`{LT0e%yWKvn-2jc&iL%FT3fx?MnN`
zsQTZYo7b+oGGWcl;8Kxkv-j4i>&=`y`|`4z7aN*xp1FMHAaks?SDN{)Ao<s<J3d~1
z6CmHJs<`>R=b6njmrG`DzPR<(oi}dLuhMTi9QS3LGyi6EvP4+Q8M|q39-p~azH)Z;
ztUxpA*wyFG=51SQrzf@J|B+L-c27@Re)-MylyBe7?^VmaXu9b>xpvCUYW>n;f8#@G
zi?grpym@xpmGf85^53qV&GhDs*yT6&m!@wot5(+M-GAjM=SjN<98&tFo1v<=-sIhW
z<#^QRvfcOpw9mbnH0$p&o>OfdH|I{MJz%YRW4UB3$d$L=v_-#4|25OJ$2@M<-mBAP
zm;d%<-fg~HdQ<V{@A0pm%}*;i>vnxr^5)rjEAP8i$t%cIXO`tWdDYf-Q+DFqnRgRb
z&aR&|(NsEibzlFn=vR+xr6bmy<@<i`?(-=(V<*m?bvFg3Ks!lswp@p(@n-dl`&Req
z&Acml^UxbL@4IqKBtth}+`33?)5(2*weO{^=bz11mR(k!qMl|R_3E3#*%h;T*NJYO
zP}>yxqj>qH_q^4c?Ps$6`@Qq`pY6$M<sRS8y-7nhrE1Nm-;3)E&8wsCOZ|-g`CtX#
z*>{uP#9b&@dAr%fxmaXkChzL&xBRu=^*&1dw&Yv?zxBTk2Hu@{SM%oHcE(FQX9Hc9
zbx%0^iY;R2(ydEG^*3)XJ*0L1cXjEpH+9=rnmWu{XxVk5#Oz?r`pG)>_abJ;&ff8A
z?%wz7K;<S+MqicWrcz$rg=Oxq+cw`UJuf_W<>3u2H$99^+y!U$b+0V@_t`qWsX9IH
zTlQNM>zV0)w#;DKJTWb!Psu2^O=sOsqpoiDyMM#YdZ*q@JiBdTz&WF9PuH_;o|yI`
zO3_Hy-(b3F`h(P`a(-cH*K@xH=UHyJ^MDU3+h@YM?UdvW9Rcgj`>*>jn@^3GN>e_2
zVXIS0R36u!cZIgmd2$<!b}w7Gy!LaXW+|vrh}CvEHgQ?Qw~M!WB(*nZ&#DT3yZBOh
z^Gy%qlC=Vo=3SoigZ<8M{7m{5=6!9Frh#;3+d5ID$8O5yixN#u&u@NnCNA{b?`gN}
zV}uNM{}(tReP(`B`@*;BOC)D4ei60NFgkt?hi_AjVf6WzGk>nVYieCN^Rv?LpJtV}
zO22Kc`0$M(la1lR)})MYLTRr%y-&XJ*ciDdG)+Ud|J{T%28LMeyfkU4ng1^cZ|==E
zHvPYFb6UJx*X4<iwtV=_3fB2Z%;+)8o0T_zZ4)d?TkiKgbo0^k_aR!-6ph{OzFp2~
zzBggRO?|ytXEn}Vka11fm7DH+`skNu599WG-h5LololT{<L>)YIo~qdo@pput~J!P
ze<JuZdz%rLq2%VduX{K6gCow&N$i@~<iZc8)nVD&3ImL)BND%HU*cg{kj=U|ahr46
z_p5vBUdc^b(7d<KK>BQ;%QF|n)%!}XYHwbhE}e4LFZW!XoMv6W$o`3G2GX(GGk7=O
zHTZq{9siDSF%|i#%T6+7vN05}x?Fj)BkPH-;^W=d*Z*3{Ew(xH&DYmo*UewQ*8Y3`
zx`}BF3~9lp)v@6|rONM5{#H<*f1&&NFU7M33wuv&k=&H~bE)I+UK{<*+VlU|fnsn5
z?`B!9$fq`&-fi9X=ghh}?B`y^6>>q-nr{1=U$?GkZ`Ssmc_?Rz<gT!eGymV*y%;1M
zDemyl=~VSx&3BT^c9gIA#B1^-=%|_<1Ec%_1%818$LEz~_%8ol;uQJv*_=7~cmITG
zq}5fujk{Uvpr(|1xB6ULeA~YIv+rV-Z}(l7CT+Yt><rJ!HNW?>o1a~IbFr`Q#hm14
zSGs0q<Som*{yS&+<l3CF^5C@Dvlrbwc4nohaq~T&x%=f_I$w|7HvR0ooKstV@AqB{
zmAdaUcR5qEU(xq9H>GD6t%Qh9)X<*GDn9q)v+p<g=bqo<X;yv3bXWT9{c~O}lh42U
zX3nuQE2pOMml)qQHCyoU>9pNBw~gOUE8bu+`|rM=vp2eb)-}9eRI=%9J4<o)w}1UJ
zgSF4fy-2glzGHH^d)AbzTkU7RFW>zB+HIS8wJVR-Y&EdY*k)Lqwe;rsDr4QnU$)le
zpJgt;xnkwQ+b_0MT-tB8DXsr{ed_(Y??u)h{j;$0-o@HGd!I&I>@=<ZfBekKsllrk
z<}Ks@8n0Tg?bqi`NmI96T(i@!FI;o>_WKR;kIns7+&s<pr|;XV?24RIZy(QpAFb{A
zw(9Kd-dXlvjZ!YW*(-0nJZxS=+pd|@%zX<#ZvOMQH1M|8`)TW^$E{=jRaaY3^4GMW
zxGwt7!z?rZ6*o_tiJL$D|8)N4s`#|i*H5oD&YhpzUswB~?$_SGat4t-|9@mHx@p-1
z_VcT>Rav1T_N)HSU2`|p)sXv7*!ovt+G=mvSA2Z6Sx-2t>?^D2<jBX{XM|2q7y9Kd
z7vdUUw=v!+y!2<}thtA0SZ*!KGQDy3ONEZWkEQ?0@9uwHU&_6W_3WYj96I09`qQ)H
z-=*C9S9L8eFK%%`aF@ENYJ1xI-w<tg&YHyS&dfV3y1Vv%<@qza^ER*7G~GR~eQQ~d
znaa|#w`%WqXZ?%aY%F2CJj`!x+wPgy-YvWREHn4`-jLr9Qm;FeP7K+6eNpWHS#oA;
z&+x}qKhScGH?eLuEY5m*CF{WKmmhCuyMB$`e1O-OGxpE#SIy?J|F@h>jXJA0pJ6rE
zndaDi9B+BAUuEB4`uDQlvsrTwLR|Ia?3NwIr@s8zk#$%6`{oO$ud<8JH{Qv4FJ0PB
z`%#*{-1AehkDo>V)}GrMx2|jRk2xnp4W3KeX&+B}{}p1uo3mehj$C__t~5LC{YS9S
z)3i%D$;)o~t<|`kbJi@4>A{s#XP1EXo}4joo|z_XoSM*PboN?KvKqsgfMqvlt>qao
z@eH%nS-EGXG)kT_iQe2hlL6$u>}^H{i*HWJ+-4ZemIm<w*nRlKnNOMAo{?#0JSB5m
znzV7+jD<d%O`|1eFoH}snGG>r(ir61V?L5IF6MxIzw9Q6KP&gljKnshvu0_h4)mmD
zgS0b3gXb6z$UKBAm=5ry&RTo()PbQQ-jf!*>?X*=py0SY!|*WF{kX(=G%n{PFXKT9
zmg$b0r#4CkfK1su6Uh{CoSj|fvpF{YY#JmnAwtU}db8>68JrBD(1XQB?@S{|+!_iq
zAV=zR>$Kujfrgmba~brUPr4^h`!OSPTiWiKpy=ENijM4UhL>~BmKkS)c2~h;@ZfyD
z#_WJeMM;mJZ_qdw%X;>hUPbtxLcvojGQ7^OEEF}7w!W#aJLBr!2TEUdN{W;*pKWWu
zHrLHH_}3ktH*6R8aqnQ0ixTebTJ~k9<PF|Mcf)kFq5MBC@2!sPTpJN)vt^l-==6o>
z&mPDVc)cwt^5cbBSs^A%!Th(|9%)r%_?l?Vy7W9P!Mvj^dgB@>|22qT3gMe-J;SNr
z3aTEzezisC&mOQ7N44Kh{Oz_!UI_j3yHV|*-wn4v4X6GnqVz9Yv_1QcmdhS+c&Po_
z@lY~w^XEHZF}(b1Gwt#!+^)Q_*)DpzYqsXzg40_KPnSm<{>@W6+h*7;)xH?a@b>OH
z9B4i{?eGjka2|yurXEm|GDthq0Lpg>ZARgan|mdV7x-*WJOjzIpq%zrOTswKVJ4E+
zQzp{Kz*>*-z_p%jGfZX!6&B1O{k@W)tdB4Sl%T<d2*fC9V*{uw2HYrcb}{P#`??oe
zQi1%=n|utM4cXE(nn9TdM*y6Ghv$I%gv{_|fBN9->+9>97#fVuvax^(2DvEZv_i?a
zom>}k7#h-;8BC-Zj1E22AhchXNck5t4^N&vxqi0(zSuqAPHkmuxM86sCbNG3H>D3!
z3=e)*Mt=KQ^LK0c)9m~hX@-v8!<)_@UH!|O{q*!0DTa=|;|}uMjq2WAVsNO@;dipT
z8XqZFSYMLLpkO!ce!-uZ`lnY4%@|xxJ$`s%_A&1Hu@Vd&6E7bNJaPJ3B4^UW^^+%0
z{x7wDs(s>r|9R6NzkTGLZzu3iwu1MDcAr%B*7;w$r+?%B8M1HF9;t=-)8v1!S$&%E
zCyL|GlK3a*U5>x?cv?B7RG@;p`k|)aOr8AWv!{t4`QlS@uP<cTKBoLc_kVov`8Z^)
zWUWfNOj+9OlH#_=Hx+&Ry6$gXsg#B7q4_okYWUWa)tM-s5|H@y!k@X%b=7O#Hvy4l
zW@)dzw?2QL!TZy5{}1!i{z_NBial$eS9U1#x6!ZJWf$8HTyQ^oI&IPCZ&$0hKRf$d
zKU}ew^9M-c!T)dX(zX{$b1|>bb*`>vyH_ohF`sY3_X3$8f8&>ao$umxNThVhnWr!2
z_}(r#?p>?0RQu)<{@ly<TlIFw{Z&2l{OqzhX}AAPJoeK%rPb#bliyl*MbmvPp?SOf
zk_#@MJkiF>^meW4oGANG-Y@T-@pXTmFwMvI?7Qcd%F-L1)M^Vt>#}02_Ez02y7tB7
zsmbg7J-s<cxL$tIetl)v7jMn|A6LoeYW>Z#SC%&8s9CGHXHDp#rBiehx7KZmTsG5h
z{{c~>m!~a+=eAsl*p^W>%V)i^`||i3Tr<C<&W*aN9ee1@Vqf+5Ck!_QHYdEE&BW>v
zr5fXPQ)t=C%vYIZ9^r1v(uN#UmNA^P&U!UfH*wayW%JVd-Br&OCu*l$5--lW%eC^L
z(6ibNoT+!~+{|(WOuT{{)`1+LmpCgUZC09_SwZWRWi#eJ-R-?{zV6oG<5MQ}$tX*2
zck&9pvhpg(o%h&h*Uh?{XC-;_fa3xa){JP?7_XN?%R1ARWu~P#hkImpSqjU>dR1=A
zSTr-@_Qc1te@)abSuMEn?)?U#pqoP5uJ@a*-FDl5dHfTFfLTI!qE%Z$wzXs~nwdIl
z?G|;`^%E|M6suXS2)8QUZ(Zv1_TJ<ThMOEizuioo8|AOJw1hL;<f6~pdvBN9YL+y%
zIC#EZ?X4Mk$i()n$=NOHN9{OXI%}+o&|6x=`PuHqk~~?<dg+@B%z_r|ah~X{8Gop+
zEooY2T1xVK{khw?19ne2w(05BfHOvCdvZK_Kc^Nh51Sq|PxsyHv!y@GHU+N!ys&!B
z8r5qcPtTiwe(Pl8O@ZZF?tg=p-mn(t-TX8|A#G;bxnjlG_Z)e(5ersS-AuQ?!F%rU
z#fiy*uXfuqFt90s&IM3ulGF3wZmrU^V8(<^|4%MDdS<Sw_?+jD{C+*Q<=S>^@sz*)
zyBYgF#23snKD)^8*Y$70nIb|#W)9)KsyWvtnx0)Gw>A8``$Vk}9}i36*qkpb&)IBp
zQ%cL!6Ry+c=;G4!wObwf>s9+q>+>gOr5X0$YA<76?xY&y)f?9N<;usjXXnc28J|s>
zmUeqtdHBNZpZOOHZ;6v$er@*ahaa5hzw%LDt*oAtnU}h96IX7F)13)XnNuT5i@#-X
z&dyBn-MnM+zi;m&TeG%vFkjDJANr_HVvk?sO{4QCUa4~1yxd)Hf9eg>t5D0WWw(0{
z=GrckycxVyS^Y)e?>R?Gq7`S#S}xhcSo-zHPABghHp;7~>%0p5t-SW~VyCIrErG8~
zZw6$2+F4;@dve-Lw`(Td)-Urnahd*dW4eA?I^b<fY~-Od=}1{iNAJyNg$}-5E%a(>
zXwuX@{AnPU)mrMMW%~U*d0Td(kW<#0&_~brcmCV7M4F}a_k}w`W|vRAI>8w^>)ngZ
zVGDztvTkT?%oO|1SAOY>iCJ;|t;+hWl<nH_ecV-7>fTO0^>($8Q`QBoi#qT2Zr$=S
zbII-ZvyV@jFLm?s3lrsYOWuDg)CzIh`)jSx?{9mz9<{MrH))Psn)lMFDNPfvP88l~
z^73clVyC4oS90TP&pfV)lz#i=azJyA;nd}lk+PPB>VH>-FZ>$0KK@13)3rj=|5x2y
zS9Z2(-G((USH8Skk(=`E^DpzcHsT$jvvQ)!MWa@Sys3IBee&79{c-xw8;_)I$WS%@
zcGi6>*szaY#ZvS3oj2OuFyT~`>b1Aus`k&&*7@%jwy!02p8dR(Zz7;Dj>-Jw8+rY$
z`sU{LkZVtZw|2izGxWTgUFB6D@^sTzi@m4bobP#4mNH9lkxBcX{M^=*vM&?7qE%nz
z#Bb?{4cz}y^X#-=A6vs_{(W;|-mztp`)}WvyUOZkrN>O!A3>+?H&neAyT83e{~!D6
zs^9zX36({he53v}|5oOK&TaXx<*UrT-As+~TD>N8(Mz-Izgy>|{Py*KDHIl8J<sK=
zit_5)`j?L7Zol34w{ZKUo2)G%x7w!4%y^Y5lJ{ei`=*vq`!J*5#md%JXFK97OS@!)
z?yfAI{C~o8AujVjoKqdQop|Hvc5V9O6f@!0kn383s{hj!?f2Q_R7u^bn0oVk|C_>`
zS&Ex~hIo{&70OIoKYilOvm0`E%-1!lo;6Xn<oDY9=a!2zt8V^zU-9ZTuPY}_xAvTo
ztF~P~U)SjO%ZX>JeoxXro~!piM*nqiiqY(sPOEaKuDE>ojnbm5H|?3f<{mMsp7lBA
z(!}8SwlDv#-f!MG>E<6N({EO{7V~EB-*M(%{}Q+VnrqM09s6*&(9`{#>gMm=zXNuk
zdG@mQTlV|@`|s*w($eN{Q!Y%N|M%tVI)l95U&_tP-@4bw_f9+WOuTIOU4zDVy5Ihu
zsf#^Q@aMY3$v5fei<8agp8J00UwPu68*$T*FTObGrcM1k+a>Ce)(SKKb~2w)=)Ij=
z`dwn-M1CXbnYx>u-)wfzw~9T!DKCG{mAs78IVs!D{QFy1d3j}?d`tcwY0b?OZyH(e
z*d-mjSR2f&OWQv?&1kl!(Ql2-so%m=zHOXcylm!~XP(tFbvN(V*vxMfoo1Yy)~&_-
zk#*+Y={Lc)q?#F3OV0c|&uyyU#G6LZX~MsPqHb)DNZoeEE$zSL%)K9*W}TUK(`fdE
zs{OiZn^U))nRb(9+Ov|S1xC?nr*jsif3w`V=3Z|HM9tBx$C}mZL21gHQ_U0~zbs@l
zty}4zcKVHLmdwV}6LZ$7PCYa2rof$jwxXf8+)}okX?w8BZ2mo)rYSd#q?@Ac>+GKh
z-D7{OxH;9V<ywqW_3mHyDvJN^YuPGeonqD!d~(Idiv`v3Y0g)AzxDoRG>T4hHZ=Tt
zIO0UkOx}lY^EtmJsoXg~?Pfs3>Ceaa$GDxyVMsA^2we3}u>5JJIgw2_@|Ar^|IPk2
V+FkhoQ-U1G6i-(_mvv4FO#qxi-1Gnd
literal 0
HcmV?d00001
diff --git a/5-NLP/16-RNN/torchnlp.py b/5-NLP/16-RNN/torchnlp.py
new file mode 100644
index 0000000..d6ca5e0
--- /dev/null
+++ b/5-NLP/16-RNN/torchnlp.py
@@ -0,0 +1,104 @@
+import builtins
+import torch
+import torchtext
+import collections
+import os
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+vocab = None
+tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
+
+def load_dataset(ngrams=1,min_freq=1):
+ global vocab, tokenizer
+ print("Loading dataset...")
+ train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')
+ train_dataset = list(train_dataset)
+ test_dataset = list(test_dataset)
+ classes = ['World', 'Sports', 'Business', 'Sci/Tech']
+ print('Building vocab...')
+ counter = collections.Counter()
+ for (label, line) in train_dataset:
+ counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line),ngrams=ngrams))
+ vocab = torchtext.vocab.Vocab(counter, min_freq=min_freq)
+ return train_dataset,test_dataset,classes,vocab
+
+def encode(x,voc=None,unk=0,tokenizer=tokenizer):
+ v = vocab if voc is None else voc
+ return [v.stoi.get(s,unk) for s in tokenizer(x)]
+
+def train_epoch(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):
+ optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+ loss_fn = loss_fn.to(device)
+ net.train()
+ total_loss,acc,count,i = 0,0,0,0
+ for labels,features in dataloader:
+ optimizer.zero_grad()
+ features, labels = features.to(device), labels.to(device)
+ out = net(features)
+ loss = loss_fn(out,labels) #cross_entropy(out,labels)
+ loss.backward()
+ optimizer.step()
+ total_loss+=loss
+ _,predicted = torch.max(out,1)
+ acc+=(predicted==labels).sum()
+ count+=len(labels)
+ i+=1
+ if i%report_freq==0:
+ print(f"{count}: acc={acc.item()/count}")
+ if epoch_size and count>epoch_size:
+ break
+ return total_loss.item()/count, acc.item()/count
+
+def padify(b,voc=None,tokenizer=tokenizer):
+ # b is the list of tuples of length batch_size
+ # - first element of a tuple = label,
+ # - second = feature (text sequence)
+ # build vectorized sequence
+ v = [encode(x[1],voc=voc,tokenizer=tokenizer) for x in b]
+ # compute max length of a sequence in this minibatch
+ l = max(map(len,v))
+ return ( # tuple of two tensors - labels and features
+ torch.LongTensor([t[0]-1 for t in b]),
+ torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)),mode='constant',value=0) for t in v])
+ )
+
+def offsetify(b,voc=None):
+ # first, compute data tensor from all sequences
+ x = [torch.tensor(encode(t[1],voc=voc)) for t in b]
+ # now, compute the offsets by accumulating the tensor of sequence lengths
+ o = [0] + [len(t) for t in x]
+ o = torch.tensor(o[:-1]).cumsum(dim=0)
+ return (
+ torch.LongTensor([t[0]-1 for t in b]), # labels
+ torch.cat(x), # text
+ o
+ )
+
+def train_epoch_emb(net,dataloader,lr=0.01,optimizer=None,loss_fn = torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200,use_pack_sequence=False):
+ optimizer = optimizer or torch.optim.Adam(net.parameters(),lr=lr)
+ loss_fn = loss_fn.to(device)
+ net.train()
+ total_loss,acc,count,i = 0,0,0,0
+ for labels,text,off in dataloader:
+ optimizer.zero_grad()
+ labels,text = labels.to(device), text.to(device)
+ if use_pack_sequence:
+ off = off.to('cpu')
+ else:
+ off = off.to(device)
+ out = net(text, off)
+ loss = loss_fn(out,labels) #cross_entropy(out,labels)
+ loss.backward()
+ optimizer.step()
+ total_loss+=loss
+ _,predicted = torch.max(out,1)
+ acc+=(predicted==labels).sum()
+ count+=len(labels)
+ i+=1
+ if i%report_freq==0:
+ print(f"{count}: acc={acc.item()/count}")
+ if epoch_size and count>epoch_size:
+ break
+ return total_loss.item()/count, acc.item()/count
+
diff --git a/5-NLP/README.md b/5-NLP/README.md
index fa9f7b3..89b9f55 100644
--- a/5-NLP/README.md
+++ b/5-NLP/README.md
@@ -52,4 +52,6 @@ if len(physical_devices)>0:
* [Representing text as tensors](13-TextRep/README.md)
* [Word Embeddings](14-Emdeddings/README.md)
* [Language Modeling](15-LanguageModeling/README.md)
-
+* [Recurrent Neural Networks](16-RNN/README.md)
+* [Generative Networks](17-GenerativeNetworks/README.md)
+* [Transformers](18-Transformers/README.md)
diff --git a/README.md b/README.md
index 42d2a50..ad51a05 100644
--- a/README.md
+++ b/README.md
@@ -64,9 +64,9 @@ For a gentle introduction to *AI in the Cloud* topic you may consider taking [Ge
<tr><td>13</td><td>Text Representation. Bow/TF-IDF</td><td><a href="5-NLP/13-TextRep/README.md">Text</a></td><td><a href="5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/13-TextRep/TextRepresentationTF.ipynb">Tensorflow</td><td></td></tr>
<tr><td>14</td><td>Semantic word embeddings. Word2Vec and GloVe</td><td><a href="5-NLP/14-Embeddings/README.md">Text</td><td><a href="5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/14-Embeddings/EmbeddingsTF.ipynb">Tensorflow</a></td><td></td></tr>
<tr><td>15</td><td>Language Modeling. Training your own embeddings</td><td><a href="5-NLP/15-LanguageModeling">Text</a></td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>16</td><td>Recurrent Neural Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>17</td><td>Generative Recurrent Networks</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
-<tr><td>18</td><td>Language Modelling. Transformers. BERT.</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
+<tr><td>16</td><td>Recurrent Neural Networks</td><td><a href="5-NLP/16-RNN/README.md">Text</a></td><td><a href="5-NLP/16-RNN/RNNPyTorch.ipynb">PyTorch</a></td><td><a href="5-NLP/16-RNN/RNNTF.ipynb">Tensorflow</a></td><td></td></tr>
+<tr><td>17</td><td>Generative Recurrent Networks</td><td><a href="5-NLP/17-GenerativeNetworks/README.md">Text</a></td><td><a href="5-NLP/17-GenerativeNetworks/GenerativePyTorch.md">PyTorch</a></td><td><a href="5-NLP/17-GenerativeNetworks/GenerativeTF.md">Tensorflow</a></td><td></td></tr>
+<tr><td>18</td><td>Transformers. BERT.</td><td><a href="5-NLP/18-Transformers/README.md">Text</a></td><td><a href="5-NLP/18-Transformers/TransformersPyTorch.md">PyTorch</a></td><td><a href="5-NLP/18-Transformers/TransformersTF.md">Tensorflow</a></td><td></td></tr>
<tr><td>19</td><td>Named Entity Recognition</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
<tr><td>20</td><td>Text Generation using GPT</td><td>Text</td><td>PyTorch</td><td>Tensorflow</td><td></td></tr>
<tr><td>VI</td><td colspan="4"><b>Other AI Techniques</b></td><td>PAT</td></tr>
--
GitLab