diff --git "a/Transformer Mechanism/QA/tf/W4A3_UGL/QA_dataset.ipynb" "b/Transformer Mechanism/QA/tf/W4A3_UGL/QA_dataset.ipynb"
new file mode 100644--- /dev/null
+++ "b/Transformer Mechanism/QA/tf/W4A3_UGL/QA_dataset.ipynb"
@@ -0,0 +1,2996 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "TBjVwYpHJ7ra"
+ },
+ "source": [
+ "# Transformer Network Application: Question Answering\n",
+ "\n",
+ "Welcome to Week 4's third, and the last lab of the course! Congratulations on making it this far. In this notebook you'll explore another application of the transformer architecture that you built.\n",
+ "\n",
+ "**After this assignment you'll be able to**:\n",
+ "\n",
+ "* Perform extractive Question Answering \n",
+ "* Fine-tune a pre-trained transformer model to a custom dataset\n",
+ "* Implement a QA model in TensorFlow and PyTorch"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "SoRb7ykXJ_C4"
+ },
+ "source": [
+ "## Table of Contents\n",
+ "\n",
+ "\n",
+ "- [1 - Extractive Question Answering](#1)\n",
+ " - [1.1 - Data Cleaning](#1-1)\n",
+ " - [1.2 - Tokenize and Align Labels with 🤗 Library](#1-2)\n",
+ "- [2 - Training](#2)\n",
+ " - [2.1 TensorFlow implementation](#2-1)\n",
+ " - [2.2 PyTorch implementation](#2-2)\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "C0k56ZVXLDbi"
+ },
+ "source": [
+ "\n",
+ "## 1 - Extractive Question Answering\n",
+ "\n",
+ "Question answering (QA) is a task of natural language processing that aims to automatically answer questions. The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question. For example, when tasked with answering the question 'When will Jane go to Africa?' given the text data 'Jane visits Africa in September', the question answering model will highlight 'September'.\n",
+ "\n",
+ "* You will use a variation of the Transformer model you built in the last assignment to answer questions about stories.\n",
+ "* You will implement extractive QA model in TensorFlow and in PyTorch.\n",
+ "\n",
+ "**Recommendation:**\n",
+ "* If you are interested, check out the [Course 4: Natural Language Processing with Attention Models](https://www.coursera.org/learn/attention-models-in-nlp/home/welcome) of our [Natural Language Processing Specialization](https://www.coursera.org/specializations/natural-language-processing?=) where you can learn how to build Transformers and perform QA using the [Trax](https://trax.readthedocs.io/en/latest/) library. \n",
+ "\n",
+ "\n",
+ "### 1.1 - Data preprocessing\n",
+ "\n",
+ "Run the following cell to load the [QA bAbI dataset](https://research.fb.com/downloads/babi/), which is one of the bAbI datasets generated by Facebook AI Research to advance natural language processing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "XxU0G_PYLSXJ",
+ "outputId": "44e7877f-5c33-45fc-ed83-3aa4920dcc40"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'story': {'answer': ['', '', 'office'], 'id': ['1', '2', '3'], 'supporting_ids': [[], [], ['1']], 'text': ['The office is north of the kitchen.', 'The garden is south of the kitchen.', 'What is north of the kitchen?'], 'type': [0, 0, 1]}}\n"
+ ]
+ }
+ ],
+ "source": [
+ "from datasets import load_from_disk\n",
+ "\n",
+ "# Load a dataset and print the first example in the training set\n",
+ "babi_dataset = load_from_disk('data/')\n",
+ "print(babi_dataset['train'][0])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "XJwacC3bMhZM"
+ },
+ "source": [
+ "Take a look at the format of the data. For a given story, there are two sentences which serve as the context, and one question. Each of these phrases has an ID. There is also a supporting fact ID which refers to a sentence in the story that helps answer the question. For example, for the question 'What is east of the hallway?', the supporting fact 'The bedroom is east of the hallway' has the ID '2'. There is also the answer, 'bedroom' for the question."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "aizPXfGlLZ1D",
+ "outputId": "0e1d47bc-9c1a-458a-983e-22f47f8184bd"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'story': {'answer': ['', '', 'bedroom'],\n",
+ " 'id': ['1', '2', '3'],\n",
+ " 'supporting_ids': [[], [], ['2']],\n",
+ " 'text': ['The bedroom is west of the office.',\n",
+ " 'The bedroom is east of the hallway.',\n",
+ " 'What is east of the hallway?'],\n",
+ " 'type': [0, 0, 1]}}"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "babi_dataset['train'][102]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ewtXZUPjMm2l"
+ },
+ "source": [
+ "Check and see if the entire dataset of stories has this format."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "id": "55BSWxwuM1hN"
+ },
+ "outputs": [],
+ "source": [
+ "type_set = set()\n",
+ "for story in babi_dataset['train']:\n",
+ " if str(story['story']['type'] )not in type_set:\n",
+ " type_set.add(str(story['story']['type'] ))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "bdJ8VMF1UT7S",
+ "outputId": "2b959467-75e8-4e25-e7bb-481b657a2fce"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'[0, 0, 1]'}"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "type_set"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "JsHx1tcyMq_k"
+ },
+ "source": [
+ "To make the data easier to work with, you will flatten the dataset to transform it from a dictionary structure to a table structure."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "id": "YxixFI-pVOK9"
+ },
+ "outputs": [],
+ "source": [
+ "flattened_babi = babi_dataset.flatten()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "kXU43CqCdX98",
+ "outputId": "e968ff5e-0db0-4e9d-e1e9-e93f965b2582"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "DatasetDict({\n",
+ " train: Dataset({\n",
+ " features: ['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'],\n",
+ " num_rows: 1000\n",
+ " })\n",
+ " test: Dataset({\n",
+ " features: ['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'],\n",
+ " num_rows: 1000\n",
+ " })\n",
+ "})"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "flattened_babi"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "OQw59MgT6Luh",
+ "outputId": "ea5eac53-027e-42d3-d19f-98ed7863de2b"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'story.answer': ['', '', 'office'],\n",
+ " 'story.id': ['1', '2', '3'],\n",
+ " 'story.supporting_ids': [[], [], ['1']],\n",
+ " 'story.text': ['The office is north of the kitchen.',\n",
+ " 'The garden is south of the kitchen.',\n",
+ " 'What is north of the kitchen?'],\n",
+ " 'story.type': [0, 0, 1]}"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "next(iter(flattened_babi['train']))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "4vXfmhOPMvt1"
+ },
+ "source": [
+ "Now it is much easier to access the information you need! You can now easily extract the answer, question, and facts from the story, and also join the facts into a single entry under 'sentences'."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "id": "O5NcABwkdbrf"
+ },
+ "outputs": [],
+ "source": [
+ "def get_question_and_facts(story):\n",
+ " dic = {}\n",
+ " dic['question'] = story['story.text'][2]\n",
+ " dic['sentences'] = ' '.join([story['story.text'][0], story['story.text'][1]])\n",
+ " dic['answer'] = story['story.answer'][2]\n",
+ " return dic"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 115,
+ "referenced_widgets": [
+ "44b7bea3e09d4e5684921c66dd4c7514",
+ "6af3ec5091d74bd1a95bf02a87dd240b",
+ "7e1325e57bf9417e93d7ef180794ab3c",
+ "3dab28395f3f475d8242e4d4d45ed059",
+ "ca722dcd857c433c9058585e31a1673d",
+ "7fb1118c0b4443b6b6dbb5803e9ec2e8",
+ "58718e12f1b7459989ab5296846c4be6",
+ "63b4ebafcead4c0784b5511219a6a198",
+ "c42644a4e6184a1cbdb2b453b5dbb7d6",
+ "364ba960eb474c9084cc71851594d345",
+ "e8f1abd85f3e49f991d4c1312ffd416b",
+ "929946fdfaa04cf59d3b31cf92fc08d1",
+ "aa5c0d374889482697fc0f7ce9c81afe",
+ "ff444b253e9a40e5bec755926d83740f",
+ "89fdda6e6688476495ca297bfe010bf8",
+ "cda72c45821a4eb89f1a3ab5510b26d3"
+ ]
+ },
+ "id": "LHKNQ75afMoZ",
+ "outputId": "6ceeae5c-392c-4553-c487-14a648eb9209"
+ },
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "fb9930a912294d718a82d7c9d9e5bb19",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/1000 [00:00\n",
+ "### 1.2 - Tokenize and Align with 🤗 Library\n",
+ "\n",
+ "Now you have all the data you need to train a Transformer model to perform Question Answering! You are ready for a task you may have already encountered in the Named-Entity Recognition lab - tokenizing and aligning your input. To feed text data to a Transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using! In this exercise, you will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "c892hk9NNF9O"
+ },
+ "source": [
+ "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word with a `tokenize_and_align()` function. In this case, since you are interested in the start and end indices of the answer, you will want to align the index of the sentence to match the index of the token for a word. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "id": "UI-9P7VYitxv"
+ },
+ "outputs": [],
+ "source": [
+ "from transformers import DistilBertTokenizerFast\n",
+ "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "id": "Pex-YXJnnwb9"
+ },
+ "outputs": [],
+ "source": [
+ "def tokenize_align(example):\n",
+ " encoding = tokenizer(example['sentences'], example['question'], truncation=True, padding=True, max_length=tokenizer.model_max_length)\n",
+ " start_positions = encoding.char_to_token(example['str_idx'])\n",
+ " end_positions = encoding.char_to_token(example['end_idx']-1)\n",
+ " if start_positions is None:\n",
+ " start_positions = tokenizer.model_max_length\n",
+ " if end_positions is None:\n",
+ " end_positions = tokenizer.model_max_length\n",
+ " return {'input_ids': encoding['input_ids'],\n",
+ " 'attention_mask': encoding['attention_mask'],\n",
+ " 'start_positions': start_positions,\n",
+ " 'end_positions': end_positions}"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 115,
+ "referenced_widgets": [
+ "4d9152a30e824931983a425ee6d607a6",
+ "1f2773e3e80c4dd8b6b26e171bf33bc7",
+ "013f041c3e0b4e35bf2432fc345cb7bf",
+ "ef4e12f29f1e458f811a400faf21bdcc",
+ "f0e34f2bf626434fa73f0def26b3d1a5",
+ "1e6c02317171453cbd3d4d665879b0d4",
+ "5b6dbe662ca24834b7678638e101e1ff",
+ "39029f730ae140c7902fca6dac5361ad",
+ "723acefae33d448199fa5c1a9ec3f246",
+ "32a5c82c7a9845c09c11bb4e30c2f1aa",
+ "77273c2e4b4e4e4c8ee4b6b344749518",
+ "f0ac3b9b8f664479940c6ee18fc2f13e",
+ "393697738e724e9fad4d163de0a77840",
+ "e592db98c0c34c5e800f5d7b6d3c099e",
+ "568f11b4462f4b4e95f3ad5947bb275e",
+ "7fefe9e1121a43558d773500aef8935c"
+ ]
+ },
+ "id": "kKyLNWCvksOr",
+ "outputId": "7af3d914-4546-430c-c2f0-206b732e5131"
+ },
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "f656f4d63c5141f888c90c79afe27fe1",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/1000 [00:00What you should remember:\n",
+ "- The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question.\n",
+ "- Transformer models are often trained by tokenizers that split words into subwords.\n",
+ " - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rFfJozZvNZWG"
+ },
+ "source": [
+ "\n",
+ "# 2 - Training \n",
+ "\n",
+ "Now that you have finished tokenizing and aligning your data, you can feed it into a pre-trained 🤗 Transformer model! You will use a DistilBERT model, which matches the tokenizer you used to preprocess your data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {
+ "id": "8sdX5XY0Gwwc"
+ },
+ "outputs": [],
+ "source": [
+ "train_ds = qa_dataset['train']\n",
+ "test_ds = qa_dataset['test']"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "Be5k3ilHsJ6q",
+ "outputId": "f2f7fea3-1394-4aaf-b159-994a38476994"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "All model checkpoint layers were used when initializing TFDistilBertForQuestionAnswering.\n",
+ "\n",
+ "All the layers of TFDistilBertForQuestionAnswering were initialized from the model checkpoint at model/tensorflow.\n",
+ "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertForQuestionAnswering for predictions without further training.\n"
+ ]
+ }
+ ],
+ "source": [
+ "from transformers import TFDistilBertForQuestionAnswering\n",
+ "model = TFDistilBertForQuestionAnswering.from_pretrained(\"model/tensorflow\", return_dict=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-aQVOG4ANcd2"
+ },
+ "source": [
+ "\n",
+ "### 2.1 - TensorFlow implementation\n",
+ "For this assignment you will execute two implemenations, one in TensorFlow and one in PyTorch.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "8pCRo_parYMc"
+ },
+ "source": [
+ "\n",
+ "#### Train and test datasets\n",
+ "\n",
+ "**Note:**\n",
+ "* In the TensorFlow implementation, you will have to set the data format type to tensors, which may create ragged tensors (tensors of different lengths). \n",
+ "* You will have to convert the ragged tensors to normal tensors using the `to_tensor()` method, which pads the tensors and sets the dimensions to `[None, tokenizer.model_max_length]` so you can feed different size tensors into your model based on the batch size. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "metadata": {
+ "id": "FbpplBxNtanH"
+ },
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf\n",
+ "\n",
+ "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+ "\n",
+ "train_ds.set_format(type='tf', columns=columns_to_return)\n",
+ "\n",
+ "train_features = {x: train_ds[x] for x in ['input_ids', 'attention_mask']}\n",
+ "train_labels = {\"start_positions\": tf.reshape(train_ds['start_positions'], shape=[-1,1]),\n",
+ " 'end_positions': tf.reshape(train_ds['end_positions'], shape=[-1,1])}\n",
+ "\n",
+ "\n",
+ "train_tfdataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels)).batch(8)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "0_Jj8Av6rEuN"
+ },
+ "source": [
+ "#### Training \n",
+ "\n",
+ "It is finally time to start training your model! \n",
+ "\n",
+ "* Create a custom training function using [tf.GradientTape()](https://www.tensorflow.org/api_docs/python/tf/GradientTape)\n",
+ "* Target two loss functions, one for the start index and one for the end index. \n",
+ "* `tf.GradientTape()` records the operations performed during forward prop for automatic differentiation during backprop. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "PtZz249vQbLn",
+ "outputId": "24cdf861-af63-4581-a0ae-2de29d1880ed"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Starting epoch: 0\n",
+ "Training loss (for one batch) at step 0: 3.3659\n",
+ "Training loss (for one batch) at step 20: 1.2481\n",
+ "Training loss (for one batch) at step 40: 0.5994\n",
+ "Training loss (for one batch) at step 60: 0.6191\n",
+ "Training loss (for one batch) at step 80: 0.4003\n",
+ "Training loss (for one batch) at step 100: 0.1229\n",
+ "Training loss (for one batch) at step 120: 0.7184\n",
+ "Starting epoch: 1\n",
+ "Training loss (for one batch) at step 0: 0.4000\n",
+ "Training loss (for one batch) at step 20: 0.1253\n",
+ "Training loss (for one batch) at step 40: 0.5118\n",
+ "Training loss (for one batch) at step 60: 0.3670\n",
+ "Training loss (for one batch) at step 80: 0.1260\n",
+ "Training loss (for one batch) at step 100: 0.0788\n",
+ "Training loss (for one batch) at step 120: 0.5340\n",
+ "Starting epoch: 2\n",
+ "Training loss (for one batch) at step 0: 0.3193\n",
+ "Training loss (for one batch) at step 20: 0.0859\n",
+ "Training loss (for one batch) at step 40: 0.4815\n",
+ "Training loss (for one batch) at step 60: 0.2841\n",
+ "Training loss (for one batch) at step 80: 0.0586\n",
+ "Training loss (for one batch) at step 100: 0.0777\n",
+ "Training loss (for one batch) at step 120: 0.0956\n"
+ ]
+ }
+ ],
+ "source": [
+ "EPOCHS = 3\n",
+ "loss_fn1 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+ "loss_fn2 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
+ "opt = tf.keras.optimizers.Adam(learning_rate=3e-5)\n",
+ "\n",
+ "losses = []\n",
+ "for epoch in range(EPOCHS):\n",
+ " print(\"Starting epoch: %d\"% epoch )\n",
+ " for step, (x_batch_train, y_batch_train) in enumerate(train_tfdataset):\n",
+ " with tf.GradientTape() as tape:\n",
+ " answer_start_scores, answer_end_scores = model(x_batch_train)\n",
+ " loss_start = loss_fn1(y_batch_train['start_positions'], answer_start_scores)\n",
+ " loss_end = loss_fn2(y_batch_train['end_positions'], answer_end_scores)\n",
+ " loss = 0.5 * (loss_start + loss_end)\n",
+ " losses.append(loss)\n",
+ " grads = tape.gradient(loss, model.trainable_weights)\n",
+ " opt.apply_gradients(zip(grads, model.trainable_weights))\n",
+ "\n",
+ " if step % 20 == 0:\n",
+ " print(\"Training loss (for one batch) at step %d: %.4f\"% (step, \n",
+ " float(loss_start)))\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Q8ggB0JUWQuW"
+ },
+ "source": [
+ "Take a look at your losses and try playing around with some of the hyperparameters for better results!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 282
+ },
+ "id": "fK91EPvRYFcX",
+ "outputId": "6b7099dd-f918-4905-e3a3-fcce2880e506"
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8jUlEQVR4nO3dd3gc1bk/8O/2VZdlWcW23BvuxmAjCJgEg3EI4LRLSDEhQALX/AIhIYm5CSSQG3MvoaRwKSHEacQJzSR0Y7CNcQFX3DC4ykWSm3rZOr8/ds/smdmZbZJ2Je338zx6bG2d2V3tvPOe97zHoiiKAiIiIqIMsWZ6A4iIiCi7MRghIiKijGIwQkRERBnFYISIiIgyisEIERERZRSDESIiIsooBiNERESUUQxGiIiIKKPsmd6ARASDQRw/fhwFBQWwWCyZ3hwiIiJKgKIoaGlpweDBg2G1muc/+kQwcvz4cVRVVWV6M4iIiCgFR44cwdChQ02v7xPBSEFBAYDQzhQWFmZ4a4iIiCgRzc3NqKqqUo/jZvpEMCKGZgoLCxmMEBER9THxSixYwEpEREQZlVQw8thjj2Hq1KlqhqK6uhqvvfaa6e2XLl0Ki8Wi+XG73V3eaCIiIuo/khqmGTp0KO6//36MHTsWiqLgT3/6E66++mps3boVkyZNMrxPYWEh9u7dq/7O2TBEREQkSyoYufLKKzW///d//zcee+wxbNiwwTQYsVgsqKioSH0LiYiIqF9LuWYkEAhg2bJlaGtrQ3V1tentWltbMXz4cFRVVeHqq6/Grl274j62x+NBc3Oz5oeIiIj6p6SDkR07diA/Px8ulws333wzXnzxRUycONHwtuPHj8fTTz+Nl156CX/9618RDAZx/vnn4+jRozGfY8mSJSgqKlJ/2GOEiIio/7IoiqIkcwev14uamho0NTXhueeew1NPPYXVq1ebBiQyn8+Hs846C9deey3uu+8+09t5PB54PB71dzFPuampiVN7iYiI+ojm5mYUFRXFPX4n3WfE6XRizJgxAICZM2figw8+wK9//Ws88cQTce/rcDgwY8YM7Nu3L+btXC4XXC5XsptGREREfVCX+4wEg0FNFiOWQCCAHTt2oLKysqtPS0RERP1EUpmRxYsXY/78+Rg2bBhaWlrwzDPPYNWqVXjjjTcAAAsXLsSQIUOwZMkSAMC9996L8847D2PGjEFjYyMeeOABHD58GDfeeGP37wkRERH1SUkFIydOnMDChQtRW1uLoqIiTJ06FW+88QYuvfRSAEBNTY1mVb6GhgbcdNNNqKurw4ABAzBz5kysW7cuofoSIiIiyg5JF7BmQqIFMERERNR79FgBa3/y9NqDOHiqDd+oHo5x5bFXFCQiIqKekdUL5f37w+P4y4bDOHiqLdObQkRElLWyOhjJd4USQy2d/gxvCRERUfbK6mCk0O0AALR2+jK8JURERNkrq4ORAjczI0RERJmW1cGIGKZp9TAYISIiypSsDkYKwsM0zcyMEBERZUxWByP5bmZGiIiIMi2rg5FIzQgLWImIiDIlu4MRUTPCYRoiIqKMye5gJFwzwtk0REREmZPVwUg+h2mIiIgyLquDEbVmhAWsREREGcNgBKHZNMFgr1+8mIiIqF/K7mDEFaoZURSg3RfI8NYQERFlp6wORtwOK+xWCwDWjRAREWVKVgcjFosl0viMM2qIiIgyIquDESBSN8KW8ERERJmR9cFIfrhuhC3hiYiIMiPrgxG2hCciIsqsrA9GClkzQkRElFFZH4zku0RmhMEIERFRJmR9MBJZn4bDNERERJmQ9cFIPlvCExERZVTWByORAlYGI0RERJnAYMTFAlYiIqJMYjAiakY8rBkhIiLKhKwPRvKZGSEiIsqorA9GWDNCRESUWQxG1GEaBiNERESZwGCE7eCJiIgyisFIOBjp9AXhCwQzvDVERETZJ+uDkbxwASvAIlYiIqJMyPpgxGGzIsdhAwC0sm6EiIgo7bI+GAEiLeGbWTdCRESUdgxGwOm9REREmcRgBJHpvawZISIiSj8GI4isT8OW8EREROmXVDDy2GOPYerUqSgsLERhYSGqq6vx2muvxbzPs88+iwkTJsDtdmPKlCl49dVXu7TBPUEM0zAzQkRElH5JBSNDhw7F/fffj82bN2PTpk34zGc+g6uvvhq7du0yvP26detw7bXX4oYbbsDWrVuxYMECLFiwADt37uyWje8uYn2aZgYjREREaWdRFEXpygOUlJTggQcewA033BB13TXXXIO2tja8/PLL6mXnnXcepk+fjscffzzh52hubkZRURGamppQWFjYlc01dO+/d+Pp9w7ilotH40eXT+j2xyciIspGiR6/U64ZCQQCWLZsGdra2lBdXW14m/Xr12Pu3Lmay+bNm4f169fHfGyPx4Pm5mbNT0/KZ0t4IiKijEk6GNmxYwfy8/Phcrlw880348UXX8TEiRMNb1tXV4fy8nLNZeXl5airq4v5HEuWLEFRUZH6U1VVlexmJqWQNSNEREQZk3QwMn78eGzbtg0bN27ELbfcguuuuw67d+/u1o1avHgxmpqa1J8jR4506+PriZoR9hkhIiJKP3v8m2g5nU6MGTMGADBz5kx88MEH+PWvf40nnngi6rYVFRWor6/XXFZfX4+KioqYz+FyueByuZLdtJQV5YT6jDR1cJiGiIgo3brcZyQYDMLj8RheV11djZUrV2ouW7FihWmNSaYU5zoBAA3t3gxvCRERUfZJKjOyePFizJ8/H8OGDUNLSwueeeYZrFq1Cm+88QYAYOHChRgyZAiWLFkCALjtttswZ84cPPjgg7jiiiuwbNkybNq0CU8++WT370kXFOcyM0JERJQpSQUjJ06cwMKFC1FbW4uioiJMnToVb7zxBi699FIAQE1NDazWSLLl/PPPxzPPPIOf/OQnuOuuuzB27FgsX74ckydP7t696CIRjDS2+6AoCiwWS4a3iIiIKHt0uc9IOvR0n5FOXwATfvo6AGDHzy5T16ohIiKi1PV4n5H+xO2wwWUPvRSN7RyqISIiSicGI2EDwkWsDEaIiIjSi8FImFo30sEZNUREROnEYCRMBCMNzIwQERGlFYORsOKc0DBNE3uNEBERpRWDkbABecyMEBERZQKDkbCiHBawEhERZQKDkbBI4zMO0xAREaUTg5GwAepsGmZGiIiI0onBSJgYpuFieUREROnFYCRMZEaaWDNCRESUVgxGwopFB1YO0xAREaUVg5GwXKcNANDm8Wd4S4iIiLILg5GwnHAw4vEHEQz2+oWMiYiI+g0GI2EiMwIAnf5ABreEiIgouzAYCXPbI8FIu5fBCBERUbowGAmzWi1wO0IvRweDESIiorRhMCLJcYSyIx0+BiNERETpwmBEkuu0A2BmhIiIKJ0YjEjEMA1rRoiIiNKHwYhEZEY6OUxDRESUNgxGJKwZISIiSj8GIxLR+IzDNEREROnDYETCzAgREVH6MRiRiMxIh5fr0xAREaULgxFJJBgJZnhLiIiIsgeDEQmHaYiIiNKPwYgkl8M0REREacdgROJmZoSIiCjtGIxIcjm1l4iIKO0YjEhEzQg7sBIREaUPgxEJm54RERGlH4MRCWfTEBERpR+DEYlYKK+DmREiIqK0YTAiyXGGXg5mRoiIiNKHwYgkx8HMCBERUboxGJFE2sEzGCEiIkoXBiMSFrASERGlH4MRiciM+IMKvH4ulkdERJQOSQUjS5YswbnnnouCggKUlZVhwYIF2Lt3b8z7LF26FBaLRfPjdru7tNE9RWRGAGZHiIiI0iWpYGT16tVYtGgRNmzYgBUrVsDn8+Gyyy5DW1tbzPsVFhaitrZW/Tl8+HCXNrqnOO1W2K0WAOzCSkRElC72ZG78+uuva35funQpysrKsHnzZlx00UWm97NYLKioqEhtC9Ms12lDc6cfrR4/yjO9MURERFmgSzUjTU1NAICSkpKYt2ttbcXw4cNRVVWFq6++Grt27Yp5e4/Hg+bmZs1PuhS4HQCA1k5/2p6TiIgom6UcjASDQdx+++244IILMHnyZNPbjR8/Hk8//TReeukl/PWvf0UwGMT555+Po0ePmt5nyZIlKCoqUn+qqqpS3cyk5btCyaJWD4MRIiKidLAoiqKkcsdbbrkFr732GtauXYuhQ4cmfD+fz4ezzjoL1157Le677z7D23g8Hng8HvX35uZmVFVVoampCYWFhalsbsK++Ng6bD7cgMe/PhOXT+4bQ0tERES9UXNzM4qKiuIev5OqGRFuvfVWvPzyy1izZk1SgQgAOBwOzJgxA/v27TO9jcvlgsvlSmXTuqzAHXpJWjp9GXl+IiKibJPUMI2iKLj11lvx4osv4u2338bIkSOTfsJAIIAdO3agsrIy6fumA4dpiIiI0iupzMiiRYvwzDPP4KWXXkJBQQHq6uoAAEVFRcjJyQEALFy4EEOGDMGSJUsAAPfeey/OO+88jBkzBo2NjXjggQdw+PBh3Hjjjd28K91DZEZYwEpERJQeSQUjjz32GADg4osv1lz+xz/+Ed/85jcBADU1NbBaIwmXhoYG3HTTTairq8OAAQMwc+ZMrFu3DhMnTuzalvcQMZumhZkRIiKitEgqGEmk1nXVqlWa3x9++GE8/PDDSW1UJolhmhZmRoiIiNKCa9PosGaEiIgovRiM6HA2DRERUXoxGNFhASsREVF6MRjRyXeF28FzmIaIiCgtGIzo5LtZwEpERJRODEZ0WDNCRESUXgxGdAqk2TQpLttDRERESWAwoiOGaYIK0O4NZHhriIiI+j8GIzo5DhtsVgsAFrESERGlA4MRHYvFwi6sREREacRgxEAkGGERKxERUU9jMGJAbXzGYRoiIqIex2DEALuwEhERpQ+DEQOsGSEiIkofBiMG8t2hlvAtHKYhIiLqcQxGDHCYhoiIKH0YjBgo4GwaIiKitGEwYiDfxdk0RERE6cJgxIC6WB6DESIioh7HYMSAWsDKmhEiIqIex2DEgDpMw5oRIiKiHsdgxEAhO7ASERGlDYMRA/luNj0jIiJKFwYjBiLDNAxGiIiIehqDEQMiM9Lq9SMYVDK8NURERP0bgxEDheHZNIoCtHmZHSEiIupJDEYMuOxW2K0WACxiJSIi6mkMRgxYLJbIUA3rRoiIiHoUgxETogtrM4MRIiKiHsVgxES+K1Q3wmEaIiKinsVgxEQBp/cSERGlBYMRE+pieWwJT0RE1KMYjJjIZ0t4IiKitGAwYkJ0YWVLeCIiop7FYMREQbjxWTOHaYiIiHoUgxETRTmhYKSpg8EIERFRT2IwYkIEI80MRoiIiHoUgxETzIwQERGlR1LByJIlS3DuueeioKAAZWVlWLBgAfbu3Rv3fs8++ywmTJgAt9uNKVOm4NVXX015g9OlODcUjDS2MxghIiLqSUkFI6tXr8aiRYuwYcMGrFixAj6fD5dddhna2tpM77Nu3Tpce+21uOGGG7B161YsWLAACxYswM6dO7u88T2JmREiIqL0sCiKoqR655MnT6KsrAyrV6/GRRddZHiba665Bm1tbXj55ZfVy8477zxMnz4djz/+eELP09zcjKKiIjQ1NaGwsDDVzU3KkTPtuPB/34HLbsXeX8xPy3MSERH1J4kev7tUM9LU1AQAKCkpMb3N+vXrMXfuXM1l8+bNw/r1603v4/F40NzcrPlJt6LwMI3HH0SnL5D25yciIsoWKQcjwWAQt99+Oy644AJMnjzZ9HZ1dXUoLy/XXFZeXo66ujrT+yxZsgRFRUXqT1VVVaqbmbJ8px1WS+j/HKohIiLqOSkHI4sWLcLOnTuxbNmy7tweAMDixYvR1NSk/hw5cqTbnyMeq9WCQtaNEBER9Th7Kne69dZb8fLLL2PNmjUYOnRozNtWVFSgvr5ec1l9fT0qKipM7+NyueByuVLZtG5VlONAY7uPwQgREVEPSiozoigKbr31Vrz44ot4++23MXLkyLj3qa6uxsqVKzWXrVixAtXV1cltaQYU53B6LxERUU9LKjOyaNEiPPPMM3jppZdQUFCg1n0UFRUhJycHALBw4UIMGTIES5YsAQDcdtttmDNnDh588EFcccUVWLZsGTZt2oQnn3yym3el+3GYhoiIqOcllRl57LHH0NTUhIsvvhiVlZXqzz/+8Q/1NjU1NaitrVV/P//88/HMM8/gySefxLRp0/Dcc89h+fLlMYteewv2GiEiIup5SWVGEmlJsmrVqqjLvvzlL+PLX/5yMk/VK4gurAxGiIiIeg7XpolBzYy0e3v8uf69/ThuX7aVPU2IiCjrpDSbJlukc5jm//19KwBg4uBCfPui0T3+fERERL0FMyMxiGDkTBpn05xu6/ksDBERUW/CYCSG0YPyAQDr9p3CR3XpaUlvs1jS8jxERES9BYORGM4ZUYLLJpbDH1Rwz0u70vKcVgYjRESUZRiMxHHnvPEAgG1HGtPyfIxFiIgo2zAYiUPUjfiD8ac1dwcLoxEiIsoyDEbisNtCL1EgqCCYhoDEyliEiIiyDIOROBy2SHTgCwZ7/PlYM0JERNmGwUgcDlvkJfIFej4zYmNqhIiIsgyDkTg0wYi/5zMjTIwQEVG2YTASh81qUes4fAEO0xAREXU3BiMJENkRHwtYiYiIuh2DkQQ4RTDSQ8M0ASnIsYDRCBERZRcGIwmwh2fU9NQwjV+apcNRGiIiyjYMRhKgDtP00GwaOTPC2TRERJRtGIwkIBKM9FRmJBKMsICViIiyDYORBDjtPRuMBAJyMNIjT0FERNRrMRhJgD0cIXjTkBkhIiLKNgxGEiCGafw9VDMiF7AyLiEiomzDYCQBjh4eppGDnKDCaISIiLILg5EEOKw9O7VXnk0TYGqEiIiyDIORBPT01F65ZoSJESIiyjYMRhLQ08M0cjaEwzRERJRtGIwkoKeHaVjASkRE2YzBSALEMI03DR1YmRkhIqJsw2AkAWKYxt9DmRG5FkVhMEJERFmGwUgCHD28UJ42M9IjT0FERNRrMRhJgMPa07NpIkEOp/YSEVG2YTCSAIc9fZkRDtMQEVG2YTCSgHSu2svECBERZRsGIwno6aZnAbaDJyKiLMZgJAGigNXrZ58RIiKi7sZgJAHqqr3Bnh+mYc0IERFlGwYjCVCHafxsekZERNTdGIwkoKf7jPgDLGAlIqLsxWAkAWpmpIciBTkzwj4jRESUbRiMJCAyTMOaESIiou7GYCQBPT5Mw9k0RESUxZIORtasWYMrr7wSgwcPhsViwfLly2PeftWqVbBYLFE/dXV1qW5z2kVW7U1HzQijESIiyi5JByNtbW2YNm0aHn300aTut3fvXtTW1qo/ZWVlyT51xqhTe3uq6Rk7sBIRURazJ3uH+fPnY/78+Uk/UVlZGYqLi5O+X2/Q88M0rBkhIqLslbaakenTp6OyshKXXnop3nvvvZi39Xg8aG5u1vxkUk+vTRPQ1IwwGCEiouzS48FIZWUlHn/8cTz//PN4/vnnUVVVhYsvvhhbtmwxvc+SJUtQVFSk/lRVVfX0ZsbU02vTcKE8IiLKZkkP0yRr/PjxGD9+vPr7+eefj/379+Phhx/GX/7yF8P7LF68GHfccYf6e3Nzc0YDkh5ftVcuYGU0QkREWabHgxEjs2bNwtq1a02vd7lccLlcadyi2NJZM8JhGiIiyjYZ6TOybds2VFZWZuKpU9LTwzQB9hkhIqIslnRmpLW1Ffv27VN/P3jwILZt24aSkhIMGzYMixcvxrFjx/DnP/8ZAPDII49g5MiRmDRpEjo7O/HUU0/h7bffxptvvtl9e9HDenyYhpkRIiLKYkkHI5s2bcKnP/1p9XdR23Hddddh6dKlqK2tRU1NjXq91+vF97//fRw7dgy5ubmYOnUq3nrrLc1j9HY9PUwT0Ezt7ZGnICIi6rWSDkYuvvjimL0wli5dqvn9hz/8IX74wx8mvWG9SXpn0zAaISKi7MK1aRLgsPf0bBr2GSEiouzFYCQBDms6Z9P0yFMQERH1WgxGEiCGaYKKtr6ju2jWpmE0QkREWYbBSALEMA3QM9kR1owQEVE2YzCSADGbBuiZYCQQ4DANERFlLwYjCXBYIy9TbVMn7v33bhw5095tj8/MCBERZbOMtIPva6xWC2xWCwJBBVf9bi06fUHsO9mKP39rVrc8vtyBlbEIERFlG2ZGEiSGajp9ocBh9/GmbntsZkaIiCibMRhJkDxUAwCjBuV322NrVu1lMEJERFmGwUiChpfman5vavd122MH2GeEiIiyGIORBC29fhaunDZY/f1Mu7fbHtsvr9rLaISIiLIMg5EElea78NtrZ2DD4ksAAA1t3phr9CQjwJoRIiLKYgxGkjQgzwEgVHTa3OHvlsdkO3giIspmDEaS5LLbkO8KzYjurqEauYC1u7ItREREfQWDkRSI7MiZNk+3PJ6mZoSxCBERZRkGIykoyXMBAM60dc+MGtaMEBFRNmMwkoKS3MQyI6/vrMNVv1uLAydbY96ONSNERJTNGIykINHMyEvbjuHDo01YtfdkzNvJmRHWjBARUbZhMJKCkgRrRjp8AQBApz8Q83ZyZiTA1AgREWUZBiMpEJmR022xZ9N4wuvYiPVszPgDcgErgxEiIsouDEZSIDIjDeFgpKHNi7d212uCCiCSEfH4Es+MMBYhIqJsw2AkBZGakVAwsuS1Pbjxz5vw1p56ze061cxI7GCEs2mIiCibMRhJQXF4Nk1jR6iAtbapEwBwrLFTcztPODMSd5iGs2mIiCiLMRhJwYBc7TCNyHy0ebTt4dWakTgFrMyMEBFRNmMwkoLiXCcAoLnTj0BQUWfNtHm1wYgIUmIN0yiKopva291bS0RE1LsxGElBUY5D/X9Thw8dXpPMiD/+bBq/blyGU3uJiCjbMBhJgcNmRUF4sbyGdq8abLR7tBmQRDIj8iJ5AIdpiIgo+zAYSVFxeHpvY7tPHaZplTIj/kBQzXp0+s0zI76g9jomRoiIKNswGElRcU6obqSx3asO07R7IxkQjxSAxOozos+MsB08ERFlGwYjKRLTextMMiPy0EzsYRp9ZoTBCBERZRcGIykaEJ5RU98c6S3SLs2mkTMjsQpYfUF9zUh3bSEREVHfwGAkRSIzcryxQ72szWOcDYnVZ4SZESIiynYMRlIkeo3UNUUyI9phGjkzYh6M+KJqRrprC4mIiPoGBiMpKg73GjneZDZMI2dJgqaFqX7dbBr2GSEiomzDYCRFA8JTe2ubIsM0voBiuh6Nx2R6L/uMEBFRtmMwkiIxTNPY7tNcLhqfeXR1Ih6TIlZfuGbEbrUASH2YpsMbiKo/ISIi6gsYjKSoWGoJLxN1I/rMiFkRq2iM5rSH3opUMiPtXj8u/N+38dXfb0z6vt2p0xfAH9YexMFTbRndDiIi6lsYjKRITO3VE43P9JkRsyJWkRnpSjBytKEDp1q92H60Men7dqf/W7Uf9728G5/+1aqMbgcREfUtSQcja9aswZVXXonBgwfDYrFg+fLlce+zatUqnH322XC5XBgzZgyWLl2awqb2LmbBiMiM6IdlzHqNiJoRp00EI8lvSyQAMi+UTYdNh85k7LmJiKjvSjoYaWtrw7Rp0/Doo48mdPuDBw/iiiuuwKc//Wls27YNt99+O2688Ua88cYbSW9sb1KYY1frPGRiRo1+WMYsMyJm04jMCJB8S/h2j3GztXRz2ZloIyKi5NmTvcP8+fMxf/78hG//+OOPY+TIkXjwwQcBAGeddRbWrl2Lhx9+GPPmzUv26XsNi8WCAXlOnGzxaC5vU2tGEh2m0daMAKHsiC06zjGlXxPH7bAlfudu5GQwQkREKejxo8f69esxd+5czWXz5s3D+vXrTe/j8XjQ3Nys+emNBuZFD9WILqxRwzRxpva67JEAItleI+1SoBNrUb6e5rRnJggiIqK+rceDkbq6OpSXl2suKy8vR3NzMzo6Ogzvs2TJEhQVFak/VVVVPb2ZKSkxCka6YZgm2SLWDpM1cdKNwzRERJSKXnn0WLx4MZqamtSfI0eOZHqTDBkGI2aZkTjDNC6bXDOS3HZoh2kymRlJve6FiIiyV9I1I8mqqKhAfX295rL6+noUFhYiJyfH8D4ulwsul6unN63LjIdpjDMjZk3P/IGuZ0bkYCTWCsE9Tc6MdPgCyHX2+MeLiIj6gR7PjFRXV2PlypWay1asWIHq6uqefuoeV5IXCZjExJo2rx9/WHsQf91Qo7mtWdMzX9CogNU4GGlq9xlmHMzWxEk3h5TdkVcwJiIiiiXpYKS1tRXbtm3Dtm3bAISm7m7btg01NaGD7+LFi7Fw4UL19jfffDMOHDiAH/7wh/joo4/wf//3f/jnP/+J733ve92zBxlUkh/JjAwqCAUm+0+24b6Xd0fd1rRmRGRGbNrZNHpbaxow4743seS1j6Ku0wzTZDAzIq+zIwdIREREsSQdjGzatAkzZszAjBkzAAB33HEHZsyYgbvvvhsAUFtbqwYmADBy5Ei88sorWLFiBaZNm4YHH3wQTz31VJ+e1ivIwzTDSnIBALuONWluYwlnTOI1PXPEqbf48GgTggqw7Uhj1HUd8jBNBjMj8grErR4GI0RElJikB/UvvvjimMWJRt1VL774YmzdujXZp+r15C6sw0ry8MGhBpxu82puU5TjQGO7z7yANZhYZuRM+HGbO3xR1/WWzIhPkxnhMA0RESWmV86m6SsG5kdnRvTEgnpx28HbI13OjPqMNLQnFoxkMjMSYGaEiIhSwGCkC+SpvUMGGM8MKhLBiNmqveGaEbvVClu4CtYo86RmRjqjD/KaAtbeUjPSzwtYT7R0Yu5Dq/Hkmv2Z3hQioj6PwUgXyMM0OQ6b4Vo1hSIYMRm2ELNp7DaLOiPHaJhGZEZaPX41gBH07eAzxSdteJtBAetjq/bj5Q+Pp3OTeswHBxuw70Qrlm/tH/tDRJRJDEa6wCYFHzlOK0rzo3ujiCGXNq8fb+yqw5u76jTXi8DCYbPCEq52NZrae6YtMjzTosuOaApYM9gOXg6S2nTDNIdPt+F/Xv8IP1m+M92b1SNaOkPvR3Nn9LAZERElh8FIF9139SR88eyhmDOuTJ3eC4Sm+o4szcP8KZUAgJMtHnznL5vx7b9s1tRTiKJPu1XOjBjUjEiFsfoDYLuvd7SDj1XAKgp7G9t98AUyt43dRQSETQY1PERElBwGI130jeoRePA/psFmtaBMCkYe+9rZeOcHF6M8fFl9c2R139Otkf+L6bB2mxVWi6gZ0T6Hoig40y4FIx3mmZFUm555/UH8+PkP8dqO2pTuD2in9uozI3I2p7G97x/ARUDY6vEjmOTChkREpMVgpBuVFUaCkbICNwAg3xWaPV3f3Kled0oORkSfEatFDUb0mZF2bwBeKeOhPxuXu52m2g7+mY2HseyDI7jlb1tSuj+gLWDVByPyLKCmDu30575IBFeKEj1sRkREyWEw0o0GSTUjIjDJDQcjfuns+WRLJBhRh2lsVrVBmv5E+4yud4k8TBMMKujwdT0zcqzReAXlZMjDL226YRr5gN3QjzIj+v8TEVHyGIx0o0GFoWxIgdsOt8MGAMh32aJuJwcjYmjDYYtkRvR9RqKCESnLoJ8ynGpmRK73SJUccOnbwcsH7P4wTCMHV6wbISLqGi6r2o1EzYhcO5Lnin6JNcGIVMBq1Gfk/tc+wuOrtb0s5IOfvlA01QJWbzcUlcqzaVo9+syIHIz0h2EaKTPCYISIqEsYjHSj80cPxKfGlOKz4Rk0AJDrNAhGWiMHYzG0ESpgDV0mJ0b0gQigzTJ06IORFKf2+rphFo5mNk0/L2BlZoSIqPswGOlGBW4H/nrjbM1lec54wzThAlabJarPiNkaQPJsGn1zsc4Ugwp/N8wI0cym0QVJcvagsR8VsAKsGSEi6ioGIz3MbrPC7bBqajlOtsoFrJF28Po+I0at34E4wzRJZkb21DZj+dZjUQv8pSLWbJr+XMDKzAgRUdcwGEmDPKcdnb7Iwf6UUc2IVMAqEiLydGAAmDS4ELuON8cepkkyM/LoO/vw8ofa3iKKoqhZmmT4pMxIrALWpj4ejCiKos2MdHBqLxFRV3A2TRroi1hPtnrUIZjIbBprVJ+R2qZQMDJqUB7+9K1ZuHnOaADaIQ99ZiTZdvANBsWkqc6skTMj+iBJUzPSx4dpOnwBzYwnZkaIiLqGwUga6IMRrz+oDsHI7eBFMkIc6OrDwUjVgFzMGTdIXftGO0wTehynzao+djJaDYaCUm3Xrilg9QU0NS+aYZq2vn3w1jc5Y80IEVHXMBhJg1hFrMaZkdBt6sLDNBXh/iVF4RWAT7V61RbkIksi1sUxyow88tbH+M5fNhkGKi2e7gtG5AJWRdEOGWk7sPbtg3eLLvjo6/tDRJRpDEbSwKjXiFifRq4Z0fcZEcM0FUWhYGR0WR4KXHY0dfiw9UgDgMiBUAQj+pqRM21ePPLWJ3hjVz3WHzgdtR1GrcxT7Tni1w3viKGaYFBBq1cuYO3bwzT6wmL2GSEi6hoGI2mQbxCMiFoPeTaNvh28KGAVwYjLbsPcieUAoBadigNjmUkw8vZHJ9T/HznTHrUdxsM0qdWM6DMqok19i8evWfyv3RtIuW19b6AP4JgZISLqGgYjaZArDdOI7IcIRuQ+I/oC1rom7TANAFwRbqj26o5aePwBdWaKWAtHP0zz5q469f/7TrRqrvMHgpp1bYRk607Ux9P1KlGDkfCwRmgfQ9f15QO42J8CdyjINJuCTUREiWEwkgbyMI3IYIgDtV9aKE/uM3LoVBuONIQyGSIzAgAXjitFca4D9c0eLPrbFpwJD3mIVYL9QUVty97pC2DNJyfV++4/qQ1G2jzG2YlUakYURVELbx220I6IYRox9bUox6G+Fn15pVuxP0OKcwD07cCKiKg3YDCSBvlGwYhXzKYRwzSRzEhdUyeu+t1atHT64XZYUVWSq97fZbfhd9eeDZfdirf2nMCK3fWaxwUiNR/7TrRqmq3pMyMtHuODaCqZETkrUuAOFdrqMyMFbgdc9tBHLtUi2d5A7I+o0/H6g6bdcqn77DvRiq/+fgPW74+ufSKivo3BSBrkSiv3igNY9DCNVW00tnLPCTR3+jFiYC5euOWCqJqTT40txWcmlGkuGyQFIyIAOXS6DUCoTwkQKohtlWbPmGUnUgkU5OLVwvDwhciMiI6rhW57ylOQgeR7qPQU8RoOzHOql3XHqscU281/3Yx1+0/j2t9vyPSmEFE3YzCSBnIwIXqFRBWwSvUU74aHVq6aPgQTBxcaPqacCQGA4lynOjwiikMPnQoFI9OrilGaHzpwHpCGaloNpvWGtin5A6vcfVVkRtq9Aew+3owfPf8hAGDogFw4UsyMrNp7Amfd/Tr++N7BhG5/5Ew7vvLkerzz0QnsO9GKPbXNST1fLGJ4qzhXDkb6bqanrzjW0JHpTSCiHsJgJA3ywiv35jhsatGjvmbEYY30GREFkReMHmj6mIN0wUhRjgNuRygDI3qYHDwVqjkZOTAPowflA9AO1RjNpAHiZy18gSD+tf24dsG/gDxME9rHTl8Af9lwGE0dPpxVWYgfz58Ah5oZSS7guf0f26AowM//vTuh29/53HZsOHAG1y/9AHMfWo35v34Xjd00pVg0mhN9XwAGI+kQ4FAYUb/FYCQN8sLDNPluO3LCgYk4oIlGYXabBVZrZD0Yt8OK6cOKTR9TH4wU5thRPSoUvDz1bih7IIZpRpTmYUyZNhh5YctRPLf5qOFjxzuw/nv7cXz371ux5LU96mWiaNZiAXLVfQzgaLgI94ZPjURVSW5kmCbOcwSDCr7xh4245on12HjgNHId0Y3jYqlv9kRdpq+ZSZXIahW47ep07FR7s1Digt2wsjQR9U5cKC8NxLBFgduuTvNt94bapfs0C+VF7jNz+AC47OYHYDF7Rih0O3Db3LF4c3c9/v3hcXz3kjHqMM3I0jycCjdZ23eiFSeaO/H9Z7fD7EQz3oF19/HQkMfeuhb1Ml8wkuER+9jhC6i9TYYOCM08UYdp4mRf6po78e4npwAA1/5+Q8zXwojcDVbojpWJgUggme+yw2GzwusPsmYkDYLMjBD1W8yMpMHM4QNw2cRyfPvCUeqBulO32Jo8TAMA5YXuqMeRyZkRl90Kt8OGSYOLMPesMigK8Kd1h9WD74jSyDDN/pOt2FvfYhqIAPEzIwfDQU6N1ETNL9W+5ISzGG0eP441hsb5xYwgZ7iuJd5zyAsABhUY9kOJJWAQHJzppmBE1IzkOG1qpidecEVdx8QI9Xav76zFgkffQ83p6AaTFBuDkTRwO2x4cuE5+MqsYeqBut0b0EyH1WdGCt0O/cNoyMGIqNEAgCunDQYA/GXDYfV2+S67Okxz+HS7mtkwk2gw0tLpV5uuyQv+5YQDrkOn2+ALKLBbLWrjNqc9sWEa/aq/skSm0eobsAHAqZbooZtUiMxIntOuFg2zZoSyndcfVE8+stXNf92CbUcacedz2zO9KX0Og5E0k+sp5AOYPLUX0AYYRuRppfKB99MTymCXopoJFQUAgMoiN/KcNviDCt7aUx/zseUCVv2B3x8IajIi4v/ygn8iGPmkPlSjMbg4R+0860hwam+b17wpWluMQEUwSunXt3TGvV8iRNYm12VT94fDNJTtrnlyPS64/21sO9KY6U3JuBqDpTcoNgYjaZbjDL3kHd6AZgZKqOlZ5HbxghG7LfLWeaTGZoVuByYPKVJ//9HlEwAAFosFo8PZkQ8ONcR8bG94u7bWNODs+1Zg2fs16nVHGzo0wY8ajEi1LyL788mJUE1JVUmOenunycHb6w9qOpnGyow0JDDcEjDIjNQ1dVdmJLRtocxI32ri5vUH8d6+U72mZwv1H1trGgEAL24xLozPJn25w3SmMBhJsxxHZDaN3JvDJnVgBSJFr4nQLzp3x6XjUF7owkP/MU0TmIwJ143o6QOf1XtP4tF39uHul3ahod2HH7+wQ71ODNEIIhiRF/yL1MWELqsaEOkgKwpYvbptXvDoe7jg/rfV6bftMYKRLz++Ht/9+1bT6wHjYZoTuszIx/Ut2FITOzAzIrI2uU6bOuzUV4KRX725F197aqPa+6W7sTU+uZOc+dYfmfVwInMMRtJMnWkiZUYcNgssFos6lAHEz4zI9Mfdi8YNwsa75uILZw/VXD51aBGMnFWpbaz21p56PPDGXuw41hR1W7NgRF7wT/9lJGbSAMaZkWONHdhd24xWj1+dodMeY5imrrkT/9p+HG0x/uCNpoGKhQeFrz21EV95YgOaO5M7gEaGaSI1I31lau+Taw4AAF7adrzbH/updw9g2s/fNJ0yTtlBLPnQl5xp82L9/tNc1iGD+t6npo9Tp/b6IsGI3Rp6GywpZkYSde3sYeribrJvnDccj399proisJE3dtXhzV11OHAqVAciOsAe0WdGbFbNKsUANGvriGDkVJsHf3+/Bo3tXs1aI+KrIJHZM7HOPgwLWFs96qwffyCIky0eeAPBhIZ9BF8gqNa75DlZMyL7xSuhvjM/eJbFe31FU7uvW87i5WFRVx/MjMx7ZA2u/f0GvL6zLv6NqUcwGEmzHKnPiE9qeAZAN5smfmbku58ZAwC4c974hJ7bZbfhX7degC+cPQS/++oM9XKPP4jLJ1dgYL7T9L7f+ctm3PK3LWrvj0snlgOAurKwX55NE5UZkYdpQjv5xOoDWPzCDix6ZgvW7T+lXi8yImYrCstaYmQ0jGpGggpwqjUUeHRKBbSeJKblysNHOXIwwqm91Md0+gKYdu+bmHzPG13OCMjdnPtiZkR0kxYLj1L69b1PTR8nZtN4/ZEzbHFAS7Zm5Pa54/DWHXPwnxePTvj5B+a78NB/TMfnpg7GxPDwzEVjSzXbYSYQVHA4PH9eLNQnenfIs2ncUZmRSDZG/xzv7TutyYyIIKQjxjCN0ByjSMwoMwKEhnhCjx8JKpIp5hTBkt1qgdNmlYad+lYw4ozzXlP/V98cGbZMZIZaLMkOdfZW8lB5KuThdRaJJ4cdWNNMHsIQFddiKq78Z5BIZsRqtaj9Q1KxfNEFaOn0YWB48T5ngmc0BS47plUVAwjtgz8Q1HSSlVu3u+xWDMqP9EQxeo5aqZZDHOxjFbAKZmvrxHKmLXQGJH9RdPqSz4zkOm2wWCxqVquv1IwI+UnUJFHyFEXRDLv2di2dvqjVwZMhByPJZBp7m64GIy67DS0IfS+dbvMaDouTMZ4epZnLblXXM2kOzzwQ2QK5TqInakb0nHarGojI2xHPpCGFKJYWiWvu9GsW/MuRAq6hA3I0X8rxzshFZqQ9gbOKVKbPicBDfq31s5H0Trd61ILY9vD25YW/uPtSzYhc1NuVAw/FdrLFg/OWrNSs3ZQsXyCIf+sWo+xuctDQ1amo8v37WpZQZu1iMCLv++nWnnvv+qOUgpFHH30UI0aMgNvtxuzZs/H++++b3nbp0qWwWCyaH7c7dqvz/sxiidRUHG8KdSsUC+nJs0PcjvTHiaJVezxThxbDbrOqKcnGdq9mwT85+yMXr4aeI/Z+qZmR8GtRkmdexxKrZsSMyIhoh2m0X54vbTuG/1u1DwCw+fAZzPzFW7j171sARKb1ioArHX1GPqprxtL3DhrWwSRDnnbLYKTr/IEgHlu1Hx8cOqO5fMexRtQ3e/D2nhMpP/YTq/fj//19K655cn1XN9OU3J+ouYtTsuVgJF5Dw97M1sVslrzvp1u7Z/mJbJH0N9I//vEP3HHHHXj88ccxe/ZsPPLII5g3bx727t2LsrIyw/sUFhZi79696u99KX3ZE3KdNrR7A9hyONTjYlx5qEuqPG6bidcoXmakNN+JU61etXdJca4DLZ1+NHb4pGEaq2ZqrzytF4j0GTHT0unH1poG9cBZkuc0XVMmlbM5kRHp8BnXjLR5/Lht2TYAwBVTKvHE6tBU2Fd3hKrsO6SGZwDgtPd8O/jLH3kXQOisbWH1iJQf53Rb5Eytry061xtX7F1/4DT+5/WPAAC//sp0XD19CADj7FuyXth6DABw4GRbnFumrlPKCHY9MxIJZvp0MNKNmZFTzIwkJenT74ceegg33XQTrr/+ekycOBGPP/44cnNz8fTTT5vex2KxoKKiQv0pLy/v0kb3deKseku4Y6Ho8xGrb0Y6GAUjZQUufPuiUXj862fjR5dPwOemVuLSs0LvX3FOKGvR1O5DQGRGdLNp5IZnZs8h++N7h/D5/1uHd/aeBACU5GozI3+47hzMn1wBAGgxeb1izQwwHqaJfIHoi2n12Qi54Zm8P+n4Al4dfk1SdUo6U4u3vZsPn8Gf1h3qNX0XemNNTnNH5PP30+U71f+L4LYrBYzp+C7QZEa6WIAqZ1b62jCN/DfelWAkGFQ0hfPdtUp4tkgqM+L1erF582YsXrxYvcxqtWLu3LlYv948ndja2orhw4cjGAzi7LPPxi9/+UtMmjQp9a3u43LDXVhFw7CzKkOZkUx37TMqLi3Nd+Guz56l/v7lc6rU/xfnhupG/rD2INbuC03PtVst6owhwGCYJk5mRH/QGZCnrZ25YEwpNoczSmbDNLEK6MQBwmOSGVnzSeSA3+ELwKcLRjJZM3KmvWtfbnLaOF6R4RcfC/095zptmvc8U/Tb2x0Foh/VNaMk14myOCtkm/EGIp+b5nAht91mjQS8XZihksjU9q7ydGtmRBqm6WPBiPz335VgRL/fXa0ZOdXqwU+X78TXZg/Hp8IzHvuzpDIjp06dQiAQiMpslJeXo67OuFnM+PHj8fTTT+Oll17CX//6VwSDQZx//vk4etS8S6PH40Fzc7Pmpz/J0U19nVDROzIjRvUcsWo2isJFrCIQAUIHZ7nPgH6YxqwuxayWRP/8LrtVnQli9gUa6yDgiTNMs+bjk5rLA0HtF0x7BmpGBLPhqsTvH/lyTPSA8WJ4uCBZ3Z1R0RcZpxL8KYqifjb21Dbj8kfexZefSL4m48WtR3HV79bi4CntYmgiYOqUPmOpvg7pODGRa6W6HIx45JqR3pFNS5T899+V+FYfMDe2dy3b9L1/bMNrO+vw9T9s7NLj9BU9XiVZXV2NhQsXYvr06ZgzZw5eeOEFDBo0CE888YTpfZYsWYKioiL1p6oq82dm3Uku8CzKcaCyKHRmlulhcdGQTBYrGBGZEZndZoHVasGkwYUoyXNGTT02G6apLDY+O5Wf32kPrWwsZhqZZUY6Y8yOEc3OOrzRTc9qmzpw6HTkABNaWVk/TCNqRkLvoQiu0hKMdLEgTh6m8SQ4hCAWP0tWd0/v9OiKjP3B5B//B89+iLPufh0HT7Xhz+sPA4DaN8dIY7sX6/afigoovveP7fjwaBN+s/ITzeVi6Et8/oJK784SyAFeV4dpNDUjcfb58dX7cduyrXjq3QNdes7uIp+YBLqQ4dR/B3R28W9ANJjMFkkFI6WlpbDZbKiv13apq6+vR0VFRUKP4XA4MGPGDOzbt8/0NosXL0ZTU5P6c+TIkWQ2s9eTMwcTKgp6TUGvUaAQMxjJib5OtLZ//pbzseaHn9YM2QDmwzSDi4zn4w+Qakbc4fuKHixmZ4+x+oaIM2NNzUj4//XN2rRqu9cfVTPSrtaMaIdp0nHQMauRSdTpBDMjfum6Dl8gak0fI/oDdndn+fTBTSqZkefDq8k+vfYg3j94Os6tga88uQFf/f1G/Gt7Yuv4RDIjkW3t9PbeYESbGelizYg8tTfGQfjw6Tbc/9pHeGnbcfzilT3qchKZJL8OXTmp0NdhdaVmSP57ciQ4y7GvSyoYcTqdmDlzJlauXKleFgwGsXLlSlRXVyf0GIFAADt27EBlpfk6KC6XC4WFhZqf/sQuHfS/qFvMLpOSDUaKcqIzI+IPx+2wGU4fNcuMDDZpDiQ/v1jzoiDOME2sLwGj4kJxBqNfcbbTF4jq5CrG8tUC1nCA5O9izcjJFg9ONEcf9PVfjrEWEIxHDra8/qDpEIK+XuG9ffHP0PTBQiJN65KhH6bxJ3DQONXqwQ+f2x61MnOrx4/90iwVs9fho/Cijc9uSmzhP3EwkrNOHdLn7S8bDnd5qK07dWfNiFzAGivQrdUFtr1hlWf5u8Cb4t9xbVNH1L51JTsoZ2hHm6y23t8kPUxzxx134Pe//z3+9Kc/Yc+ePbjlllvQ1taG66+/HgCwcOFCTYHrvffeizfffBMHDhzAli1b8PWvfx2HDx/GjTfe2H170cd8fsYQjCrNw8PXTMN/nBs9BGXv4vSyVMlZixyHDQ6bBeeMGGB6+yKTYZqYz2ESjAxJYJhGZJTyXWKYxo99J1rw4+c/xE5pheGYwYg6TBOdGdH3Wmj3RteMqFN7owpYU//iaen0Yc4D7+Czv3k3atv1B/XjjR0pPUenL6CZKRRUzFvmt3i0r0OdQZBk9Pgys6zVkTPtKU15TCYzcqrVgy01Dfif1z7CPzcdxRf+b53mevl1MHpsvUTrN8TBXX4tPq5vwXv7TuGP7x3CT5fvxGUPr477OHLA2ZPrvHRrzUiCTc/0732mWqY3tHnx/X9ux4YDpzVBWSp/x52+AC57eA2++Ni6qMtTJYr0AaCXTGjrcUn3Gbnmmmtw8uRJ3H333airq8P06dPx+uuvq0WtNTU1sFojf0ANDQ246aabUFdXhwEDBmDmzJlYt24dJk6c2H170cd8dkolPmuwQu4DX5qKn760E09+45wMbJU2ULj1M2Nw44Uj4bKbr8BZbJAZideYy3SYxiQzMjAv0iHWrcuMHDzVhrkPrQEQmkb3+4Wh181omMZps8IbCGqKCwVxe/1ZWoe0srKgn9rbHTUjz246inZvAO3eAI42dGjqbPRfaKHrC5J+jtUfn0Srx48BuQ40hAvrvP6gYaZKf/BNJBujf82NhmmaOny47OE16PQHsOfeyzX9aOLR14zEer1vW7YV7+07jVKThR/1wZXHF4y5LYlmo4yGaW780yZ4A0GMHpQHIFS309zpQ2GMDsty9iTRrsip0GZGuq9mJFZwd6pFPxSamWDkrT31eH7LUTS0e3HDp0aql6fyd3y6zWsYzHUlMyKfXMXrEN1fpPRJv/XWW3H48GF4PB5s3LgRs2fPVq9btWoVli5dqv7+8MMPq7etq6vDK6+8ghkzZhg8Kn35nCrs/Nk8XDRuUEaeX/7iy3fZYwYiAFCcG/1lf7wx9lm0eQGrcTAiF8mKs8QCg3VVNkldMI0KWMUMHONhmtD/o4IRr3aYxh8Iatamkfdnb10LHl+9P+lhlEBQwdJ1h9Tf63UHSv3MoGMpZkZe+bAWALBgxhD1MrNeI/o1f/QHjEBQ0Yz1//C57fiGruJfBDSf1Lfgz+sPwRcIorapIzzDJFK/kSh96t8sqwMAR86EXqNEp8cafV7kJmvy48SaHePRFbACke2W77ZiV+yVYeVgpCf713RqOrCmlhnxB4L44XPbNcNesbb5lK4IuyuN4WSN7V786o292HeiNaHbi+ChpdOn+RtLNTNiJNEicSNycNOX1/pJBtem6WXsPXgmFI9cKJWXQLtwo9k08Q6WZpkRs2EaeTsimZHo521o96lf4kZfAgW6YET+Emzq8GHV3hNR64CEhmkiRxFvIKj2DhCFtSIY2VLTiPtf+whv7DKe4q6nKApOtHRi1/Emtd8MgKhiUf2X9Sf1iX3Z6p9r1d5Qa/LPTR2sDgOafcnpMyP6gOh/Xv8IF/7vO/j39uPo9AXwz01H8YnuICACmEsfXoO7X9qFf207rsluPLnmQELt7YNBBYqiRL2nsWpG9NOvjfZBZnQwkYuFzRrk6RkN0wjy38UrO2pNHwPQNsvyBsxre7qqOzIj24824Z+6mppMDNM88MZe/O6dfZj3yJqEbi/e0zZPQBs8pjAtWZ+1E7qyb/I2JRKMHD7dhsUv7MChUz3XsbenMRghlTYzEj+FbjRME6+w0Kgy3GIByk0aT8lr9ERqRiIBisUCDB8Yaqy263gotWl0thUJRrS9IABg1d6T+OYfP1AzFGIbO3wBzRRSrz+IE+GARTTK0re3T3Q9ivtf/wiz/ntlVHFkXXOn5iChz0rITdkSVXOmHc2dfjhtVkwZUqQGhKaZEV0wol9e/sk1oSmZj63ab7qQW6tHOxPp4Kk2zWt++HQ7dkipaCP+QBCf/c27+OrvNyZVMyJeM/n5xTpQRoy+7OX6ocZ2r/q5jjWs4DUYpjF6DvE5NaOfwt2VmVp1TZ34zIOr8OSa/VHXdUfNiNHJR+zMiPbzkmpjuGBQwfV/fB/f/vMmKIqCPbWhXlSJrt8UWR3cr3kdUnmtzYZR4gURsZY4kAMcORAPBhV8XN8Sdd+v/2Ej/v5+DRY9syWRTe6VGIyQSi6WSyQzIhew3j53LIYOyMEDX54W8z5GBaxFOQ51rRc9tzRUJLZP7pI4eXCRulbOruOhLySjg0FBuOjVaKE8PREYdXgD2i8Ff1A9+JYVuML7ow2uxOOu239KM3SkJ9a8+cuGw5rLH3xzLybd/Qbe+eiEZnsri9ywWS04cLIt6emQO4+FXpcJlQVw2q2RYETqILqnthlzH1qNV3fURg3TdEhDT7XSQX1MWb4anOm1efw4eCqSLSkvckf1XdhyuEF/N436Fg8+qmvB+gOno/pgmPUZURRFDUbl+9TE6CdidAYrD9kFldC2ALGnLIuDT7wx/hMtnpjZgwZdp92uDNX88tU9OHCyDb989aOo6+TtbPX6U1r/py78efjslAr8+VuzAMTOjJwMB1riJCNWcNfU7sOXHluHP68/FP28zZ14Z+9JvLm7Hu3eACql1gCJZJLE87Z6Apr3P9a0ZDP67xsxhGuWGQkGFVz9u7X43G/Xmr5WHpPMyB/WHsRlD6/Bsg+07S7E0KT4DtRTFAU/+9cu/O7tTwyv7w0YjJBKzowkEoy47DZ8YcYQfHr8IHz3M2Ox9kefwbkjSmLex2iYpjjHYbp0t9VqUe9jVGQ4eUgRJg0OTf0WRV9GXwIiM3Kq1YMlr+7BphgHwgopGJG/LOubO9VahdL8UDCir4Fp8wbQ0unDN5/+ANc9/b7hgSTW2ZuY5XL3v3aq2wCEMjEzh4VmNq36OLnsiMhAiKBNBHXyl+if1x/CvhOt+M+/bVEDDPGWyK+B3IjJaoFpZqTN49d8MXr9waj3ZXNN7GBEDopO6HrAmH2Jd/qCan2GfEw6GCN9/dCKj3HB/W/jaEMkYNHPrKoNZwBizayJlRmRKQpMgzggek2TrtQM7K41714tP66ihAKSZIkasWEleepUf/1nfuWeety+bCtOt3rUAlaxZlWsmpH39p/CpsMN+KsuYAe0GZlOXwCDCiKF7icTmK0l/q5CmZGu1Yzog0/xXWP2vjW0e7H9aBN21zbj/YPaE5ZDp9rwSX2L5iTIH1TUzJyoiTlw0ni41qjdAhD6vC1ddwgPrfg4oWnxmcBghFTykEOiS8w/dM10/PH6WabBRNRzmGRGYhHNzuTMzVXTBqPAbcdtl4zFxPBCg3vDfSGMDgaigLWh3Ycn1hyImZauCHfEbfcFNAWpRxtCX4AleU41QNLX+HR4/aht6oQ3EESbNxBVkAoYHxjl4Sjh6kffw09fCgUlOQ4r5owPFTa/l2RnRhGkTR4cCkYimZHI6yS/34+vDqX0ywpCr0ObSTDS3Ok3/eJv8wawWwpGOn2RM1DxUYmXGWmVphjrn8dsmMasgDhWMLJq70kca+zAz/+9W71MX8x8PFzLE6tAWd8OPpbaGLVV+jVNlm89ZnhATsTh0+b7ra/DSaXeQGTKBhe7pc+V9r35wbPbsXzbcVzz5Ab1fRRrVulfqz+tC02BDgYVdUjHqDfLsQYpGPEHNatQHzoVP3MogqB2b6AbCli19xE1bZ0mywE0SG3i5b+nYFDBxb9ahUsfXoNTbdrPgPhbFcFwu/S6yc8x0KQvlDihCCpAYy/o7WKEwQipki1gTe05Ih+5ArcdU4cWRfVaGVWah7s+OwH/vvVTACIZETkz8uuvTMemn8xFRZFbzVKIA4jRwSDWVEo9kRlpbPdqWvSLM+cy6SxMP0zTrgtAjjZEH3SMagb0jY2OnOnA9iONaiOlHIcNU8KZjf0mZ0VGFEXBzvDzifuLobJVe09i8+HQmZl8NtuuZmNC+ymGaRRFwfr9UjDS4Us4MyIPec0aWQKb1YLaps6YfVNapVks+ucxazJnlvY/FOOgLHxUF9lefTAihiNaY8zQUZuexchkiIPF8RhdbfXNs37xyh78ZPlO09faTKdPu5zBz/61C1f85l314Kvfzqt+9546PJgosa0VhVIwossUiIPvvhOt6mtUFV6zSg4EjjV24J5/7cJfNhzGjmNN6v42tPuihpDkzEiHVzvUksh7LX9O5AUoYzU98wWCuPp3a3Grri5DnxkRHaLNevk0Ss8nCssBbWZK/3ch/nZEMCK/bvIaOEa9nwDtd2JvarwnYzBChtw91GxJHqapKHTjX7d+Cl+bPVz73A4bvn3RaEwZWqT+DmgzIxaLRZ16rO/IajRV02g6sBmRGdEXo4rAQk4J6zM97d6AZkjB6GC722BcV7+Gj16u044RA0O9Kg6faUcwqOCdj07gjn9si9nF8mSrB43tPlgtwLiK0HOI1+03Kz/BN5/+AMGgYngQF5kRcd3+k62aqZlNHT6cbDE+qLZ6/GpRIRA6ExVnoyV5TnWl6tvC6XvDx5CHaXQHYp9JzUgywYh+hVYx7g5Er9Uiute2x6wZMZ9NI4wtD70HsTIj+mBEaExy1WaRKRSe2ViDXcebsbvWfDhTX8MUjximGVycowa53kAQzZ0+9YxdFJgL+S47BoSDMvkM/5mNkeduaPeqmZFAUIl6P/TDNHJglUiGRz6Yy3/nsTIjWw43YPvRJrz8oXY2lD6ok2f7Gb3Gcmbko7oWdQadnK3VZ1vEc6iZEYOMLWA+BCxvR6JF9unGYIRUJblOlOSFfox6iHQHuYDVarImj36GihjCcJk0phJ//KEmZUHDqXapBCP6yn/xRy/P/IkORvyolw7QIhgJBBU89OZevPvJScMis+EluVGXydwOGwYXu2G3WuD1B1Hf0onrl36AF7Yew1ee3IC5D63GH9YejLqfOKDnSX1j5ICwxeNHm9evOSgIIjMiDvAbDoSyKOK1bO40z4ycbPFoah86pGEat92GO+dNQJ7Thg8ONeDx1fuhKErU2a9cLHpSN9ylz4x0eAP4xh824nfvGK95JQcaglFKW2yjPsATX/5GNSOiYNGbwDDN+PJQEGYWcCiKYhqoNCc540U/W0mk+k+2hKfAh7f3iW/MxJ/CxadrPj6JhgTPnD3+gPo3UlkUyYx0+oKY/vM3cf/roaJZfYDY6vFHijzVLE0Af38/UpR5ps2rbicQXUdzXBeM6GdqyYyK1eWDuZwpiBWMyFmmNo9fzaTph7vype8aoyFj/ev7cX0oaNQXjstEoNumBiOR55RrnRJZr0tfIN1bMBghld1mxboffwbrF38m6qyxu8gHQrM6k3Ip8wBIwzQm2Rq53qHNEzD88jHqTWJGDNPoz3jELJayJDIj4gzu9Z11+M3b+/CNP7xv2Jgp3vblOK2w26wYGk5vy+Pie2qbse9EK+57eXfU/cSXUI4UyOmLiFs9fsMzfrGf4ot7Y7jY7tKJoW7LzR3+qGBEPM8eXeFkpzegbovLYcOccYPw08+FujBvP9KEuQ+txlee3KAprpN7fehrRvRFeFtqGvDuJ6fw7wQXtQMiRcgysR6NCEYKXGJRRu3BQCY+f0YdWGU2qwWjwsNxtSZTjZs7/WqNziDd30GibekFoyFCIBJkiwN4gcuOOeMGYWJlIfxBJW4fFKG+KfQ4LrsVJXlOzd9CUAE+PBIKhsRnSwxfuOxW9XMismVrPzmlCQrOtHk1JwNn2rw40dyJF7Ychccf0NSMhALdyGt+QMqMbD58BpPueR0///cuzbbLB3M50Ik1m0aewXX7P7bh8kfexb+3H4/6nnDZrWoWt6HdGxWc6oMB8ZmK1etFPIfInsjfcUekYKTdZBhRHkrSB3a9BYMR0nA7bHE7r3aFHOToa1l/e+0MnD2sGPdcNUm7TeHtMcuMOKU//uZOn9qyXZZMDYxRMzcg8uWuqRmx66b2+gI4IWVGRDAiH3zqDYY28lx2fOXcKk3QIBOXDwsP1dScaTNct0R/1iW+7OV6G/392jz+mMM0nb4gAkEFGw+cBgBcFg5GOnwBHNN13B0ltT2XdfojzaVEpksMWbx/6Az2n2zD+4fO4KVtx3Gq1YN7/71bM5ylL1jV94NIZcG10oLoYOSNXXVQFAVN4Y6kleFmfOKAqu+5AsizJ0IFi0bDhEAoaBHLHtQ2daLmdDt+/dYnmuEXkbIvznVE9fFJtDFZc6cPdU2dptOQRZpeHOBc4fdj3qTQyutyYfFHdc144I2P0NQe/dziM11Z5IbFYokKctu8fiiKombdln27GldMqcSjXz1b/TyKz+cruqGP07pg5HSrF//z+l7c8c/teOXDWt0wTVBzsN13okX9fcOBMwgqwB/fO4TV0iy0Ds2whbyadfQwh6IoOHSqTRN0rNgd6qJ738u7o4INpy3yfXTZw2vwmV+t0gTPDbrXslUNRmJkRsLBVpvXKDMSeS2MvvsAXWaklwYjPVOlSJQA/TDNldMG48ppg6NuJ74sYy0aVuC2w9PqRavHb5jOFl8QiUyTzDHpeSK+wMpiDNO0efya1XHFl6YcDBi1Qchz2bDkC1Pws6sm4ZIHV0c1kxLbNGJgLtYgtKpnjtMWtT/bjjbi0+PL1N/FF2VOjGCkpdOv7tugAldULxUglOk40eKB02bFhWMjyxXoh7JGDco3HIaSiwzFazGqNLpO5tcrP8Fv3/5Es2qpEf0wTSrBiNEwzWOr9mNArkOd2ltZlIOP61vVL3nDzIg7MqU11DHV+PkKc+yoDA8BHm/sxNWPrkVDuw/HGtvxv18K9ec5rh7gc6BPHLZ2+tHU4cOPnvsQ15xbhU9PKIOeoij4zK9WobnTjwvHlBpuhz4zIk4+BobX8mn1+HHjnzbhZKsH2480Agh9fm79zFjN44ihJtHjQ9/QsM3j10y1HjYwF49+7WwAwMsfhjJY7d4APP6AenC/aNwgrPn4JM60ejVZt4Z2r5qZ3H6kUXMw1mdGfAEFe2pbML2qWNPR+MfPf4g3vncRCt0Obc1InGGaZ96vwX+9uBPTwjVsshMtnqi/QafdCrfDpn4PHW/qxP6TbRhfERqi09f+iM+Uvi4GCA0BitdIURR1KEceZpI7OLd5QgGgRffdKgdrLGAl0jGrGdErDJ8hxqr7EMMcLZ1+9UAi395ptyS8MJtZdkKINUzT4dVmRo43dkBRogvwSnQHwgK3HRZLaBv16Xl5m4aFa0tqTrdrDoxiqEAcPOTtAQC303yYps0TUB9rQvgLEwhlDsRbJM4qp1cVI89lN536PXRAjuagJHezFQcMkekaEK5PktWcaY8biADRTc+SDUacdqvm8+GyW9UF01buOaE+3uBibRGvUQapUOorEavHSIHLoQYjp1o96hmyvEJrbaM4wLuj3qeWTj+eevcAXt9Vh+uXfmD4HHvrW3Cq1QuvP2ja4fZoQzte3HpUDZpFpkq8p8ebOvDWnnrNZ+n9Q9HTsMVBVbyH+oaGbR7t1Hj570puDLbxwBm0ePwoL3Th0rNCAVbNmXbNQV4ettmm+4x3SoGuWOrgw6Oh28iLItY2deJnL+2CPxDUBCNy0adRMCKWYNh+1Pj11A+fOW3WqO8a+b2IGqYJb4tRZkS0PfD4g/D4g+rsHPE59AeCmkxWUDGezaUpYGUwQqSVaF3KLXNG41sXjMTlk6JXOhby1bF9n3rgl2sCHFLqNJ5cZ+xgZGC+eTAS6i0SOaPr9AXR0O6LSnOL3iiC3IG2zDAYCT2PmFHzcX2LOnSx/Z7LcOe88QAMvqj9Ac39AUQNw7V6IouFjSuPBCMFbjtyw1+qb4enfM4eFWpqV2gSGOY6bJpumGJ7O3xBtdBP7qkiVrMFgKkGZ55m9MM2yQYjBS675oBR4LbjU2NDmYQ2r1+TGQEiBxyjug0R1Hj9wZiLoxW47SjJcyJP9/mSA7I6aehDf3Bv8fg1WRe5cFF4W5qa22gwtAIA7+w9ie/9Y7v6u/g8iM+9XPN0/uiBAIAdRxujemZ0iHqk8P0sFotmm9u8keE/t8Oq+XtXh2m8AXXq+eyRA9W/2U9OaGcCnW6NBCP6Rm6d/oBaPDxrZOjzuT1cr3IiHIx8+6JRsFiAF7Yew2d/866mHklmFIzEa5X/oS5IkYeNhR3h4AiIDNOIv/NYny3RksDjD2iuF3+vHx5rQnOnXxNYGz1OXximYTBCGWNLMDMyeUgR7r5youkcekA7vVesQCp/ydut1oQaGn3z/BFw2a2ItWmajIvugNHU4VO/GMVZzWOr9kWlRseVF2ieQ65pEbNYZOILXzSLknuN5LvsmF5VDADYeOAM9tQ2o8MbwL4TrWrGQ1PAqtvmVk9APTsbL2VGClx2dXhInL3PHhk6OBVK9QzibB8IzYQaLC16OKI0FGx0euWakci2yP1Vfvn5KYbN32SinkdfwKrvmBpPvtuuKYjOc9nVgLDNE1ADWrFv7QkWsMbMjLgdsFgsGD4wT3O5WHTxREsn1u0P1eUMLs6JqpHSz7ZYbdCJV+4TIupq4jUVFMOg4jMoDvoD85xYev0sOO1WNLT7omepGAwBylkxuRZJv9yDXMD6cbhoeHxFgfo3q685qm/uVIc99IGoPAQoOkDrMyOfm1qJX31pGgpcdnwcY7FJX0CJCrri1eps1jXvcxhkRj6UMiMioySK0WMVsKr1SL6g5rPXHm6o9u7Hob4/F44tVYNJoyJWFrASxWDtxk+fOCDIwzRyTYDTbonZenrq0CJsu/tS3HPlRFgslphDNfIQhcNuHLUU5Thw66fHAAB+/+5BPLtZuyDe8IG5mseRAxxROCoTX27iy1pklnMcNtisFkwZUoRZI0vQ4Qvgyt+uxaR7Xsfch1bjqXcPau4PGMymkZZRl/ud5LvtyJMWTLRbLTh7eDEAbRO5c6QlALz+oFqkCQAjS0VmRBqmkQIOUfBanOvApMGFuGXOmKh9l4nMzcNvfYKf/WsXNh06g05fIGZmxGgJgnyXXXOwz3VG9rXNEwloRWZErRmJMVPL6w+aFq8CkWzSiFLtNO5Q7U/ofRPLFFQUGmRGOn2a/XznI20w0tThizowiseKRZ8ZEZ+tPJcdTrsVk8PLLWw9Enrsv79fg28t/UAt/MwxGQIMKpH6hBxdNkj83uENqDOYxpcXqHUremL6q5EOqc+IyIzsO9mKow3tat1JeaEbX5w5FNXhTE8s+mAn2VlMRpmR3ceb1ZMhkRkZGm6Jb1bA6rJHghpvIKi5PhBU4A0E8W544cwLxw5CrgimDYpYmRkhiiHRmpFE5Etrz4gzQvmLzWGzav4gq0cNVIMFIFTHUJzrVAu/zIIRh82i+aIxam8PhFKwN100Cl+eOdTw+mEluZoDupwZGTUoL+r24otGf5Yr7me1WvD7hedgxrBi+IOKekD5JDyNOFYBa2OHT33NRpXm4Xtzx+GOS8ch12nX3G9ceYG6HYU5ke2Vh5w8/gCGFEcP08i9IOTAaPbIgbBYgDnjBsFiseC2uWOx7e5Lo/Y/dD+ruqZJU4cPS9cdwpceX48Z967Aqr3m6/WUGPTMyXfZtQtDOm3q2Xtje+T1qCgKZalEYWDszEggamaFTAScI3SZkXZvqG5CDO/ZrBZMqyqOep9CxdmRYOStPfV49J19eOejEzjnF2/h/97ZB6OeVxVFsYMRty4zIojfZ4TXRNpa0wgAWPzCDrz90Qn8I7xYm1uTGdFus5iSrR/6zHWIYVU/DpwMTcUNZUb00/pDj/eJwXR4ua+JeN2HDsjB7JElUBTgv1/Zg6ASej3F8I/RDCo9fQY12WBEDiIEjz+IT+pboSiKGgxUlegzI9HBiPgM6DMjQGgBTLG+04VjS9WV1o0+o/oOrIqiIBBU8Niq/dgSZ42odOFsGsqY7uxlIg7sov+A1QIU5WiHaWR///Z5AKA2yXLphgdynDbAoJFjvsuuqVQ3C0ZEY7RRujbvQlWJNjMi///ySRX4/cJzcKrVg8Uv7AhtjyPSsCzPaVPP0OWMSlGOAy/ccj5qmzrx5JoDWLrukHpdrAJWubtpjtOG2+ZGZk3IByj5jF4ephlfEdlHjy+I0VJ2RdxHbnom16xMqyrG29+/WFMnU+h2wG61RLXSHjEwzzDLESvjZbNaUJhjh9TpHUDodZMPGHkuu7qv8rThQfmh91EUBhp90RfEKWDNcdjQ4QuoGRR9MNLhDeCdcFvweZPKcc+Vk0IdTQ0KWO3hYRAx6+lXb+5V60ieWHPA8DWIlxkRGRh9MCIObjOGFQMIBSPyNFjx/sTqYSMyE7m6YRq3M3yQDWc08pw2DCnOgYLQ365468+qLFSDIL1RpXn4qK4lFOiKacp2G267ZCy++tRGvLazDgAwKN+lfteUmqzdIvP6g1i1txZ/WBt6PZOdfRKaTRP9Od13shVVJTnq6yYyI20eUcCqze65HTb1e0lfMwKEZp4pCjD3rDIMHZArZUai/x7kz6U3EESrx4/VH5/E/4Qb0x26/4qk9rEnMDNCGdOtmRFpJgAQSp3LZ5b6fiB6+jMZuxQoyWd1+brCTX0qXRAH10qDs9JCtx1DB+Soj2W3arMtdpsVl04s19RT5Dgj18vdceVhFCBURDi4OCeqQDaRA4bNaonaH3nfq6QusfLlY8sidSZFOQ51mMZqiazO2ikN0+hT9iNL8zQHQqvVEjXLBggdxOWahCHFObjj0nFRt5PlOGyGU7XzXfpgxBY1QyjfZde8323hbrV62mAkdCCQX0dRQ6NmRkq1wUib169mdj4/Y6j6+hkVsIohyPuunozyQpfpNGJZvMyICK71hbX6zMie2mbDYaDcGIFuJBjRDdPo/t7GVRTAarWEg8dIoPv5GUNMt1v8fbR6/OqMGLfDiurRA9UaKgAol2qwEsmM/PD5D7HomS3YUtOILTWNps3jzISK5aMzq/tPtKpFxS67VR1GFkGGviWByxF5HI8/GBWMrAnXDH1nzmgAke9Aw8yIfr2gNp+mUDmRxR17GoMRypjubPIqvuhFZqQox6E5cOkzI3r6YESeXvolaailwKUdJtH3VhBELxL9geBP35qFFxddALcjcvDL02VbhJK8yHPJ2yc3ZdMXBgr6oClWMHJCOmDot0O+3zApGGloi5zFDSnOwW+unYHLJ1XgW58aiXHl+bBZLRg9KF99X3wBRT2QJ7LukVEwMjDfqXkfC6XAx4zbYY06yALhAlbp7DXXGfpd/kzmu+ywWSP1Q+3egGHL7nyXVDMS/lKXi62/fE4VKovcuCDc90NfM7K3rgUHT7XBYbOoM3qA6GydXDNSlOMwLUzVB8BGAbERffZCBCODi9wYVOCCP6jgn5uORN1P/ozoP8VmmRF9MDJemsUlzwL64tlDDbNhQGQmlty3w+0IfYa/fE7kb1ZevsGo667eit31MQvY43HarJr3TnTx3X+yVZ3WOyDXGRU86D9bbrstMkxjEIwAofdWFO3mJjhMAwCn2zyaE5lYNTnpwmCEMmbK0OJueyxx8BULdxXm2DVDKPr1bgSxiNfVumZrZ4dT01+bPUw7nKI7yJsNNYnMyOAi7cFyxrBi9YxOPJZZz44BUgZEzpzIwYhZ7xV9yt0dY2qvWPfFaEqz/DhyMCLXLlitFlw1bTAe/8ZM5LnsqCzKwUuLLsCfvjVLE0SJg0wi/V4KDQ60A/NdmuCvOMeBwXEOtC67zXC/8l0Ozesght/k4E68tuJLu6nDZ7j8urje4w+qwwXye3pd9QisX3wJJodXTR6U79IESGLGyYiBeZr7Rc166ow09CvMsZsGI2JYRYiXGVGfz27VvL754dfCYrFgRjjT8Nae6FV95SFA/TCVWc2I3WbV7J88i0t9XIcVeS47pg6JnvKd57SpAb8cvIjHvGJKpA2AXPRr1OgOCNVpFUiv/WUTyzFLKsxORqiANbK/08Pvx/6TbWrxanGuQ/3bUmtGPNrPVigzImpGAoZBhhxoRWaD+dHQ5kWNdEKl7z1yps2rWYHaaPHOdGMwQmn38v/7FG6fOxb/efHobntMdUZDeLy/0O2AXQ5GbBb1C1Fu7PWvRZ/C87dU45KztN0sH/jyNPzy81Pw86smac7MCnQHeaOMBhD5kigv0p6JyfcX/zcLRuThGDkjUJwjD9OYZEZ0wzfaIkPtNp8wOXvV3294SWR44fuXjYfbYcWPLp9g+PyThxSFpqdK06TFQSGRYMToNucMH6DWTACh7EBlApkReb9E7Dh6UJ4uMxKeUSK9biJYFPc/1tihDouI2ztskcyJXMA6YmAuvnJuFa6/YETUsJTFEipQ1dMHF0Y1I/EyIxYLME0K8uXiTQD4zbUz8NkpFZq/AZn8WsmfLTFUY0TOcnh0wwEiM6IfTgS0AbKcGbnj0nFw2q34242huq6Zw6Ofe8iAHPU9EAGi025V17sqznWqwzNihg1gPkxz7vASzQlLSZ4LA/JiT4k2Y7daNPsmXruDp1pxpi30egzIdaqvr/lsGps648vjDxpm5eRaK3U2mDeAC/7nbVz0wDtqA0Z9/5szbV7N4+nXksoEFrBS2k0eUqSeJXYXfZBQ6HbAKXcCtVrxz+9U44k1B3DnZePVy4tyHZg5PPoMaPSgfDWDIR8U9JkRM6JXiD4LIQcvkWEa44OzzWrBT644C6davZo6A3kIwDwY0X6RygdE/arGoqDOKIMgn41VSv1DplcVY8fP5pkW8ApimrTcuTReLxFAmxV47uZqfHKiFReNG6Sp/C/OdcQdgshxajMj371kLD43tRKjB+Wr02iByOsY+jd0wBDvj7i/aEc+INcBm9WCdm8ATptV/XzITc/cDhvu/+JU0+167Gsz8f6hM7jpz5vUy+IFI3KhbmGOwzB7VDUgV9PBN9dpUx/XYgE+O7kCV00bjNuXbVWn1MryXXY14JED2nNHJBaMRGVGwsFIjiP6c5rjjLRMlzMj371kLG6eM1rd/7OlYKQ034VTrR4MLs5RA1YxTKOfffTKdy/Ei1uO4WvnDYvcP884GDlnxACs+jiS9RmQ6wAQuyCnJM9pWNwaUBRNMD1pcCEcNgs6fUE1AzEgzxEZpvGGeobogxG3nBnxB6OmHAPQvdehxzvR3Kn+ve2ta0FZgVt9X0Rh+Jk2r6b+Sd9ILhMYjFC/oB+uKMyxazMjdismDynCb6+dkfRjywdGsyyGXrlBrxA9kc2JtYjfjReOirpMXkBNH4QJ+gBHPmCYzT4xCkbktWf0gUe8QERwRwUj8TMj8pj7OSNK1F4m8nMW5TjiPpbbbtMEYjkOG8aEC27dUqAohk3kYRoxQ0u85yIYKclzqn1Z5JS8PJsm3nYV5TrUoUB5f2RmC1ZaLaEhlEKDlZ7nTSrXfJ7ynKFi6c/PGIKyQpf6N/HV2cOxfNtxTB6iLXQ2K9aeOXwAlnxhCg6dasO7n5zSHLzk4mp9bYL4/BgF3OIzWZrv1HQ1BrSB2NlSVmbGsGKs2F2PaUOL1fubDf+V5oem18vkKemyc0eUaD5bxbmOOKFIKCthFIz4A4omMCrJc2L4wDzsO9GqBsChzEhoewNBBQ3tPk1beiCcGVGDkUDU9aFtkIZpwo+3UxpyEZk88b6UF7pxrLEDZ9q9muBnT22L4Zo26cRghPoFfcai0O3QfLnYu1AtK3+xFBgcAIwYdVHVK8oRPTuSSwcXJ5QZMS8YNKucN5p1Eq8VdiL0xYruBFaFNmvdL7+PsTryqs/lsGkCDPlx5YAn1yBLpWZGwv+KBckG5kcWE3TatZkRkVUwy3bJ9MNi+s+BWeFmgdsBq9USFbxs/slclOQ5sT7cxTW07aGCzoevma657ayRJXjzexdpesLI+xrah8j/LRYLrp0VyjCUrT2I3S/vVq+TgwD9dGxBP1QVuiz0+Eb1IrJBBS51wbj/+uxZ+NHlEzCyNA/vHzyjec5EMm7aafkWNdswfGCu5qSjOMcJS1Q5rlZhjgNuhzUqG+QPajMjhW4HRpWGghExTXlArlPzuZRX9Rbc8mwaX9BwzZlBmmGa0OPtkNrTi8+jmE0zuDgcjLR6NY/X6vGj1eNP+PutJzAYoX5Bf/At1M2mSfQs3oimZsRgmOYHl43DB4caUJjjwL+3H0ehroeF0RcWAHx2SiW2HWnEN6qHJ7U92qm9iRaw2gz/r7mPwQHjJ1echW/84X388PLxBvdIjP4goZ8lYmRiZSFewLGoy+26zIgZm9WCQFCB22HVdQiVXgd7dOBh1BVXvC5qMJLnVKfYyt02Pf4ADpwKNefS9xIx4naE6mnE2Wu8YRpB3E6+vcUSOsBZLJaozIgZeR2iyO2jX5NYtwHiLyxpth1ivaTx5YVR1+mtuvNinGzxaIYroz5XCQS5stJ8Fx740jRUFLlhsVi0Wbdch1p/YibXaUOB24FOn3blan1jxMIce9Ssr+Lw44sgq15a0E/en0ifkaC6no74bAO6mhFndJ8cUWguhmZD29EQanyme76mDh+DEaKu0v8Rhab2hv6QbeH+BanS1IwYfEGLpdV/sjzUoKxc12TqsokV+Nf241FnoWWFbjzyleSHjRIZphELA4qzHzkA+eYFI/DevlO4YEwpHlrxsXq50dnr7FEDsfPn80wPjImQH9diMc96yBZWj8DpNi/mjBukuVw7myYUlOW77FHTHgvddjS0++By2DQHT/m5DQtYpYNmvjqbJjxME542PjDfiWONof87pQUYvf6g2k10tEmzO5mYvSO2PWqYxiSAFkMNRbrPgTh4ylnCeIs+6mkKWE0CmVx91i2B5zC6jcgETaiMnRkBQsMR+mUS9I+ZSGZE1ubxa6ZSy0s7FOtOZozkOm0ocNnVLNk3zx+BT0604PLJFfjbhhr1doVuR9SMJjFTLs9lR7s3gP0nojssuh2RGUcef0BtijYwz6kWncsZWKMTE31mRCxvcKbdG3WC1tThw1Dz0qAex2CE+oVCtx0jBuaq/UEKc+xqSr8rQzQA4LTFP1sEIl/k+iGa+66ejNGD8rFgxmCjuyUtkcwIENpWjz96bZBCtwP/+E41PP6AJhgxO/h0JRABolvRJzIu7bQbz9SRZxWJg/FlE8vxwtZjaldSIBQQNrT7kOOwaQ6wclZGXptG7KP8eqoFxuHXTiyAWJLnUgMQp92mnpEHlUjb8kSCESD0vpgFI/ogWH87+fbykFW+yVBLIuSiVdOsm0lr91iMPlvfmzsOEyoKcdW01P4u9MN9iQz/yfS9bOSD84A8Z9zPfY7DrrnND+aNV197OTuR67RpGq+FHj9Sj3SyxYMVe+oBhNbIEqsA6zMjoknZsJJcNRiRh2mMprmLNZZEZnZIuAj9TJs36j3RryyebpzaS/2CxWLBD+ZFhhJyHHZ1qp5Zl9REJTqbRvxx64tXi3IduG3u2KjVWlOlrRkx/wKWDyZGqXSX3aZ5bZI9i05UIkNEidJkRsKvw8+unoQ7Lh2HZ79Tjff/6xJsvOsS9bqoYRppf+Usic2gC6koENVnAkrzIwcquWYECBUjOu1WDBkQe8qxID9fdAFr5HEvGlcadTs5AJGLWeVgJOnMiMkSBbKojITT+DXVPK7BdkyrKsaP509I+TOh345Ehv8A4JmbZmN6VTF+99WzNZc7NNPnHZpgxSh+znXaNO+93MxPnrFmsViisqXFamYktA+i/uXSs8oj+yMVR3dIQznyyYg8bfu8UQM1U6QBKTPi02VGWiOzafRT7zOFwQj1G1dMqcR5o0rgtFkxZWiReuCxx0m3xhOvZkT43LRKzBk3SDOVsCdohmlibE9+nGAE0AYziaTbU6EJRpI8e9XTz6YBQgfi714yFiNK81BW4EZ5oVt9TrfdptlHly5LI4jPiCYzoqsZEUrynOpBwmWzRp1BjxyYl/CwoFw0rC/IlQ+A3zx/pPp/f7joUpMZkf4f6qQb3ockMyPyvpoFunm6vi1ygPfszdW4ZEJZVA+hnvhs6YOYRD9b548uxfJFF0S1F5B7pBTmODBACkbGhWdglUqLb+Y4tcG8XM/kDWiLxPXBiDpMo8tOXDpJCkYckdk0tU2d8AcVWCza90X+e7BaLfjOHO3sIX3NiJie3+LxqzOBxPAxgxGibmKxWPCXG2bjg5/MxZDiHHV4pivFq4D2y1bfDl42elA+/vStWYZ9S7qTPOsi3jCNYDaeLmdrYhU7dkWOrpi3K+TpjbFm04jndDtsmh4XcgBisVhw4dhSjB6UhylDigEYz6bRv8YD81zqZ8JhD9UjyRkbo1WXzcTKjMgzU+SuquKMVr69nBmxWCxq91SjRnaxaGpmEpg2nuPQLiEwdWgx/vDNczFV1125Jz5bqRRGx9Kmm34u12NNGlKIV797IR65JlLjleOwmQ7lXBkeehJTp6ODkXDWTbPGU76m8FnOjIgapdJ8V8x1ZBZMH4I7543HheFamOYOH4JBRR02Kitwq4GymCknimuNugunE4MR6lccNqv6JS3Ogrta85BK07Oe5HbYUJrvgtUSe60N+aDhNjkzfeBLUzEi3BI/mYNoMnK6cZimXWrUlB/jACeeM8epy4zoPgt//tYsvPm9OYY1I5HZNLpgJN+pHvhEUCIHrMm8jjmxghGpyZXDZsXfbpyNCRUF+OG8CVG319830sQtudc7L4GaETlgMct4RE0t74HMiNOmXUuoq1m3dl0RtBxk+QMKJg4uxEApM5LrtJkOS02oKMT6xZ/B87ecD8Bgtp/BEOA151ZpOhbLq/YKlUVuzBo5EIDxulhWqwWLPj0G11WPABAKRuQpvLlOmxoICUN7SWYk89+sRD1k0uBCdeimKzTrdSSZ9u4pT3/zHDS0+2IGI/nS2bLZMM3Y8gK8+b05ONrQjlEJFl0mSzubpmtDZvLZa6ypl1dNH4x9J1txyVllmgJmfWBqsVggf6cbTe0t1n15D8xzRoKQ8OPJ23V2jNbpevKqu7EyIwBwwZhSvH77Rerv7vCZudcfjGrmle+2A83JZ0ZEAGKxmNebyJebBZe5uuyJvoCzO4juvuK1d3Ux0JXfQz1fOLMg/x3pa0b0KovM64bEZ/fz04dg/4lW3HjhKHVBTtGx2CVNGxfKC934xnnDkeu04YLRpVGPK4jsaXOnX5NJcdmtKMlz4lRrpFnbYAYjRD3LYbPi0a+dHf+GcchtmGPVaKSTPg1uRMyMsFstMYeqnHZrjwUiAHDpxHIsXXdI3Zau0LeyN3Px+DJcPD603pBIcQPxe1HIB28xXfyicYNQVZKDI2dCj1OcK2VGwo83elAe9p9sw4Lpg/GZCWVIlDzrQn9gnx1eUyXW0FZRjgMnWzzdlxkJ73++03glaUCXGTEJAAZJQfLD10zvsf4VbjkY6WIGVM666U2sDA23aLr5Ou2avjXxyP1BhLkTyzF3YrnmMhGMuB22qCnBFYVuOO1WtQGdGfF5aOrwqdN67VYL7DZr1AKcItuT6dk0veOblagXk7+Tu/qFl07iwJJIU6qedMGYUvzlhln45asf4etdLO792nnD8LeNh5OaDirXZcQrLDWqGXE7bHj+lvNx0583Y8ygfNisFnW6t8iQPPQf07H/ZCsWTB+SVPbHHzAPrqpKcrH6zos1syf0RDCi795aLBX3JkNkPWLVIonVfX0BxXT4paokF7/76gxUFrl7tIaqO2dqGa398vL/+xRW7K5X28q79ZmRJOrRXHarZlkEM+I5XHYrygrcGFWahwOnQn1IEl2BWWTKmjt8UUsUyOs55bvsauaPmRGiXm760GJcOrEcIwbmZnTthmSJA0pX09fd4cKxg/DabYPi3zCO0nwXNt51SVLvg3zADCqxVxwRAZxVN0xRVuDGS4suUH+PZEZC/06rKjZciTceowOgLN508GKDniMA8N1LxmBkaV7UatTxiMfRD03p5TpDC+rFCgA+N7V7+urEMqI0V8189cSJgn5RTzmwz3HYcO6IAXh+y9GEHsuZYDAiPq9if2aPKokEI4UJBiPhINQfVLA1vLikyLCdVVmI5duOAwh9R8hZlExiMEIUh9Vqwe8XnpPpzUiaGFKSFzLrD5INCJ02Kz49fhAa2n1x27SL3hIlea6YzyOamo3uYtGvL0ZmJBE3XjgKA/KORnWqnTm8JKWMxMzhA3DznNFx66xynTY0dfh6rDdNou64dDze27cOgHaF6VRMG1qE7UebohYwlDlsFlgtoQZ3OU4b/uOcKviCCmaNiP9a3/XZs/DD5z7EwjjLPwwrycW+E60YOiBUWH7eqIH4+/tHACSeGcl12tQVeu/453YAkSFKObjKl4KRxo7oRf/SicEIUT8lMiOZHqbJNIvFgj9ePyuhVUmrSnJx39WTMLQkN+btvnj2EJw7YgCGxbldPGYLyyXq8skVuHxyRZceQ2a3WfHj+dGdb/VEEJLpz9bM4QPwxbOH4vktR3HBGPOCzkQ8/o2ZWPb+kZh9guSi2VynDVarBd84L7G1pb48cyjOHVES9zPz8H9Mx+EzbZg4OFSnMjs8ewbQdlyNxWKxoDDHoVlVWMyqmTQ4shZQUFFQlMOaESLqQQxGtBLNqHwjPC0y3mN1R0fdJV+Ygq/+fgO+f1nqCxFmQm/6bP3qy1Pxg3njYs5eSURlUQ6+d+m4uLerCmcukn0+i8WCkaXxPzNFuQ5MzS1Wf68ocuPr5w3DmTYvxiRRaK7PFJ1qDbWQl2uQjpxpVzMjLR4/AkGlS+t4dQWDEaJ+avrQYhS47Dhv9MD4N6aMOHdECXb8bF6Xiy/TTWRGzPrXpJPFYulyIJKMP39rFk63eRPOUnSHXyyYkvR95P4iQGgNJ702b0ANRhQFaOn0xSyY7kkpDSY/+uijGDFiBNxuN2bPno33338/5u2fffZZTJgwAW63G1OmTMGrr76a0sYSUeKGDczF1rsvxeL5Z2V6UyiGvhaIAL1nplYmlBW6cVZlYfwbZpgYDvrq7GF47uZq/PILkYDmC2cPAQDMPascTrtVDS4zWcSadDDyj3/8A3fccQfuuecebNmyBdOmTcO8efNw4sQJw9uvW7cO1157LW644QZs3boVCxYswIIFC7Bz584ubzwRxWbvYit8IiM5vaRmhMz94bpzsHj+BPz8qkk4Z0SJpkHiLxZMxv9+aSr+90tTAaBXzKhJ+pvqoYcewk033YTrr78eEydOxOOPP47c3Fw8/fTThrf/9a9/jcsvvxx33nknzjrrLNx33304++yz8bvf/a7LG09EROknhijk9ujUu4wtL8B35ow2bHiY67TjP86pUmePnT1sAKpHDYTdmrmTl6RqRrxeLzZv3ozFixerl1mtVsydOxfr1683vM/69etxxx13aC6bN28eli9fbvo8Ho8HHo9H/b25uTmZzSQioh50y8WjMbI0D5+fMSTTm0LdoDs6VXdVUmHQqVOnEAgEUF6uLYQpLy9HXV2d4X3q6uqSuj0ALFmyBEVFRepPVVVVMptJREQ9qKzAjYXVI3qszTtln145oLx48WI0NTWpP0eOHMn0JhEREVEPSWqYprS0FDabDfX19ZrL6+vrUVFh3HinoqIiqdsDgMvlgsuVvmlTRERElDlJZUacTidmzpyJlStXqpcFg0GsXLkS1dXVhveprq7W3B4AVqxYYXp7IiIiyi5JNz274447cN111+Gcc87BrFmz8Mgjj6CtrQ3XX389AGDhwoUYMmQIlixZAgC47bbbMGfOHDz44IO44oorsGzZMmzatAlPPvlk9+4JERER9UlJByPXXHMNTp48ibvvvht1dXWYPn06Xn/9dbVItaamBlZpetD555+PZ555Bj/5yU9w1113YezYsVi+fDkmT57cfXtBREREfZZFUeKsqd0LNDc3o6ioCE1NTSgs7P2d74iIiCjx43evnE1DRERE2YPBCBEREWUUgxEiIiLKKAYjRERElFEMRoiIiCijGIwQERFRRjEYISIiooxKuulZJohWKM3NzRneEiIiIkqUOG7Ha2nWJ4KRlpYWAEBVVVWGt4SIiIiS1dLSgqKiItPr+0QH1mAwiOPHj6OgoAAWi6XbHre5uRlVVVU4cuRI1nR2zbZ95v72b9m2v0D27TP3t29TFAUtLS0YPHiwZqkYvT6RGbFarRg6dGiPPX5hYWG/eNOTkW37zP3t37Jtf4Hs22fub98VKyMisICViIiIMorBCBEREWVUVgcjLpcL99xzD1wuV6Y3JW2ybZ+5v/1btu0vkH37zP3NDn2igJWIiIj6r6zOjBAREVHmMRghIiKijGIwQkRERBnFYISIiIgyKquDkUcffRQjRoyA2+3G7Nmz8f7772d6k7rFz372M1gsFs3PhAkT1Os7OzuxaNEiDBw4EPn5+fjiF7+I+vr6DG5xctasWYMrr7wSgwcPhsViwfLlyzXXK4qCu+++G5WVlcjJycHcuXPxySefaG5z5swZfO1rX0NhYSGKi4txww03oLW1NY17kbh4+/vNb34z6v2+/PLLNbfpS/u7ZMkSnHvuuSgoKEBZWRkWLFiAvXv3am6TyGe4pqYGV1xxBXJzc1FWVoY777wTfr8/nbuSsET2+eKLL456n2+++WbNbfrKPj/22GOYOnWq2tiruroar732mnp9f3t/4+1vf3pvU6ZkqWXLlilOp1N5+umnlV27dik33XSTUlxcrNTX12d607rsnnvuUSZNmqTU1taqPydPnlSvv/nmm5Wqqipl5cqVyqZNm5TzzjtPOf/88zO4xcl59dVXlf/6r/9SXnjhBQWA8uKLL2quv//++5WioiJl+fLlyvbt25WrrrpKGTlypNLR0aHe5vLLL1emTZumbNiwQXn33XeVMWPGKNdee22a9yQx8fb3uuuuUy6//HLN+33mzBnNbfrS/s6bN0/54x//qOzcuVPZtm2b8tnPflYZNmyY0traqt4m3mfY7/crkydPVubOnats3bpVefXVV5XS0lJl8eLFmdiluBLZ5zlz5ig33XST5n1uampSr+9L+/yvf/1LeeWVV5SPP/5Y2bt3r3LXXXcpDodD2blzp6Io/e/9jbe//em9TVXWBiOzZs1SFi1apP4eCASUwYMHK0uWLMngVnWPe+65R5k2bZrhdY2NjYrD4VCeffZZ9bI9e/YoAJT169enaQu7j/7gHAwGlYqKCuWBBx5QL2tsbFRcLpfy97//XVEURdm9e7cCQPnggw/U27z22muKxWJRjh07lrZtT4VZMHL11Veb3qcv76+iKMqJEycUAMrq1asVRUnsM/zqq68qVqtVqaurU2/z2GOPKYWFhYrH40nvDqRAv8+KEjpg3Xbbbab36ev7PGDAAOWpp57KivdXUSL7qyj9/71NRFYO03i9XmzevBlz585VL7NarZg7dy7Wr1+fwS3rPp988gkGDx6MUaNG4Wtf+xpqamoAAJs3b4bP59Ps+4QJEzBs2LB+se8HDx5EXV2dZv+Kioowe/Zsdf/Wr1+P4uJinHPOOept5s6dC6vVio0bN6Z9m7vDqlWrUFZWhvHjx+OWW27B6dOn1ev6+v42NTUBAEpKSgAk9hlev349pkyZgvLycvU28+bNQ3NzM3bt2pXGrU+Nfp+Fv/3tbygtLcXkyZOxePFitLe3q9f11X0OBAJYtmwZ2traUF1d3e/fX/3+Cv3xvU1Gn1gor7udOnUKgUBA88YCQHl5OT766KMMbVX3mT17NpYuXYrx48ejtrYWP//5z3HhhRdi586dqKurg9PpRHFxseY+5eXlqKury8wGdyOxD0bvrbiurq4OZWVlmuvtdjtKSkr65Gtw+eWX4wtf+AJGjhyJ/fv346677sL8+fOxfv162Gy2Pr2/wWAQt99+Oy644AJMnjwZABL6DNfV1Rl+BsR1vZnRPgPAV7/6VQwfPhyDBw/Ghx9+iB/96EfYu3cvXnjhBQB9b5937NiB6upqdHZ2Ij8/Hy+++CImTpyIbdu29cv312x/gf733qYiK4OR/m7+/Pnq/6dOnYrZs2dj+PDh+Oc//4mcnJwMbhn1hK985Svq/6dMmYKpU6di9OjRWLVqFS655JIMblnXLVq0CDt37sTatWszvSlpY7bP3/72t9X/T5kyBZWVlbjkkkuwf/9+jB49Ot2b2WXjx4/Htm3b0NTUhOeeew7XXXcdVq9enenN6jFm+ztx4sR+996mIiuHaUpLS2Gz2aKqs+vr61FRUZGhreo5xcXFGDduHPbt24eKigp4vV40NjZqbtNf9l3sQ6z3tqKiAidOnNBc7/f7cebMmX7xGowaNQqlpaXYt28fgL67v7feeitefvllvPPOOxg6dKh6eSKf4YqKCsPPgLiutzLbZyOzZ88GAM373Jf22el0YsyYMZg5cyaWLFmCadOm4de//nW/fX/N9tdIX39vU5GVwYjT6cTMmTOxcuVK9bJgMIiVK1dqxvD6i9bWVuzfvx+VlZWYOXMmHA6HZt/37t2LmpqafrHvI0eOREVFhWb/mpubsXHjRnX/qqur0djYiM2bN6u3efvttxEMBtUvgb7s6NGjOH36NCorKwH0vf1VFAW33norXnzxRbz99tsYOXKk5vpEPsPV1dXYsWOHJghbsWIFCgsL1dR4bxJvn41s27YNADTvc1/aZ71gMAiPx9Mv318jYn+N9Lf3NiGZrqDNlGXLlikul0tZunSpsnv3buXb3/62UlxcrKlW7qu+//3vK6tWrVIOHjyovPfee8rcuXOV0tJS5cSJE4qihKbNDRs2THn77beVTZs2KdXV1Up1dXWGtzpxLS0tytatW5WtW7cqAJSHHnpI2bp1q3L48GFFUUJTe4uLi5WXXnpJ+fDDD5Wrr77acGrvjBkzlI0bNypr165Vxo4d22unusba35aWFuUHP/iBsn79euXgwYPKW2+9pZx99tnK2LFjlc7OTvUx+tL+3nLLLUpRUZGyatUqzVTH9vZ29TbxPsNiKuRll12mbNu2TXn99deVQYMG9dqpkPH2ed++fcq9996rbNq0STl48KDy0ksvKaNGjVIuuugi9TH60j7/+Mc/VlavXq0cPHhQ+fDDD5Uf//jHisViUd58801FUfrf+xtrf/vbe5uqrA1GFEVRfvvb3yrDhg1TnE6nMmvWLGXDhg2Z3qRucc011yiVlZWK0+lUhgwZolxzzTXKvn371Os7OjqU//zP/1QGDBig5ObmKp///OeV2traDG5xct555x0FQNTPddddpyhKaHrvT3/6U6W8vFxxuVzKJZdcouzdu1fzGKdPn1auvfZaJT8/XyksLFSuv/56paWlJQN7E1+s/W1vb1cuu+wyZdCgQYrD4VCGDx+u3HTTTVFBdV/aX6N9BaD88Y9/VG+TyGf40KFDyvz585WcnByltLRU+f73v6/4fL40701i4u1zTU2NctFFFyklJSWKy+VSxowZo9x5552aXhSK0nf2+Vvf+pYyfPhwxel0KoMGDVIuueQSNRBRlP73/sba3/723qbKoiiKkr48DBEREZFWVtaMEBERUe/BYISIiIgyisEIERERZRSDESIiIsooBiNERESUUQxGiIiIKKMYjBAREVFGMRghIiKijGIwQkRERBnFYISIiIgyisEIERERZRSDESIiIsqo/w/90IxEb6C6jgAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ "
"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import matplotlib.pyplot as plt\n",
+ "\n",
+ "plt.plot(losses)\n",
+ "plt.show()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "64OtEmyUWUiM"
+ },
+ "source": [
+ "You have successfully trained your model to help automatically answer questions! Try asking it a question about a story."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "eFniMzpp1bpz",
+ "outputId": "0ce0e2a3-3d6a-4e6e-adff-d0c16b622c9a"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "What is south of the bedroom? Garden\n"
+ ]
+ }
+ ],
+ "source": [
+ "question, text = 'What is south of the bedroom?','The hallway is south of the garden. The garden is south of the bedroom.'\n",
+ "input_dict = tokenizer(text, question, return_tensors='tf')\n",
+ "outputs = model(input_dict)\n",
+ "start_logits = outputs[0]\n",
+ "end_logits = outputs[1]\n",
+ "\n",
+ "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
+ "answer = ' '.join(all_tokens[tf.math.argmax(start_logits, 1)[0] : tf.math.argmax(end_logits, 1)[0]+1])\n",
+ "print(question, answer.capitalize())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "f07OtnCpuKFa"
+ },
+ "source": [
+ "Congratulations! You just implemented your first QA model in TensorFlow. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "9UaM5pY9u8EW"
+ },
+ "source": [
+ "\n",
+ "## 2.2 PyTorch implementation\n",
+ "\n",
+ "[PyTorch](https://pytorch.org/) is an open source machine learning framework developed by Facebook's AI Research lab that can be used for computer vision and natural language processing. As you can imagine, it is quite compatible with the bAbI dataset."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "nD9akXoXxMjd"
+ },
+ "source": [
+ "#### Train and test dataset\n",
+ "\n",
+ "Go ahead and try creating a train and test dataset by importing PyTorch."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {
+ "id": "JxMYWSG173ch"
+ },
+ "outputs": [],
+ "source": [
+ "from torch.utils.data import DataLoader\n",
+ "\n",
+ "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
+ "train_ds.set_format(type='pt', columns=columns_to_return)\n",
+ "test_ds.set_format(type='pt', columns=columns_to_return)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "OeuzZKlPHAAQ"
+ },
+ "source": [
+ "For the accuracy metrics for the PyTorch implementation, you will change things up a bit and use the [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) for start and end indicies over the entire test dataset as the loss functions. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {
+ "id": "aD9tDpZfJsIB"
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.metrics import f1_score\n",
+ "\n",
+ "def compute_metrics(pred):\n",
+ " start_labels = pred.label_ids[0]\n",
+ " start_preds = pred.predictions[0].argmax(-1)\n",
+ " end_labels = pred.label_ids[1]\n",
+ " end_preds = pred.predictions[1].argmax(-1)\n",
+ " \n",
+ " f1_start = f1_score(start_labels, start_preds, average='macro')\n",
+ " f1_end = f1_score(end_labels, end_preds, average='macro')\n",
+ " \n",
+ " return {\n",
+ " 'f1_start': f1_start,\n",
+ " 'f1_end': f1_end,\n",
+ " }"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "laX5cYQRHMXb"
+ },
+ "source": [
+ "#### Training\n",
+ "\n",
+ "Now it is time to load a pre-trained model. \n",
+ "\n",
+ "**Note:** You will be using the DistilBERT instead of TFDistilBERT for a PyTorch implementation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "del model # We delete the tensorflow model to avoid memory issues"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "YXFCsNcY79jx",
+ "outputId": "09af112f-e1e9-4a47-c988-37ee2a068df2"
+ },
+ "outputs": [],
+ "source": [
+ "from transformers import DistilBertForQuestionAnswering\n",
+ "\n",
+ "pytorch_model = DistilBertForQuestionAnswering.from_pretrained(\"model/pytorch\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "xCUdMmCxHP6_"
+ },
+ "source": [
+ "Instead of a custom training loop, you will use the [🤗 Trainer](https://huggingface.co/transformers/main_classes/trainer.html), which contains a basic training loop and is fairly easy to implement in PyTorch."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 329
+ },
+ "id": "1htmS3TV-2Bk",
+ "outputId": "cc21bfbb-da09-47f9-ee16-7db0096d35e7"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "The following columns in the training set don't have a corresponding argument in `DistilBertForQuestionAnswering.forward` and have been ignored: end_idx, sentences, question, answer, str_idx. If end_idx, sentences, question, answer, str_idx are not expected by `DistilBertForQuestionAnswering.forward`, you can safely ignore this message.\n",
+ "***** Running training *****\n",
+ " Num examples = 1000\n",
+ " Num Epochs = 3\n",
+ " Instantaneous batch size per device = 8\n",
+ " Total train batch size (w. parallel, distributed & accumulation) = 8\n",
+ " Gradient Accumulation steps = 1\n",
+ " Total optimization steps = 375\n",
+ " Number of trainable parameters = 66364418\n"
+ ]
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "