3. Getting hands-on with BERT/3.04. Extracting embeddings from all encoder layers of BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zLbXrzA0_ndC"
   },
   "source": [
    "# Extracting embeddings from all encoder layers of BERT\n",
    "We learned how to extract the embedding from the pre-trained BERT in the previous section. We learned that they are the embeddings obtained from the final encoder layer. Now the question is should we consider the embedding obtained only from the final encoder layer (final hidden state), or should we also consider the embedding obtained from all the encoder layers (all hidden states)? Let's explore more about this. \n",
    "\n",
    "Let us represent the input embedding layer by $h_0$  and the first encoder layer (first hidden layer) by $h_1$, second encoder layer (second hidden layer) by $h_2$ and so on to the final twelfth encoder layer by $h_{12}$ as shown in the following figure:\n",
    "\n",
    "\n",
    "![title](images/4.png)\n",
    "\n",
    "\n",
    "Instead of taking the embeddings (representation) only from the final encoder layer, the researchers of the BERT have experimented with taking embeddings from different encoder layers.\n",
    "\n",
    "For instance, for a named-entity recognition task, the researchers have used the pre-trained BERT for extracting features. Instead of using the embedding only from the final encoder layer (final hidden layer) as a feature, they have experimented using embedding  from other encoder layers (other hidden layers) as a feature and obtained the following F1 score: \n",
    "\n",
    "\n",
    "![title](images/5.png)\n",
    "\n",
    "As we can observe from the preceding table, concatenating the embeddings of the last 4 encoder layers (last 4 hidden layers) gives us a greater F1 score of 96.1% in the  NER task. Thus, instead of taking the embeddings only from the final encoder layer (final hidden layer), we can also use embeddings from the other encoder layers.\n",
    "\n",
    "Now, we will learn how to extract the embeddings from all the encoder layers using the transformers library. \n",
    "\n",
    "## Extracting the embeddings \n",
    "First, let us import the necessary modules: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "id": "A9uDL3-i_rG-"
   },
   "outputs": [],
   "source": [
    "%%capture \n",
    "!pip install transformers==3.5.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "id": "KSwq5tr9_ndJ"
   },
   "outputs": [],
   "source": [
    "from transformers import BertModel, BertTokenizer\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oI91BJ8H_ndJ"
   },
   "source": [
    "\n",
    "Next, download the pre-trained BERT model and tokenizer. As we can notice while downloading the pre-trained BERT model. We need to set output_hidden_states = True. By setting this to true helps us to obtain embeddings from all the encoder layers: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 164,
     "referenced_widgets": [
      "9ba62db0115d4930990c3c6d087989c1",
      "7003e521a65b49a9b9727f3f1bfa293f",
      "94dc71fe23e040ceae5df4c4dd573684",
      "26c1ceb389ad46ddbf841770d07a5e93",
      "84ce79c32f354db9ad3d19acd02625c1",
      "7706547847d7465299292bb77f2dff25",
      "f769dd1c71324598b9e27be922759bab",
      "2bdc25c58d624b3594fc76892aed8e09",
      "d70b8c6e81774b3aa9e737d28be1dd48",
      "437d981ba7c14c39b8cf0e62ee9e055e",
      "b9b4261f0dc74e9a9210df1ec4273e6a",
      "7c7c261fe4ac435c856fcfd568b4e298",
      "4fdbc441590f4e4abc99b64b98169f8a",
      "b260f036b5b141188688b837a4845790",
      "0cc09eca07734399a4d7716f3911d7bd",
      "f5f075e096554be9a389edbd8cf1e089",
      "95e442aacd1e4aa5b361569341870f47",
      "eb6a67dab3534dae96aaafa975ef3465",
      "7bf00e5dbb204358a0a9985b914db244",
      "a043bae8118f439ca6d5b5a9c818d6b6",
      "a808c5be6f344f3db86cf6e3d26ec81c",
      "e0a7468cea174a7597f30e619fb99bcb",
      "f4060c1c44144dfa9a367d8614708b45",
      "ed2edfd213544216a1edf723e587b3be"
     ]
    },
    "id": "eM0UkRBp_ndK",
    "outputId": "213cd2ae-471c-421f-e5a8-0f82a77009ee"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9ba62db0115d4930990c3c6d087989c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d70b8c6e81774b3aa9e737d28be1dd48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=440473133.0, style=ProgressStyle(descri…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "95e442aacd1e4aa5b361569341870f47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=231508.0, style=ProgressStyle(descripti…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states = True)\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z6lcm9mn_ndK"
   },
   "source": [
    "\n",
    "Next, we preprocess the input before feeding it to the model. \n",
    "\n",
    "## Preprocess the input\n",
    "Let's consider the same sentence we saw in the previous section. First, we tokenize the sentence and add [CLS] token at the beginning and [SEP] token at the end: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "id": "1-hLzFu6_ndL"
   },
   "outputs": [],
   "source": [
    "sentence = 'I love Paris'\n",
    "tokens = tokenizer.tokenize(sentence)\n",
    "tokens = ['[CLS]'] + tokens + ['[SEP]']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YUKVAdmT_ndL"
   },
   "source": [
    "\n",
    "Suppose, we need to keep the token length to 7. So, we add the [PAD] tokens and also define the attention mask: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "id": "5OISI_-t_ndM"
   },
   "outputs": [],
   "source": [
    "tokens = tokens + ['[PAD]'] + ['[PAD]']\n",
    "attention_mask = [1 if i!= '[PAD]' else 0 for i in tokens]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ic5yVHnB_ndM"
   },
   "source": [
    "\n",
    "Next, we convert the tokens to their token_ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "id": "8YtAw5Gr_ndM"
   },
   "outputs": [],
   "source": [
    "token_ids = tokenizer.convert_tokens_to_ids(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-MmpLWP6_ndN"
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "Now, we convert the token_ids and attention_mask to tensor: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "id": "lm7Nqj6w_ndN"
   },
   "outputs": [],
   "source": [
    "token_ids = torch.tensor(token_ids).unsqueeze(0)\n",
    "attention_mask = torch.tensor(attention_mask).unsqueeze(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_FQ7RzxU_ndN"
   },
   "source": [
    "\n",
    "\n",
    "Now that we preprocessed the input, let's get the embedding. \n",
    "\n",
    "## Getting the embedding \n",
    "Since we set output_hidden_states = True while defining the model for getting the embeddings from all the encoder layers, now the model returns an output tuple with three values as shown below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "id": "DhmqjLG1_ndN"
   },
   "outputs": [],
   "source": [
    "last_hidden_state, pooler_output, hidden_states = model(token_ids, attention_mask = attention_mask)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BjN7Ojnh_ndO"
   },
   "source": [
    "\n",
    "In the preceding code, the following applies: \n",
    "\n",
    "The first value last_hidden_state contains the representation of all the tokens obtained only from the final encoder layer (encoder 12). \n",
    "Next, pooler_output indicates the representation of the [CLS] token from the final encoder layer which is further processed by a linear and tanh activation function. \n",
    "hidden_states contains the representation of all the tokens obtained from all the final encoder layers. \n",
    "Now, let us take a look into each of these values and understand them in more detail. \n",
    "\n",
    "First, let us look at last_hidden_state. As we learned, it holds the representation of all the tokens obtained only from the final encoder layer (encoder 12). Let us print the shape of the last_hidden_state: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AlmKmXrO_ndO",
    "outputId": "5408b910-3a71-48f5-e0d6-5afadae3a8b1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 7, 768])"
      ]
     },
     "execution_count": 9,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "last_hidden_state.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sCSrepOZ_ndO"
   },
   "source": [
    "\n",
    "The size [1,7,768] indicates the[batch_size, sequence_length, hidden_size].\n",
    "\n",
    "Our batch size is 1, the sequence length is the token length and since we have 7 tokens the sequence length is 7, and the hidden size is the representation (embedding) size and it is 768 for the BERT-base model. \n",
    "\n",
    "We can obtain the embedding of each token as: \n",
    "- last_hidden[0][0] gives the representation of the first token which is [CLS]\n",
    "- last_hidden[0][1] gives the representation of the second token which is 'I' \n",
    "- last_hidden[0][2] gives the representation of the third token which is 'love' \n",
    "\n",
    "Similarly, we can obtain the representation of all the tokens from the final encoder layer. \n",
    "\n",
    "Next, we have pooler_output which contains the representation of the [CLS] token from the final encoder layer which is further processed by a linear and tanh activation function. Let us print the shape of the pooler_output: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lK5MrEuW_ndP",
    "outputId": "bd3f4085-12f4-4b52-d73a-6a866ec8860c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 768])"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pooler_output.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ERlJpFL3_ndP"
   },
   "source": [
    "\n",
    "The size [1,768] indicates the[batch_size, hidden_size].\n",
    "\n",
    "We learned that [CLS] token holds the aggregate representation of the sentence. Thus, we can use the pooler_output as the representation of the given sentence 'I love Paris'. \n",
    "\n",
    "Finally, we have hidden_states and it contains the representation of all the tokens obtained from all the final encoder layers. It is a tuple containing 13 values holding the representation of all encoder layers (hidden layers) starting from the input embedding layer  to the final encoder layer . \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yzF8Qtxu_ndP",
    "outputId": "58848c36-f41f-451f-afc6-3de66a3c2663"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(hidden_states)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-ZBdQEeI_ndP"
   },
   "source": [
    "\n",
    "As we can notice, it contains 13 values holding the representation of all layers. Thus: \n",
    "\n",
    "- hidden_states[0] contains the representation of all the tokens obtained from the input embedding layer \n",
    "- hidden_states[1] contains the representation of all the tokens obtained from the first encoder layer \n",
    "- hidden_states[2] contains the representation of all the tokens obtained from the second encoder layer \n",
    "\n",
    "Similarly, hidden_states[12] contains the representation of all the tokens obtained from the final encoder layer \n",
    "Let's explore this more. First, let's print the shape of the hidden_states[0] which contains the representation of all the tokens obtained from the input embedding layer : \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "CUT57PxO_ndQ",
    "outputId": "8a8be1de-9357-4e87-8837-e4a6b425149d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 7, 768])"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_states[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PPc_0HL3_ndQ"
   },
   "source": [
    "\n",
    "The size [1,7,768] indicates the[batch_size, sequence_length, hidden_size].\n",
    "\n",
    "Now, let's print the shape of hidden_states[1] which contains the representation of all tokens obtained from the first encoder layer : \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3yGnFpzH_ndR",
    "outputId": "7e984ae0-b575-4703-d17a-37a1a5819823"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 7, 768])"
      ]
     },
     "execution_count": 13,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.Size([1, 7, 768])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Xy_0LBQi_ndR",
    "outputId": "d8e2da0c-e37a-4953-a1af-3a34f8ab0622"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 7, 768])"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_states[1].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "46j8VT75_ndR"
   },
   "source": [
    "\n",
    "Thus, in this way, we can obtain the embedding of tokens from all the encoder layers. We learned how to use the pre-trained BERT to extract embeddings, can we also use pre-trained BERT for a downstream task like sentiment analysis? Yes! We will learn about this in the next section. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "3.04. Extracting embeddings from all encoder layers of BERT.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0cc09eca07734399a4d7716f3911d7bd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "26c1ceb389ad46ddbf841770d07a5e93": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2bdc25c58d624b3594fc76892aed8e09",
      "placeholder": "​",
      "style": "IPY_MODEL_f769dd1c71324598b9e27be922759bab",
      "value": " 433/433 [00:14&lt;00:00, 28.9B/s]"
     }
    },
    "2bdc25c58d624b3594fc76892aed8e09": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "437d981ba7c14c39b8cf0e62ee9e055e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4fdbc441590f4e4abc99b64b98169f8a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "7003e521a65b49a9b9727f3f1bfa293f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7706547847d7465299292bb77f2dff25": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7bf00e5dbb204358a0a9985b914db244": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e0a7468cea174a7597f30e619fb99bcb",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a808c5be6f344f3db86cf6e3d26ec81c",
      "value": 231508
     }
    },
    "7c7c261fe4ac435c856fcfd568b4e298": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f5f075e096554be9a389edbd8cf1e089",
      "placeholder": "​",
      "style": "IPY_MODEL_0cc09eca07734399a4d7716f3911d7bd",
      "value": " 440M/440M [00:11&lt;00:00, 39.1MB/s]"
     }
    },
    "84ce79c32f354db9ad3d19acd02625c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "94dc71fe23e040ceae5df4c4dd573684": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7706547847d7465299292bb77f2dff25",
      "max": 433,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_84ce79c32f354db9ad3d19acd02625c1",
      "value": 433
     }
    },
    "95e442aacd1e4aa5b361569341870f47": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7bf00e5dbb204358a0a9985b914db244",
       "IPY_MODEL_a043bae8118f439ca6d5b5a9c818d6b6"
      ],
      "layout": "IPY_MODEL_eb6a67dab3534dae96aaafa975ef3465"
     }
    },
    "9ba62db0115d4930990c3c6d087989c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_94dc71fe23e040ceae5df4c4dd573684",
       "IPY_MODEL_26c1ceb389ad46ddbf841770d07a5e93"
      ],
      "layout": "IPY_MODEL_7003e521a65b49a9b9727f3f1bfa293f"
     }
    },
    "a043bae8118f439ca6d5b5a9c818d6b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ed2edfd213544216a1edf723e587b3be",
      "placeholder": "​",
      "style": "IPY_MODEL_f4060c1c44144dfa9a367d8614708b45",
      "value": " 232k/232k [00:00&lt;00:00, 805kB/s]"
     }
    },
    "a808c5be6f344f3db86cf6e3d26ec81c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "b260f036b5b141188688b837a4845790": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b9b4261f0dc74e9a9210df1ec4273e6a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b260f036b5b141188688b837a4845790",
      "max": 440473133,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4fdbc441590f4e4abc99b64b98169f8a",
      "value": 440473133
     }
    },
    "d70b8c6e81774b3aa9e737d28be1dd48": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b9b4261f0dc74e9a9210df1ec4273e6a",
       "IPY_MODEL_7c7c261fe4ac435c856fcfd568b4e298"
      ],
      "layout": "IPY_MODEL_437d981ba7c14c39b8cf0e62ee9e055e"
     }
    },
    "e0a7468cea174a7597f30e619fb99bcb": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "eb6a67dab3534dae96aaafa975ef3465": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ed2edfd213544216a1edf723e587b3be": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f4060c1c44144dfa9a367d8614708b45": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f5f075e096554be9a389edbd8cf1e089": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f769dd1c71324598b9e27be922759bab": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}