NLP_Deep_Dive_10.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPU+WEVQFblwFOh4o0es0wS",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ashikshafi08/Learning-Fastai/blob/main/NLP_Deep_Dive_10.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D1FJLJDf2JIA",
        "outputId": "cb0947cc-2c04-4afa-87dc-4ca79ccaa53f"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Wed Jan 25 02:53:55 2023       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   72C    P0    31W /  70W |  12670MiB / 15360MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|    0   N/A  N/A      3986      C                                   12667MiB |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "-BrdQGOiVLEu"
      },
      "outputs": [],
      "source": [
        "import fastai \n",
        "from fastai.vision.all import * \n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece!=0.1.90,!=0.1.91"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yJonwUU2AZ_Y",
        "outputId": "c6905eb4-7c77-417a-bb25-0997884b07f3"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: sentencepiece!=0.1.90,!=0.1.91 in /usr/local/lib/python3.8/dist-packages (0.1.97)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Language model -> a model that has been trained to guess the next word in a text. This is a self supervised learning, hence we dont give labels to our model just feed it lots of texts. \n",
        "\n",
        "**Self Supervised Learning**\n",
        "Training a model using labels that are embedded in the independent variable (X) rather than requiring a external labels. For instance, training a model to predict the next word. \n",
        "\n",
        "\n",
        "Fine tune our language model for IMDB rather pretraining it with the Wikipedia texts. \n",
        "\n",
        "Fine tune the language model prior to fine-tuning the classification model. For instance, IMDb sentiment analysis task, the dataset includes 50,000 additional movie reviews that dont have any labels to them. We can use these reviews to fine tune the pretrained language model, which trained only on Wikipedia articles this will help in predicting the next word for.\n",
        "Fine tune the language model prior to fine-tuning the classification model. For instance, IMDb sentiment analysis task, the dataset includes 50,000 additional movie reviews that dont have any labels to them. We can use these reviews to fine tune the pretrained language model, which trained only on Wikipedia articles this will help in predicting the next word of a movie review. \n",
        "\n",
        "\n",
        "#### Text Processing \n",
        "\n",
        "First we concatentate all of the documents in our dataset into one big long strings and split it into words (or tokens), this is gives us the long list of words. \n",
        "\n",
        "Vocab will contain mix of common words that are in the vocbulary of the pretrained model we have and also new words specific to our corpus. \n",
        "\n",
        "- Our embedding matrix will be built accordingly for words that are in the vocabulary of our pretrained model. \n",
        "- For the new words we initialize a random vector. \n",
        "\n",
        "\n",
        "The steps are as follows: \n",
        "1. Tokenization \n",
        "  - Convert text into a list of words (substring, characters etc.) \n",
        "2. Numericalizaion\n",
        "  - Convert the vocab (list of all unique words) into number by looking up its index in the vocab. \n",
        "3. Language model data loader creation \n",
        "  - In here `LMDataLoader` class that automatically handles creating a dependent varibale (y) that is offset from the independent variable (x). \n",
        "  - It also does the shuffle of the training data in a way where X and y maintain their structure. \n",
        "4. Language model creation\n",
        "  - building from scratch -> RNN, for now a deep neural network. \n",
        "\n",
        "### Tokenization \n",
        "\n",
        "How do we need how to split the words? The punctuation, other chemical corpus, hyphenated words etc. Even languages like Japenese and Chinese that dont uses the convention english words. \n",
        "\n",
        "There are three tokenization methods: \n",
        "1. Word-based -> split on spaces, general punctuation marks.\n",
        "2. Subword based -> into smaller parts \n",
        "3. Character based -> split sentence into its individual characters. \n",
        "\n",
        "\n",
        "**Token** -> one element of a list created by the tokenization process. \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "blhs2dcAVaqW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Word Tokenization with fastai "
      ],
      "metadata": {
        "id": "rGhE_2_LZJ0e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from fastai.text.all import * \n",
        "path = untar_data(URLs.IMDB)\n",
        "\n",
        "path.ls()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9h9jFFpteAOX",
        "outputId": "3062b2c4-a558-4a58-93f9-b80d11de0e77"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(#7) [Path('/root/.fastai/data/imdb/unsup'),Path('/root/.fastai/data/imdb/imdb.vocab'),Path('/root/.fastai/data/imdb/train'),Path('/root/.fastai/data/imdb/tmp_lm'),Path('/root/.fastai/data/imdb/test'),Path('/root/.fastai/data/imdb/README'),Path('/root/.fastai/data/imdb/tmp_clas')]"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "path.ls()[:5]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hY1ZisD7eHNv",
        "outputId": "86790447-cab8-4c8d-bdf4-42b2df8a3957"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(#5) [Path('/root/.fastai/data/imdb/unsup'),Path('/root/.fastai/data/imdb/imdb.vocab'),Path('/root/.fastai/data/imdb/train'),Path('/root/.fastai/data/imdb/tmp_lm'),Path('/root/.fastai/data/imdb/test')]"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Getting the text files \n",
        "files = get_text_files(path , folders = ['train', 'test', 'unsup'])"
      ],
      "metadata": {
        "id": "oz5yxXyneRcM"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "(path / \"unsup\").ls()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tUf3slAWehws",
        "outputId": "72cbb515-463f-4763-80bf-e4d22dc483d3"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(#50000) [Path('/root/.fastai/data/imdb/unsup/2087_0.txt'),Path('/root/.fastai/data/imdb/unsup/31436_0.txt'),Path('/root/.fastai/data/imdb/unsup/15120_0.txt'),Path('/root/.fastai/data/imdb/unsup/43451_0.txt'),Path('/root/.fastai/data/imdb/unsup/7427_0.txt'),Path('/root/.fastai/data/imdb/unsup/49120_0.txt'),Path('/root/.fastai/data/imdb/unsup/8469_0.txt'),Path('/root/.fastai/data/imdb/unsup/4359_0.txt'),Path('/root/.fastai/data/imdb/unsup/7972_0.txt'),Path('/root/.fastai/data/imdb/unsup/132_0.txt')...]"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "files[0].open().read()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        },
        "id": "XO-W_wJxexfR",
        "outputId": "54b548f3-b2ba-463d-f20d-ae92c7b09f09"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Seven years ago, when I was in high school, I was assigned to read and study John Steinbeck\\'s 1937 novella, \"Of Mice and Men\", in my English class, and it\\'s no surprise that I still remember it clearly now, as it\\'s not a story to forget. Shortly after reading the book, we were shown the 1992 film adaptation in that class. I first heard about this much earlier adaptation of the book around that time, but it took me seven years to finally get around to watching it, even though I had been intending to watch it for quite a while by the time I did. I knew this version was widely praised, which gave me fairly high expectations. It may not have fully met these expectations, but if not, at least it came very close, and I would say it\\'s just as memorable as the book.<br /><br />In the 1930\\'s Great Depression era, George Milton and Lennie Small are two migrant ranch workers in California who dream of having their own place someday. Lennie is big and strong, but he is also mentally handicapped, and George has to look after him. He means well, but often gets into trouble, doing things he doesn\\'t realize are wrong, and George is always the one who has to get him out of it. After leaving town to escape from a mob started by Lennie feeling a woman\\'s dress, the two of them soon get jobs on another ranch. Here, their co-workers include Candy, an elderly man who won\\'t likely be able to work on the ranch for much longer, and Crooks, a crippled black man who faces discrimination because of his race in this era. The ranch owner\\'s son is Curley, a belligerent young man who hates Lennie from the moment he first sees him, and has a lonely wife named Mae, whom he tends to neglect, but doesn\\'t want any other men to have anything to do with her, being the jealous type.<br /><br />The cast really helps carry this film, including the two co-stars, Burgess Meredith as George and Lon Chaney Jr. (the son of the \"Man of a Thousand Faces,\" Lon Chaney) as Lennie. Both of these actors fit their roles well, and are both memorable. Chaney\\'s portrayal of a mentally handicapped character may not be flawless, but it\\'s still good. Most of the other cast members\\' performances also stand out, and it helps that they were clearly given a good script. There are times when the score doesn\\'t fit the scene very well, and there may be certain other slight flaws, but they are far from enough to ruin the movie. In addition to the acting and script, there are a number of very interesting characters, several of whom one can easily sympathize with, and some very well crafted heartbreaking scenes, starting with Candy losing his dog. The score in these scenes does fit them well. This adaptation of Steinbeck\\'s novella was also very nicely filmed.<br /><br />This first ever film adaptation of a John Steinbeck story is not perfect, but overall, it\\'s still very good, and very faithful to the book. It does take some liberties, such as giving Curley\\'s wife a name, and they obviously left out some detail, but the differences are minor, at least for the most part. Anyone who has read the book knows that \"Of Mice and Men\" is not a cheerful story. It\\'s a tragic drama set in a tragic era, and this movie is much the same as the book, so keep that in mind if you\\'re planning to watch it. If you\\'re an emotional kind of person, you could easily find tears in your eyes at times, and if you\\'re not, you could still at least come close to tears. If you\\'ve read the book and were impressed with it, and/or you like dramas from the classic era in general, I really think this 1939 screen adaptation is worth checking out.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Example text \n",
        "\n",
        "txt = files[0].open().read()\n",
        "txt[:40]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "D28bCEwoekf1",
        "outputId": "6359c9fd-0505-4655-f49a-1ec1ef547ee1"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Seven years ago, when I was in high scho'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Rather using the spacy's tokenizer `SpaceTokenizer` we will use the fastai's `WordTokenizer`. \n",
        "\n",
        "`coll_repr(collection , n)` -> function to display the results. n items of the collection. "
      ],
      "metadata": {
        "id": "SkeIx93zeyDl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Spacy's Tokenizer \n",
        "spacy = WordTokenizer()\n",
        "\n",
        "\n",
        "# Tokens -> convert words into tokens\n",
        "# [txt] -> strings into a list of strings\n",
        "toks = first(spacy([txt]))\n",
        "\n",
        "# Display the result \n",
        "print(coll_repr(toks , 30))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ubcD6tggfRlC",
        "outputId": "44b6d534-0e28-4c8b-a8c1-6ba0925c45d5"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(#773) ['Seven','years','ago',',','when','I','was','in','high','school',',','I','was','assigned','to','read','and','study','John','Steinbeck',\"'s\",'1937','novella',',','\"','Of','Mice','and','Men','\"'...]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "It also splits the term `I'd -> I 'd` as a seperate token which is true. Spacy handles this for us. "
      ],
      "metadata": {
        "id": "-E8O-iEnf3XR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Fastai's tokenizer \n",
        "\n",
        "tkn = Tokenizer(spacy) # Takes in the tokenizer\n",
        "print(coll_repr(tkn(txt) , 31))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qcpdRUYKgIdL",
        "outputId": "8856195a-3bba-4a0d-8697-f51af24b0253"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(#841) ['xxbos','xxmaj','seven','years','ago',',','when','i','was','in','high','school',',','i','was','assigned','to','read','and','study','xxmaj','john','xxmaj','steinbeck',\"'s\",'1937','novella',',','\"','of','xxmaj'...]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The tokens that start with the characters `xx` are special tokens. \n",
        "\n",
        "We are simplifying the english language to a simplified tokenized language for the model to learn better. \n",
        "\n",
        "- Capitalized word will be replaced with a special capitilization token, followed by the lowercase version of the word. XXCAP token -> lowercase word of the captial. \n",
        "\n",
        "By this tokens we can save the compute and the memory resources, and these tokens determine whether its capital or not. \n",
        "\n",
        "\n",
        "- xxbos -> beginning of stream, start of a new text. \n",
        "- xxmaj -> indicates next word begin with a capital. \n",
        "- xxunk -> indicates next word in unknown\n"
      ],
      "metadata": {
        "id": "lOq8kE9MgQTP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# The rules used \n",
        "defaults.text_proc_rules\n",
        "\n",
        "# Read page 335 for more info on these "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VzpTCqFwnvVY",
        "outputId": "693ab9cd-bd0d-4607-cf87-edd99de5c7db"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<function fastai.text.core.fix_html(x)>,\n",
              " <function fastai.text.core.replace_rep(t)>,\n",
              " <function fastai.text.core.replace_wrep(t)>,\n",
              " <function fastai.text.core.spec_add_spaces(t)>,\n",
              " <function fastai.text.core.rm_useless_spaces(t)>,\n",
              " <function fastai.text.core.replace_all_caps(t)>,\n",
              " <function fastai.text.core.replace_maj(t)>,\n",
              " <function fastai.text.core.lowercase(t, add_bos=True, add_eos=False)>]"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Subword Tokenization \n",
        "\n",
        "For chinese and japanese letters dont use spaces, and not well defined into the category word as per the english corpus. So we use subwords to tackle these problems. \n",
        "\n",
        "We handle this: \n",
        "- Analyze the corpus of documents to find the most commonly occuring groups of letters. These becomes the vocab. \n",
        "- Tokenize the corpus using this vocab of subword units. \n",
        "\n"
      ],
      "metadata": {
        "id": "UUv9dLIfnyeo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Iterating to the corpus and storing them into a list\n",
        "\n",
        "txts = L(o.open().read() for o in files[:2000])"
      ],
      "metadata": {
        "id": "zQvVV5pm9x1h"
      },
      "execution_count": 43,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we need to find the common sequences of characters in these excerpts to create a vocab. \n",
        "\n",
        "We can use the `setup` special fastai method for our data processing pipelines. "
      ],
      "metadata": {
        "id": "t-ODnf49-XVN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def subword(sz):\n",
        "  sp = SubwordTokenizer(vocab_sz=sz)\n",
        "  sp.setup(txts)\n",
        "  return \" \".join(first(sp([txt]))[:40])"
      ],
      "metadata": {
        "id": "Iuy1bthq90HT"
      },
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "subword(200)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "O6yKW2DR_5Wb",
        "outputId": "3dadcc5d-0186-4c94-843b-296a2e85f9cf"
      },
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
              "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'▁S e ve n ▁ y e ar s ▁a g o , ▁w h en ▁I ▁was ▁in ▁h i g h ▁ s ch o o l , ▁I ▁was ▁a s s i g n ed ▁to'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "subword(1000)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "wcb3GK5pAW1b",
        "outputId": "6bebe9b2-02a6-49b0-9351-f54ba0ed2272"
      },
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
              "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": []
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'▁Se ven ▁years ▁a go , ▁when ▁I ▁was ▁in ▁high ▁school , ▁I ▁was ▁as s ign ed ▁to ▁read ▁and ▁st u dy ▁John ▁St e in be ck \\' s ▁19 3 7 ▁novel la , ▁\"'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 46
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we use a smaller vocab size, each token will represent fewer characters and takes more tokens to represent a sentence. \n",
        "\n",
        "`_` -> represent the special character \n",
        "\n",
        "**Larger vocab** -> fewer tokens per sentence \n",
        "\n",
        "  --> fastest training, less memory and less state for the model \n",
        "  --> downside : larger embedding matrices, require more data to learn \n",
        "\n",
        "Subword tokenization -> scale between character tokenization and word tokenization. \n",
        "\n",
        "Importantly it can handle every human language without any specific algorithm."
      ],
      "metadata": {
        "id": "u8s-X0GdEzmu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Numericalization with fastai \n",
        "\n",
        "Mapping tokens to integers, identical to that of creating a Category variable like in MNIST. \n",
        "\n",
        "1. Make a list of all possible levels of that categorical variable (we call this vocab). \n",
        "- Replace each level with its index in the vocab. \n",
        "\n",
        "\n",
        "tokenization is done in parallel by fastai. \n",
        "\n",
        "\n",
        "This is how it works: \n",
        "- Have a vocab \n",
        "- Fit the vocab to a `Numericalize` and create an object\n",
        "- Use the numericalize object to convert tokens into numbers. "
      ],
      "metadata": {
        "id": "H3S9k3qlM4_F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Samples tokens for the demo \n",
        "\n",
        "# Map function to use the tokenizer with the texts we need.\n",
        "toks200 = txts[:200].map(tkn)\n",
        "toks200[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D2KtXbzlNgHX",
        "outputId": "e9b36704-e68e-4c68-9a21-899a2fbd6c22"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(#841) ['xxbos','xxmaj','seven','years','ago',',','when','i','was','in'...]"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Numericalize \n",
        "num = Numericalize() \n",
        "\n",
        "# Fitting / or setup in fastai \n",
        "num.setup(toks200)\n",
        "\n",
        "# Displaying\n",
        "coll_repr(num.vocab , 20)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "kSsZmSdWNvq0",
        "outputId": "de5e8c14-7100-4215-9e35-baff9a175833"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"(#2128) ['xxunk','xxpad','xxbos','xxeos','xxfld','xxrep','xxwrep','xxup','xxmaj','the','.',',','and','a','of','to','is','i','it','in'...]\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Two parameters of the `Numericalize()`\n",
        "- max_vocab : instead of adding all the words, it restricts and replacing all words other than the most common 60,000 with some special unknown token word **`xxunk`**. \n",
        "\n",
        "- min_freq : number of times a word can reappear \n",
        "\n",
        "The `max_vocab` is used to avoid having a large embedding matrix, so it wont slow down the training and dont end up using too much memory. \n",
        "\n",
        "`vocab` -> pass in the custom vocab variable we have. "
      ],
      "metadata": {
        "id": "0aSJR2FeN7Za"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nums = num(toks)[:30]\n",
        "nums"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "92Afw5J4O2vF",
        "outputId": "7c64f919-7bd9-45dc-a296-96a74b0bb2b9"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorText([  0, 176, 993,  11,  82,   0,  25,  19, 286, 242,  11,   0,  25,\n",
              "              0,  15, 458,  12,   0,   0,   0,  23,   0,   0,  11,  22,   0,\n",
              "              0,  12,   0,  22])"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can check by mapping them back to the original text "
      ],
      "metadata": {
        "id": "xzSHvaq2Pb8w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\" \".join(num.vocab[o] for o in nums)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "QLPEsrRFPlIR",
        "outputId": "787488cc-e668-436f-c66d-e8cf2af27fed"
      },
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'xxunk years ago , when xxunk was in high school , xxunk was xxunk to read and xxunk xxunk xxunk \\'s xxunk xxunk , \" xxunk xxunk and xxunk \"'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Putting our text into Batches for a Language Model \n",
        "\n",
        "We can't resize text and we want our language model to read the text in order so that they can predict what the next word is. \n",
        "\n",
        "| **So this means each new batch should begin precisely where the previous one left off** |\n",
        "\n",
        "\n",
        "So we need to divide this array more finely into subarrays of a fixed sequence length, we need to obtain the order within and across these subarrays.\n",
        "\n",
        "We shuffle order of the documents not the order of the words insid. \n",
        "\n",
        "For instance, if we have 50000 tokens and a batch size of 10. Then we will preserve the order of the tokens (1 - 5000 for the first mini-stream / mini-batch, then 5001 to 10000 and so on...) \n",
        "\n",
        "With the special tokens xxbos the model knows which one is the beginning of the excerpt or the document. \n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "nVcdzKckPqfs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Numericalize the tokens \n",
        "nums200 = toks200.map(num)\n",
        "\n",
        "# Pass into the LMDataLoader \n",
        "dl = LMDataLoader(nums200)\n",
        "\n",
        "dl"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-YfKdKteQerb",
        "outputId": "ab5d56b8-5513-41f4-ec03-246fad10dac0"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<fastai.text.data.LMDataLoader at 0x7ff54de63a60>"
            ]
          },
          "metadata": {},
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Looking into the dataloader \n",
        "x , y = first(dl)\n",
        "x.shape,  y.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AnUxc0fXS9Lg",
        "outputId": "e4e80047-a22e-4785-c74d-0fa76e2fa2ba"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(torch.Size([64, 72]), torch.Size([64, 72]))"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "doc(LMDataLoader) # Default seq len is 72 and bs is 64 "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "loQ_balmTIMt",
        "outputId": "b85e798e-32c4-4e06-b438-03cd3c5047b1"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LMDataLoader(dataset, lens=None, cache=2, bs=64, seq_len=72, num_workers=0, *, shuffle: 'bool' = False, verbose: 'bool' = False, do_setup: 'bool' = True, pin_memory=False, timeout=0, batch_size=None, drop_last=False, indexed=None, n=None, device=None, persistent_workers=False, pin_memory_device='', wif=None, before_iter=None, after_item=None, before_batch=None, after_batch=None, after_iter=None, create_batches=None, create_item=None, create_batch=None, retain=None, get_idxs=None, sample=None, shuffle_fn=None, do_batch=None)\n",
            "A `DataLoader` suitable for language modeling\n",
            "\n",
            "To get a prettier result with hyperlinks to source code and documentation, install nbdev: pip install nbdev\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Looking into the X \n",
        "\" \".join(num.vocab[o] for o in x[1][:20])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "36OkOXygTKoL",
        "outputId": "4b9d16ef-6895-47ef-8eb4-05e1e81e90b2"
      },
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"are xxunk with you it just gets worse . xxmaj about half the film xxunk xxmaj ash 's struggle against\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Looking into the y \n",
        "\" \".join(num.vocab[o] for o in y[1][:20])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "d1yd-iLbTzBa",
        "outputId": "939d31ba-4d65-4f47-dc6a-ca3990dda168"
      },
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "\"xxunk with you it just gets worse . xxmaj about half the film xxunk xxmaj ash 's struggle against xxunk\""
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Training a Text Classifier \n",
        "\n",
        "Two steos to train the sota text classifier using transfer learning\" \n",
        "- fine tune our language model pretrained on Wikipedia to the corpus of IMDB review. \n",
        "- then use the above to train the classifier. \n",
        "\n",
        "\n",
        "TextBlock -> tokenization and numericalization happens automatically when `TextBlock` is passed to the `DataBlock`. \n",
        "\n"
      ],
      "metadata": {
        "id": "cOoe7IuVT6uN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "get_imdb = partial(get_text_files , folders = [\"train\", \"test\" , \"unsup\"])\n",
        "get_imdb"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6N9WN3GjVE9l",
        "outputId": "aa141436-b6c3-470e-ca90-223c9fb50fa6"
      },
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "functools.partial(<function get_text_files at 0x7ff64193a8b0>, folders=['train', 'test', 'unsup'])"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Building a Text DataBlock \n",
        "dblock_lm = DataBlock(\n",
        "    blocks = TextBlock.from_folder(path , is_lm = True), \n",
        "    get_items = get_imdb , \n",
        "    splitter = RandomSplitter(0.1)\n",
        ")"
      ],
      "metadata": {
        "id": "3rzfWWHEWeAv"
      },
      "execution_count": 57,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# DataLoaders \n",
        "dls_lm = dblock_lm.dataloaders(path , path = path , bs = 128 , seq_len = 80)"
      ],
      "metadata": {
        "id": "X02d9fWnWe0Q"
      },
      "execution_count": 58,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "TextBlock.from_folder -> classmethods \n",
        "\n",
        "https://stackoverflow.com/questions/12179271/meaning-of-classmethod-and-staticmethod-for-beginner"
      ],
      "metadata": {
        "id": "SbrB_awwXwHk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Showing the results \n",
        "dls_lm.show_batch(max_n = 3)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 317
        },
        "id": "7LYJ-OuvYNkd",
        "outputId": "131c8de7-4eb3-47f7-e7f6-1df29bb8a3f3"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>text_</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>xxbos xxmaj but certainly a serious contender for one of the worst 10 of all time . \\n\\n i got this xxup dvd cheap , with xxmaj sandra xxmaj bullock as headliner on the case . xxmaj this is false advertising - she 's on - screen for almost 10 minutes of the movie . \\n\\n xxmaj on the other hand , there was no other selling point for this movie - the dialog was horrible , the editing was</td>\n",
              "      <td>xxmaj but certainly a serious contender for one of the worst 10 of all time . \\n\\n i got this xxup dvd cheap , with xxmaj sandra xxmaj bullock as headliner on the case . xxmaj this is false advertising - she 's on - screen for almost 10 minutes of the movie . \\n\\n xxmaj on the other hand , there was no other selling point for this movie - the dialog was horrible , the editing was apparently</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>so much depth . xxmaj the xxmaj doctor seems more vulnerable in this series , maybe because he is the last xxunk . xxmaj the range of emotions which xxmaj chris and xxmaj david show are truly remarkable , and i felt their pain , anger , and sense of adventure right along with them . \\n\\n xxmaj billie xxmaj piper ( rose ) brings a very human element to each episode , although she is much braver than i</td>\n",
              "      <td>much depth . xxmaj the xxmaj doctor seems more vulnerable in this series , maybe because he is the last xxunk . xxmaj the range of emotions which xxmaj chris and xxmaj david show are truly remarkable , and i felt their pain , anger , and sense of adventure right along with them . \\n\\n xxmaj billie xxmaj piper ( rose ) brings a very human element to each episode , although she is much braver than i would</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>xxunk xxunk xxmaj jackson and xxmaj eugene xxmaj xxunk a bad xxunk 's sort of in the vein of \" 48 xxunk i 'm not equating the 2 movies with each other , though i do seem to recall being less than impressed with \" 48 xxunk did however find \" the xxmaj man \" entertaining from beginning to end , from both a dramatic viewpoint and a comedic one.the 2 stars are both very gifted actors , so it</td>\n",
              "      <td>xxunk xxmaj jackson and xxmaj eugene xxmaj xxunk a bad xxunk 's sort of in the vein of \" 48 xxunk i 'm not equating the 2 movies with each other , though i do seem to recall being less than impressed with \" 48 xxunk did however find \" the xxmaj man \" entertaining from beginning to end , from both a dramatic viewpoint and a comedic one.the 2 stars are both very gifted actors , so it was</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Fine tune the Language Model \n",
        "\n",
        "In order to convert the integer indices to activations for our neural network, we will use embeddings. \n",
        "\n",
        "And feed these embeddings into a model architecture. \n",
        "\n",
        "The embedding of the pretrained model are merged with the random embeddings added for the words werent in the pretraining vocab. \n"
      ],
      "metadata": {
        "id": "eNVk-VqZY06u"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "learn = language_model_learner(\n",
        "    dls_lm , AWD_LSTM , drop_mult= .3 , \n",
        "    metrics = [accuracy , Perplexity()]\n",
        ").to_fp16()"
      ],
      "metadata": {
        "id": "P1xrCmQrZQHZ"
      },
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Perplexity is the often used metric in the language model, it is the exponential loss (torch.exp(cross_entropy)). \n",
        "\n",
        "`language_model_learner` -> will automatically calls the freeze using pretrained model so this will train only on the randomly initialized embedding weights. \n",
        "\n"
      ],
      "metadata": {
        "id": "SRUgNgd_Z0Nm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "learn.fit_one_cycle(1, 2e-2)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "adtteacjaF2_",
        "outputId": "f50dd068-34f2-4523-8a97-34c5350ff7a0"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "<style>\n",
              "    /* Turns off some styling */\n",
              "    progress {\n",
              "        /* gets rid of default border in Firefox and Opera. */\n",
              "        border: none;\n",
              "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
              "        background-size: auto;\n",
              "    }\n",
              "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
              "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
              "    }\n",
              "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
              "        background: #F44336;\n",
              "    }\n",
              "</style>\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>perplexity</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>3.996865</td>\n",
              "      <td>3.896230</td>\n",
              "      <td>0.301687</td>\n",
              "      <td>49.216530</td>\n",
              "      <td>25:51</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "H7DNvYoY2H61"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Encoder** \n",
        "\n",
        "Once the training after unfreezing done, we save all of our model except the final layer that converts activatiosn into probabilities of picking each token in our vocabulary. \n",
        "\n",
        "\n",
        "The *encoder* has all the layers of model responsible for the feature extraction except the final layer. "
      ],
      "metadata": {
        "id": "9KhFJyB8av4r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In Pytorch Dataloaders, we need to collate all the items in a batch into a single tensor has a fixed shape. So we use padding to expand the shortest texts to make them all the same size. \n",
        "\n",
        "The special padding token we will be assigning to adjust the lenght will be ignored by the model. \n",
        "\n",
        "We do this by approximating the traning set, sorting the documents by length prior to each epoch. We wont pad every batch to the same size instead use the size of the largest document in each batch as the target size. "
      ],
      "metadata": {
        "id": "X-lAmQGlbkqc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The class Numericalize has a decode function which gives us back the string with tokens. \n",
        "\n",
        "`numericalizeObject.decode()`"
      ],
      "metadata": {
        "id": "a_1To9HZ2suY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "nums_dec = num.decode(nums)\n",
        "\n",
        "nums_dec"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4swU-pyD26zS",
        "outputId": "50133efd-af51-4be4-f61e-9fbb77d29be7"
      },
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(#30) ['xxunk','years','ago',',','when','xxunk','was','in','high','school'...]"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`Tokenizer.decode` turns the above tokens into a single string. "
      ],
      "metadata": {
        "id": "eeG66aH83BZn"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "tkn.decode(nums_dec)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "NaYNBkZQ3Ji2",
        "outputId": "db0b56aa-6bfa-4fc5-89ca-e0b765af277c"
      },
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'xxunk years ago , when xxunk was in high school , xxunk was xxunk to read and xxunk xxunk xxunk \\'s xxunk xxunk , \" xxunk xxunk and xxunk \"'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Fine tuning the classifier \n",
        "\n",
        "For NLP Classifier its found that unfreezing a few layers at a time makes a real difference in the performance of the model. \n",
        "\n",
        "`freeze_to` -> -2 value all layers except the last two. \n"
      ],
      "metadata": {
        "id": "4LBnQ0GvzuYm"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "wxcIabQS0Asg"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}