dev/huggingtweets-dev.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.2"
    },
    "colab": {
      "name": "huggingtweets-dev.ipynb",
      "provenance": []
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oXeYEzBG7g0H",
        "colab_type": "text"
      },
      "source": [
        "# HuggingTweets - Tweet Generation with Huggingface"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nZuE848q7g0I",
        "colab_type": "text"
      },
      "source": [
        "*Disclaimer: this project is not to be used to publish any false generated information but to perform research on Natural Language Generation (NLG).*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "668-beiK7g0J",
        "colab_type": "text"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/borisdayma/huggingtweets/blob/master/huggingtweets.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZipizzvM7g0J",
        "colab_type": "text"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "Generating realistic text has become more and more efficient with models such as [GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf). Those models are trained on very large datasets and require heavy computer resources (and time!).\n",
        "\n",
        "However, we can use Transfer Learning and a single GPU to quickly fine-tune a pre-trained model on a given task.\n",
        "\n",
        "We test if we can imitate the writing style of a Twitter user by only using some of his tweets. Twitter API let us download \"only\" the 3200 most recent tweets from any single user, which we then filter out (to remove retweets, short content, etc).\n",
        "\n",
        "[HuggingFace](https://huggingface.co/) gives us an easy access to pre-trained models and fine-tuning techniques for Natural Language Generation (NLG) tasks.\n",
        "\n",
        "We will be monitoring the training with [W&B](https://docs.wandb.com/huggingface) (which is integrated in HuggingFace) to ensure the model is learning from the data and compare multiple experiments.\n",
        "\n",
        "![](https://i.imgur.com/vnejHGh.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wODF0Bx27g0K",
        "colab_type": "text"
      },
      "source": [
        "## Install dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jxRDBrFi7g0L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# install required libraries are not installed\n",
        "!pip install torch -qq\n",
        "!pip install transformers -qq\n",
        "!pip install wandb -qq\n",
        "!pip install tweepy -qq"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4IRQjZw37g0P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# HuggingFace scripts for fine-tuning models and language generation\n",
        "!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/language-modeling/run_language_modeling.py -q\n",
        "!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/text-generation/run_generation.py -q"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "guKG3bqa7g0S",
        "colab_type": "text"
      },
      "source": [
        "## Set up a Twitter Development Account\n",
        "\n",
        "In order to access Twitter data, we need to:\n",
        "\n",
        "* [create a Twitter development account](https://developer.twitter.com/en/apply-for-access)\n",
        "* [create a Twitter app](https://developer.twitter.com/en/apps)\n",
        "* get your consumer API keys: \"API key\" and \"API secret key\"\n",
        "\n",
        "The entire process only takes a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NI1ktZNM7g0S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# <--- Enter your credentials (don't share with anyone) --->\n",
        "consumer_key = 'XXXXXXXX'\n",
        "consumer_secret = 'XXXXXXXX'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zqe4FwQz7g0V",
        "colab_type": "text"
      },
      "source": [
        "## Download tweets from a user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yx3QyAXc7g0V",
        "colab_type": "text"
      },
      "source": [
        "We download latest tweets associated to a user account through [Tweepy](http://docs.tweepy.org/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b4SMOIzp7g0W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tweepy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sReCnnPC7g0Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# authenticate\n",
        "auth = tweepy.AppAuthHandler(consumer_key, consumer_secret)\n",
        "api = tweepy.API(auth)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PXwG7box7g0b",
        "colab_type": "text"
      },
      "source": [
        "We grab all available tweets (limited to 3200 per API limitations) based on Twitter handle.\n",
        "\n",
        "**Note**: Protected users may only be requested when the authenticated user either \"owns\" the timeline or is an approved follower of the owner."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOkUI-XL7g0c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# <--- Enter the screen name of the user you will download your dataset from --->\n",
        "handle = 'elonmusk'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYTl-WMU7g0f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Adapted from https://gist.github.com/onmyeoin/62c72a7d61fc840b2689b2cf106f583c\n",
        "\n",
        "# initialize a list to hold all the tweepy Tweets & list with no retweets\n",
        "alltweets = []\n",
        "\n",
        "# make initial request for most recent tweets with extended mode enabled to get full tweets\n",
        "new_tweets = api.user_timeline(\n",
        "    screen_name=handle, tweet_mode='extended', count=200)\n",
        "\n",
        "if new_tweets:\n",
        "    # save most recent tweets\n",
        "    alltweets.extend(new_tweets)\n",
        "\n",
        "    # save the id of the oldest tweet less one\n",
        "    oldest = alltweets[-1].id - 1\n",
        "\n",
        "    # keep grabbing tweets until the api limit is reached\n",
        "    while True:\n",
        "        # all subsequent requests use the max_id param to prevent duplicates\n",
        "        new_tweets = api.user_timeline(\n",
        "            screen_name=handle, tweet_mode='extended', count=200, max_id=oldest)\n",
        "        \n",
        "        # stop if no more tweets (try a few times as they sometimes eventually come)\n",
        "        if not new_tweets:\n",
        "            break\n",
        "\n",
        "        # save most recent tweets\n",
        "        alltweets.extend(new_tweets)\n",
        "\n",
        "        # update the id of the oldest tweet less one\n",
        "        oldest = alltweets[-1].id - 1\n",
        "        print(f'... {len(alltweets)} tweets downloaded so far')\n",
        "\n",
        "n_tweets = len(alltweets)\n",
        "print(f'Grabbed {n_tweets} tweets')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0F7IRQWVAQqm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# get text and remove RT\n",
        "my_tweets = [tweet.full_text for tweet in alltweets if not hasattr(tweet, 'retweeted_status')]\n",
        "print(f'Found {n_tweets} tweets, including {n_tweets - len(my_tweets)} RT, keeping {len(my_tweets)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z_FXOXqO7g0i",
        "colab_type": "text"
      },
      "source": [
        "## Create a dataset from downloaded tweets\n",
        "\n",
        "We remove:\n",
        "* retweets (since it's not in the wording style of target author)\n",
        "* tweets with no interesting content (limited to url's, user mentionss, \"thank you\"…)\n",
        "\n",
        "We clean up remaining tweets:\n",
        "* we remove url's\n",
        "* we correct special characters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GlLlFGsS7g0i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import random\n",
        "import re\n",
        "import torch"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sMGMTzkW7g0z",
        "colab_type": "text"
      },
      "source": [
        "We verify our list of tweets is well curated."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Trct87MU7g0z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f'Total number of tweets: {len(alltweets)}\\nMy tweets: {len(my_tweets)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VWYjAkFI7g02",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Original tweets\\n')\n",
        "for t in alltweets[:5]:\n",
        "    print(f'{t.full_text}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "999yftxF7g04",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('My tweets\\n')\n",
        "for t in my_tweets[:5]:\n",
        "    print(f'{t}\\n')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a3kEpYv7g09",
        "colab_type": "text"
      },
      "source": [
        "We remove boring tweets (tweets with only urls or too short) and cleanup texts."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vx-1pzGDMhp8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def fix_text(text):\n",
        "    text = text.replace('&amp;', '&')\n",
        "    text = text.replace('&lt;', '<')\n",
        "    text = text.replace('&gt;', '>')\n",
        "    return text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e0FXtb_sMkaC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def clean_tweet(tweet, allow_new_lines = False):\n",
        "    bad_start = ['http:', 'https:']\n",
        "    for w in bad_start:\n",
        "        tweet = re.sub(f\" {w}\\\\S+\", \"\", tweet)      # removes white space before url\n",
        "        tweet = re.sub(f\"{w}\\\\S+ \", \"\", tweet)      # in case a tweet starts with a url\n",
        "        tweet = re.sub(f\"\\n{w}\\\\S+ \", \"\", tweet)    # in case the url is on a new line\n",
        "        tweet = re.sub(f\"\\n{w}\\\\S+\", \"\", tweet)     # in case the url is alone on a new line\n",
        "        tweet = re.sub(f\"{w}\\\\S+\", \"\", tweet)       # any other case?\n",
        "    tweet = re.sub(' +', ' ', tweet)                # replace multiple spaces with one space (makes the previous work worthless?)\n",
        "    if not allow_new_lines:                         # TODO: predictions seem better without new lines\n",
        "        tweet = ' '.join(tweet.split())\n",
        "    return tweet.strip()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qaSI1NZtMn3G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def boring_tweet(tweet):\n",
        "    \"Check if this is a boring tweet\"\n",
        "    boring_stuff = ['http', '@', '#']\n",
        "    not_boring_words = len([None for w in tweet.split() if all(bs not in w.lower() for bs in boring_stuff)])\n",
        "    return not_boring_words < 3"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pw3L46xzNFGV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "curated_tweets = [fix_text(tweet) for tweet in my_tweets]\n",
        "clean_tweets = [clean_tweet(tweet) for tweet in curated_tweets]\n",
        "cool_tweets = [tweet for tweet in clean_tweets if not boring_tweet(tweet)]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3SrMQuZ47g1B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(f'Curated tweets: {len(curated_tweets)}\\nCool tweets: {len(cool_tweets)}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hkXwJxQl7g1D",
        "colab_type": "text"
      },
      "source": [
        "We split data into training and validation sets (90/10)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8dKVICXQ7g1E",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# shuffle data\n",
        "random.shuffle(cool_tweets)\n",
        "\n",
        "# fraction of training data\n",
        "split_train_valid = 0.9\n",
        "\n",
        "# split dataset\n",
        "train_size = int(split_train_valid * len(cool_tweets))\n",
        "valid_size = len(cool_tweets) - train_size\n",
        "train_dataset, valid_dataset = torch.utils.data.random_split(cool_tweets, [train_size, valid_size])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KhjgM8As7g1G",
        "colab_type": "text"
      },
      "source": [
        "We export our datasets as text files, simulating number of epochs by mixing up tweets, due to one batch containing multiple tweets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LIutO2HNvuRg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def make_dataset(dataset, epochs):\n",
        "    total_text = '<|endoftext|>'\n",
        "    tweets = [t for t in dataset]\n",
        "    for _ in range(epochs):\n",
        "        random.shuffle(tweets)\n",
        "        total_text += '<|endoftext|>'.join(tweets) + '<|endoftext|>'\n",
        "    return total_text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4OjOW_4x7g1H",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "EPOCHS = 4\n",
        "\n",
        "with open('{}_train.txt'.format(handle), 'w') as f:\n",
        "    data = make_dataset(train_dataset, EPOCHS)\n",
        "    f.write(data)\n",
        "\n",
        "with open('{}_valid.txt'.format(handle), 'w') as f:\n",
        "    data = make_dataset(valid_dataset, 1)\n",
        "    f.write(data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dKfJrfl57g1J",
        "colab_type": "text"
      },
      "source": [
        "## Log and monitor training through W&B\n",
        "\n",
        "In order to check our model is training correctly and compare experiments, we are going to use the W&B integration from HuggingFace."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cD3v1dDK7g1J",
        "colab_type": "text"
      },
      "source": [
        "### API Key\n",
        "Once you've signed up, run the next cell and click on the link to get your API key and authenticate this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tDu7g4dy7g1K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import wandb\n",
        "wandb.login()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MeCgMAKo7g1N",
        "colab_type": "text"
      },
      "source": [
        "## Fine-tuning the model\n",
        "\n",
        "HuggingFace includes the script `run_language_modeling` making it easy to fine-tune a pre-trained model.\n",
        "\n",
        "We use a pre-trained GPT-2 model and fine-tune it on our dataset.\n",
        "\n",
        "Training is automatically logged on W&B (see [documentation](https://docs.wandb.com/huggingface)). Urls are generated to visualize ongoing runs or you can just open your [dashboard](http://app.wandb.ai/).\n",
        "\n",
        "I quickly tested running for several epochs and my run was showing I started overfitting after 4 epochs so this is the limit I use to fine-tune my model (takes less than 2 minutes).\n",
        "\n",
        "![](https://i.imgur.com/1uIxLFe.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fA0SYlIU7g1N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Associate run to a project (optional)\n",
        "%env WANDB_PROJECT=huggingtweets-dev"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJQoCJV97g1Q",
        "colab_type": "text"
      },
      "source": [
        "We use HuggingFace script `run_language_modeling.py` to fine-tune our model (see [doc](https://huggingface.co/transformers/)).\n",
        "\n",
        "*Note: epochs are built into the dataset*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X4LWV56z7g1Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python run_language_modeling.py \\\n",
        "    --output_dir=output/$handle \\\n",
        "    --overwrite_output_dir \\\n",
        "    --overwrite_cache \\\n",
        "    --model_type=gpt2 \\\n",
        "    --model_name_or_path=gpt2 \\\n",
        "    --do_train --train_data_file=$handle\\_train.txt \\\n",
        "    --do_eval --eval_data_file=$handle\\_valid.txt \\\n",
        "    --evaluate_during_training \\\n",
        "    --eval_steps 20 \\\n",
        "    --logging_steps 20 \\\n",
        "    --per_gpu_train_batch_size 1 \\\n",
        "    --num_train_epochs 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "viSAJ7EE7g1T",
        "colab_type": "text"
      },
      "source": [
        "## Let's test our trained model!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ffHmhI9P7g1T",
        "colab_type": "text"
      },
      "source": [
        "We test our model on a few sample sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uQRNMedK7g1T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "SENTENCES = [\"I think that\",\n",
        "             \"I like\",\n",
        "             \"I don't like\",\n",
        "             \"I want\",\n",
        "             \"My dream is\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ImkNghlH7g1V",
        "colab_type": "text"
      },
      "source": [
        "We use HuggingFace script `run_generation.py` to generate sentences (see [doc](https://huggingface.co/transformers/))."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dK-R3bPP7g1W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import random\n",
        "seed = random.randint(0, 2**32-1)\n",
        "seed"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EkD5CRkn7g1Y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "examples = []\n",
        "num_return_sequences = 5\n",
        "\n",
        "for start in SENTENCES:\n",
        "    val = !python run_generation.py \\\n",
        "        --model_type gpt2 \\\n",
        "        --model_name_or_path output/$handle \\\n",
        "        --length 160 \\\n",
        "        --num_return_sequences $num_return_sequences \\\n",
        "        --temperature 1 \\\n",
        "        --p 0.95 \\\n",
        "        --seed $seed \\\n",
        "        --prompt {'\"<|endoftext|>' + start + '\"'}\n",
        "    generated = [val[-1-2*k] for k in range(num_return_sequences)[::-1]]\n",
        "    print(f'\\nStart of sentence: {start}')\n",
        "    for i, g in enumerate(generated):\n",
        "        g = g.replace('<|endoftext|>', '')\n",
        "        print(f'* Generated #{i+1}: {g}')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6kHI7_D7g1g",
        "colab_type": "text"
      },
      "source": [
        "## About\n",
        "\n",
        "*Built by Boris Dayma*\n",
        "\n",
        "[![Follow](https://img.shields.io/twitter/follow/borisdayma?style=social)](https://twitter.com/intent/follow?screen_name=borisdayma)\n",
        "\n",
        "My main goals with this project are:\n",
        "* to experiment with how to train, deploy and maintain neural networks in production ;\n",
        "* to make AI accessible to everyone ;\n",
        "* to have fun!\n",
        "\n",
        "For more details, visit the project repository.\n",
        "\n",
        "[![GitHub stars](https://img.shields.io/github/stars/borisdayma/huggingtweets?style=social)](https://github.com/borisdayma/huggingtweets)\n",
        "\n",
        "**Disclaimer: this project is not to be used to publish any false generated information but to perform research on Natural Language Generation.**\n",
        "\n",
        "## Resources\n",
        "\n",
        "* [Explore the W&B report](https://app.wandb.ai/wandb/huggingtweets/reports/HuggingTweets-Train-a-model-to-generate-tweets--VmlldzoxMTY5MjI) to understand how the model works\n",
        "* [HuggingFace and W&B integration documentation](https://docs.wandb.com/library/integrations/huggingface)\n",
        "\n",
        "## Got questions about W&B?\n",
        "\n",
        "If you have any questions about using W&B to track your model performance and predictions, please reach out to the [slack community](http://bit.ly/wandb-forum)."
      ]
    }
  ]
}