pdftotext.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Module imports\n",
    "import requests\n",
    "import os\n",
    "from io import BytesIO\n",
    "from bs4 import BeautifulSoup\n",
    "import pickle\n",
    "import PyPDF2\n",
    "import re\n",
    "import string\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import stopwords\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Guidance and sample code taken from https://medium.com/@rqaiserr/how-to-convert-pdfs-into-searchable-key-words-with-python-85aab86c544f and https://github.com/adashofdata/nlp-in-python-tutorial/blob/master/1-Data-Cleaning.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to get text out of the PDFs we downloaded earlier. To do that, we use the PyPDF2 library to extract the PDF text. The below function takes a PDF file and returns a text string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pdf_to_text(filename):\n",
    "    with open(filename,'rb') as pdfFileObj:\n",
    "        pdfReader = PyPDF2.PdfFileReader(pdfFileObj)\n",
    "        # this is the readable object to parse\n",
    "        numpages = pdfReader.numPages\n",
    "        count = 0\n",
    "        text = \"\"\n",
    "        while count < numpages:\n",
    "            pageObj = pdfReader.getPage(count)\n",
    "            count += 1\n",
    "            text += pageObj.extractText()\n",
    "        assert text != \"\", \"Text not readable from %s\" % filename\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the opinion in plain text form, we need to clean it up. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Well, first let's see if it works on one of the opinions. \n",
    "# Get the current directory:\n",
    "currentdir = os.path.abspath('')\n",
    "# Make a filepath to pass to PDFtotext:\n",
    "testpath = os.path.join(currentdir, 'Opinions\\cl_20\\B228808.pdf')\n",
    "testpdf = pdf_to_text(testpath)\n",
    "print(testpdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, excellent! Comparing the readout to the original PDF, there are some extraneous line breaks and spaces, but the text appears to be mostly present. Some things we might have to fix: footnotes interrupting other text, special characters causing following words not to be read, spaces breaking up names/keywords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Documentation from PyPDF2 for the extractText() function: \n",
    "\n",
    "extractText()\n",
    "\n",
    "    Locate all text drawing commands, in the order they are provided in the content stream, and extract the text. This works well for some PDF files, but poorly for others, depending on the generator used. This will be refined in the future. Do not rely on the order of text coming out of this function, as it will change if this function is made more sophisticated.\n",
    "    Returns:\ta unicode string object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like if I'm relying on PyPDF2 to extract the text, I'm going to have to work with incomplete text for right now. Maybe I can remove all the line breaks and compare word counts for the documents to check how many words got missed?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pickle output for later use\n",
    "pickletest = os.path.join(currentdir, 'textopinions','B228808.txt')\n",
    "with open(pickletest, \"wb\") as file:\n",
    "    pickle.dump(testpdf, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drawing from https://github.com/adashofdata/nlp-in-python-tutorial/blob/master/1-Data-Cleaning.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial round of cleanup\n",
    "def cleantext1(text): \n",
    "    # Make text lowercase: \n",
    "    text = text.lower()\n",
    "    # Strip all newlines: \n",
    "    text = re.sub('\\n', '', text)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(cleantext1(testpdf))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "testclean = cleantext1(testpdf)\n",
    "# Pickle output for later use\n",
    "pickletestclean = os.path.join(currentdir, 'textopinions','B228808_clean.txt')\n",
    "with open(pickletest, \"wb\") as file:\n",
    "    pickle.dump(testclean, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use Scikit-Learn's feature extraction module to get a count for each word in the text, ignoring commonly used words like \"the\" and \"and\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "cv = CountVectorizer(input='content',stop_words = 'english')\n",
    "data_cv = cv.fit_transform([testclean])\n",
    "data_dtm = pd.DataFrame(data_cv.toarray(), columns = cv.get_feature_names())\n",
    "# sort by frequency\n",
    "data_sort = data_dtm.sort_values(by=0,axis=1, ascending=False)\n",
    "# The resulting matrix is tall by default. \n",
    "data_sort = data_sort.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've checked one file, time to do the other 20,000.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batchconvert(folder):\n",
    "    for file in os.listdir(folder):\n",
    "        try:\n",
    "            if file.endswith('.pdf'):\n",
    "                casenum = os.path.splitext(file)[0]\n",
    "                casename = '%s.txt' % casenum\n",
    "                savepath = os.path.join(currentdir, 'textopinions', casename)\n",
    "                with open(savepath, \"wb\") as textfile:\n",
    "                    pickle.dump(pdf_to_text(os.path.join(folder, file)),textfile)\n",
    "        except:\n",
    "            print('Error occurred during %s ' % casenum)\n",
    "            continue\n",
    "    return"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = os.path.join(currentdir, 'Opinions','cl_15')\n",
    "batchconvert(folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = os.path.join(currentdir, 'Opinions','cl_20')\n",
    "batchconvert(folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Make it easy to start up again if it stops, handle errors, or both. Ideally both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = os.path.join(currentdir, 'Opinions','cl_25')\n",
    "batchconvert(folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = os.path.join(currentdir, 'Opinions','cl_30')\n",
    "batchconvert(folder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "folder = os.path.join(currentdir, 'Opinions','cl_10')\n",
    "batchconvert(folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we should have all the opinions in text form. Now we can load the pickled files and associate each case number with its corpus, check to see that it came from the Los Angeles Superior Court and is a felony, and add the word-frequency matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For testing this part, we want to have a felony case. B2500042 is a felony which originated in the Los Angeles superior court. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(currentdir, 'text_opinions', 'B250042.txt'), \"rb\") as file:\n",
    "    felony_case = pickle.load(file)\n",
    "    \n",
    "print(type(felony_case))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're looking for the presence of a string of the form \"Super. Ct. No. NA088447\" - specifically , the lower court case number should be [letter] A [6-digit number]. I am not sure whether all cases use 6 digits following the A. We're also looking for the words \"Los Angeles County\" - and we want them to be in the first page of the document. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon further consideration, it may be worthwhile to process all the opinions and look for the case number afterward. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try setting up a class Case with properties \"corpus\", \"wordcount\", \"casenum\"\"\n",
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "class Case:\n",
    "    def __init__(self, textfile):\n",
    "        assert os.path.isfile(textfile) == True\n",
    "        # We start out with just the raw text string from the pdf\n",
    "        # We should also initialize the casenumber attribute?\n",
    "        with open(textfile, \"rb\") as file: \n",
    "            self.text = pickle.load(file)\n",
    "            self.casenumber = os.path.splitext(os.path.split(textfile)[1])[0]\n",
    "    # We should have a method for doing text cleanup and a method for \n",
    "    # count vectorization\n",
    "    def make_corpus(self):\n",
    "        '''Adds corpus attribute to the Case object by cleaning\n",
    "        the raw text string'''\n",
    "        # Make text lowercase: \n",
    "        text = self.text\n",
    "        text = text.lower()\n",
    "        # Strip all newlines: \n",
    "        text = re.sub('\\n', '', text)\n",
    "        self.corpus = text\n",
    "    def make_unigram_matrix(self):\n",
    "        '''Adds frequency-matrix-of-unigrams attribute to Case object. Requires\n",
    "        that corpus attribute already exists.'''\n",
    "        # currently redundant with ngram matrix\n",
    "        cv = CountVectorizer(input='content',stop_words = 'english')\n",
    "        freq_cv = cv.fit_transform([self.corpus])\n",
    "        freq_matrix = pd.DataFrame(freq_cv.toarray(), columns = cv.get_feature_names())\n",
    "        # Sort the words by frequency, most frequent first\n",
    "        # freq_matrix = freq_matrix.sort_values(by=0,axis=1, ascending=False)\n",
    "        # freq_matrix = freq_matrix.transpose()\n",
    "        self.unigram_matrix = freq_matrix\n",
    "    def make_ngram_matrix(self, n):\n",
    "        '''Adds frequency-matrix-of-ngrams attribute to Case object. Requires\n",
    "        that corpus attribute already exists.'''\n",
    "        cv = CountVectorizer(input='content',stop_words = 'english',strip_accents='unicode',ngram_range=(1,n))\n",
    "        freq_cv = cv.fit_transform([self.corpus])\n",
    "        freq_matrix = pd.DataFrame(freq_cv.toarray(), columns = cv.get_feature_names())\n",
    "        # Sort the words by frequency, most frequent first\n",
    "        # freq_matrix = freq_matrix.sort_values(by=0,axis=1, ascending=False)\n",
    "        # freq_matrix = freq_matrix.transpose()\n",
    "        # note: transposing matrix makes the to_string method produce\n",
    "        # a nice output, but makes indexing by ngram slightly harder?\n",
    "        self.ngram_matrix = freq_matrix\n",
    "\n",
    "  \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "B300885\n",
      "   10  11  1115  11278  11280  1128111284  1128411285  1170  1192  12  ...  \\\n",
      "0   2   1     3      1      1           1           1     1     1   3  ...   \n",
      "\n",
      "   witnesses  wl  woodell  working  writ  writs  xavier  year  years  yun  \n",
      "0          1   1        2        1     4      1       1     3      5    1  \n",
      "\n",
      "[1 rows x 709 columns]\n",
      "   10  10 348  10 348 352353  10 348 352353 gallardo  \\\n",
      "0   2       1              1                       1   \n",
      "\n",
      "   10 348 352353 gallardo intended  10 years  10 years prior  \\\n",
      "0                                1         1               1   \n",
      "\n",
      "   10 years prior felony  10 years prior felony enhancements  11  ...  \\\n",
      "0                      1                                   1   1  ...   \n",
      "\n",
      "   years life robbery 25 years  years prior  years prior felony  \\\n",
      "0                            1            1                   1   \n",
      "\n",
      "   years prior felony enhancements  years prior felony enhancements 667  yun  \\\n",
      "0                                1                                    1    1   \n",
      "\n",
      "   yun lee  yun lee deputy  yun lee deputy attorneys  \\\n",
      "0        1               1                         1   \n",
      "\n",
      "   yun lee deputy attorneys general  \n",
      "0                                 1  \n",
      "\n",
      "[1 rows x 8279 columns]\n"
     ]
    }
   ],
   "source": [
    "# testing initialization method for class\n",
    "b = Case(testpath)\n",
    "print(b.casenumber)\n",
    "# print(b.text)\n",
    "b.make_corpus()\n",
    "# print(b.corpus)\n",
    "b.make_unigram_matrix()\n",
    "b.make_ngram_matrix(5)\n",
    "print(b.unigram_matrix)\n",
    "print(b.ngram_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('los angeles county super' in b.ngram_matrix.to_string())\n",
    "print('los angeles county super' in b.ngram_matrix)\n",
    "print('los angeles county super' in b.corpus)\n",
    "# The reason that these give different results is that the corpus\n",
    "# includes whitespace between \"county\" and \"super\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100 cases processed successfully. 0 cases flagged.\n",
      "200 cases processed successfully. 0 cases flagged.\n",
      "300 cases processed successfully. 0 cases flagged.\n",
      "400 cases processed successfully. 0 cases flagged.\n",
      "500 cases processed successfully. 0 cases flagged.\n",
      "600 cases processed successfully. 0 cases flagged.\n",
      "700 cases processed successfully. 0 cases flagged.\n",
      "800 cases processed successfully. 0 cases flagged.\n",
      "900 cases processed successfully. 0 cases flagged.\n",
      "1000 cases processed successfully. 0 cases flagged.\n",
      "1100 cases processed successfully. 0 cases flagged.\n",
      "1200 cases processed successfully. 0 cases flagged.\n",
      "1300 cases processed successfully. 0 cases flagged.\n",
      "1400 cases processed successfully. 0 cases flagged.\n",
      "1500 cases processed successfully. 0 cases flagged.\n",
      "1600 cases processed successfully. 0 cases flagged.\n",
      "1700 cases processed successfully. 0 cases flagged.\n",
      "1800 cases processed successfully. 0 cases flagged.\n",
      "1900 cases processed successfully. 0 cases flagged.\n",
      "2000 cases processed successfully. 0 cases flagged.\n",
      "2100 cases processed successfully. 0 cases flagged.\n",
      "2200 cases processed successfully. 0 cases flagged.\n",
      "2300 cases processed successfully. 0 cases flagged.\n",
      "2400 cases processed successfully. 0 cases flagged.\n",
      "2500 cases processed successfully. 0 cases flagged.\n",
      "2600 cases processed successfully. 0 cases flagged.\n",
      "2700 cases processed successfully. 0 cases flagged.\n",
      "2800 cases processed successfully. 0 cases flagged.\n",
      "2900 cases processed successfully. 0 cases flagged.\n",
      "3000 cases processed successfully. 0 cases flagged.\n",
      "3100 cases processed successfully. 0 cases flagged.\n",
      "3200 cases processed successfully. 0 cases flagged.\n",
      "3300 cases processed successfully. 0 cases flagged.\n",
      "3400 cases processed successfully. 0 cases flagged.\n",
      "3500 cases processed successfully. 0 cases flagged.\n",
      "3600 cases processed successfully. 0 cases flagged.\n",
      "3700 cases processed successfully. 0 cases flagged.\n",
      "3800 cases processed successfully. 0 cases flagged.\n",
      "3900 cases processed successfully. 0 cases flagged.\n",
      "4000 cases processed successfully. 0 cases flagged.\n",
      "4100 cases processed successfully. 0 cases flagged.\n",
      "4200 cases processed successfully. 0 cases flagged.\n",
      "4300 cases processed successfully. 0 cases flagged.\n",
      "4400 cases processed successfully. 0 cases flagged.\n",
      "4500 cases processed successfully. 0 cases flagged.\n",
      "4600 cases processed successfully. 0 cases flagged.\n",
      "4700 cases processed successfully. 0 cases flagged.\n",
      "4800 cases processed successfully. 0 cases flagged.\n",
      "4900 cases processed successfully. 0 cases flagged.\n",
      "5000 cases processed successfully. 0 cases flagged.\n",
      "5100 cases processed successfully. 0 cases flagged.\n",
      "5200 cases processed successfully. 0 cases flagged.\n",
      "5300 cases processed successfully. 0 cases flagged.\n",
      "5400 cases processed successfully. 0 cases flagged.\n",
      "5500 cases processed successfully. 0 cases flagged.\n",
      "5600 cases processed successfully. 0 cases flagged.\n",
      "5700 cases processed successfully. 0 cases flagged.\n",
      "5800 cases processed successfully. 0 cases flagged.\n",
      "5900 cases processed successfully. 0 cases flagged.\n",
      "6000 cases processed successfully. 0 cases flagged.\n",
      "6100 cases processed successfully. 0 cases flagged.\n",
      "6200 cases processed successfully. 0 cases flagged.\n",
      "6300 cases processed successfully. 0 cases flagged.\n",
      "6400 cases processed successfully. 0 cases flagged.\n",
      "6500 cases processed successfully. 0 cases flagged.\n",
      "6600 cases processed successfully. 0 cases flagged.\n",
      "6700 cases processed successfully. 0 cases flagged.\n",
      "6800 cases processed successfully. 0 cases flagged.\n",
      "6900 cases processed successfully. 0 cases flagged.\n",
      "7000 cases processed successfully. 0 cases flagged.\n",
      "7100 cases processed successfully. 0 cases flagged.\n",
      "7200 cases processed successfully. 0 cases flagged.\n",
      "7300 cases processed successfully. 0 cases flagged.\n",
      "7400 cases processed successfully. 0 cases flagged.\n",
      "7500 cases processed successfully. 0 cases flagged.\n",
      "7600 cases processed successfully. 0 cases flagged.\n",
      "7700 cases processed successfully. 0 cases flagged.\n",
      "7800 cases processed successfully. 0 cases flagged.\n",
      "7900 cases processed successfully. 0 cases flagged.\n",
      "8000 cases processed successfully. 0 cases flagged.\n",
      "8100 cases processed successfully. 0 cases flagged.\n",
      "8200 cases processed successfully. 0 cases flagged.\n",
      "8300 cases processed successfully. 0 cases flagged.\n",
      "8400 cases processed successfully. 0 cases flagged.\n",
      "8500 cases processed successfully. 0 cases flagged.\n",
      "8600 cases processed successfully. 0 cases flagged.\n",
      "8700 cases processed successfully. 0 cases flagged.\n",
      "8800 cases processed successfully. 0 cases flagged.\n",
      "8900 cases processed successfully. 0 cases flagged.\n",
      "9000 cases processed successfully. 0 cases flagged.\n",
      "9100 cases processed successfully. 0 cases flagged.\n",
      "9200 cases processed successfully. 0 cases flagged.\n",
      "9300 cases processed successfully. 0 cases flagged.\n",
      "9400 cases processed successfully. 0 cases flagged.\n",
      "9500 cases processed successfully. 0 cases flagged.\n",
      "9600 cases processed successfully. 0 cases flagged.\n",
      "9700 cases processed successfully. 0 cases flagged.\n",
      "9800 cases processed successfully. 0 cases flagged.\n",
      "9900 cases processed successfully. 0 cases flagged.\n",
      "10000 cases processed successfully. 0 cases flagged.\n",
      "10100 cases processed successfully. 0 cases flagged.\n",
      "10200 cases processed successfully. 0 cases flagged.\n",
      "10300 cases processed successfully. 0 cases flagged.\n",
      "10400 cases processed successfully. 0 cases flagged.\n",
      "10500 cases processed successfully. 0 cases flagged.\n",
      "10600 cases processed successfully. 0 cases flagged.\n",
      "10700 cases processed successfully. 0 cases flagged.\n",
      "10800 cases processed successfully. 0 cases flagged.\n",
      "10900 cases processed successfully. 0 cases flagged.\n",
      "11000 cases processed successfully. 0 cases flagged.\n",
      "11100 cases processed successfully. 0 cases flagged.\n",
      "11200 cases processed successfully. 0 cases flagged.\n",
      "11300 cases processed successfully. 0 cases flagged.\n",
      "11400 cases processed successfully. 0 cases flagged.\n",
      "11500 cases processed successfully. 0 cases flagged.\n",
      "11600 cases processed successfully. 0 cases flagged.\n",
      "11700 cases processed successfully. 0 cases flagged.\n",
      "11800 cases processed successfully. 0 cases flagged.\n",
      "11900 cases processed successfully. 0 cases flagged.\n",
      "12000 cases processed successfully. 0 cases flagged.\n",
      "12100 cases processed successfully. 0 cases flagged.\n",
      "12200 cases processed successfully. 0 cases flagged.\n",
      "12300 cases processed successfully. 0 cases flagged.\n",
      "12400 cases processed successfully. 0 cases flagged.\n",
      "12500 cases processed successfully. 0 cases flagged.\n",
      "12600 cases processed successfully. 0 cases flagged.\n",
      "12700 cases processed successfully. 0 cases flagged.\n",
      "12800 cases processed successfully. 0 cases flagged.\n",
      "12900 cases processed successfully. 0 cases flagged.\n",
      "13000 cases processed successfully. 0 cases flagged.\n",
      "13100 cases processed successfully. 0 cases flagged.\n",
      "13200 cases processed successfully. 0 cases flagged.\n",
      "13300 cases processed successfully. 0 cases flagged.\n",
      "13400 cases processed successfully. 0 cases flagged.\n",
      "13500 cases processed successfully. 0 cases flagged.\n",
      "13600 cases processed successfully. 0 cases flagged.\n",
      "13700 cases processed successfully. 0 cases flagged.\n",
      "13800 cases processed successfully. 0 cases flagged.\n",
      "13900 cases processed successfully. 0 cases flagged.\n",
      "14000 cases processed successfully. 0 cases flagged.\n",
      "14100 cases processed successfully. 0 cases flagged.\n",
      "14200 cases processed successfully. 0 cases flagged.\n",
      "14300 cases processed successfully. 0 cases flagged.\n",
      "14400 cases processed successfully. 0 cases flagged.\n",
      "14500 cases processed successfully. 0 cases flagged.\n",
      "14600 cases processed successfully. 0 cases flagged.\n",
      "14700 cases processed successfully. 0 cases flagged.\n",
      "14800 cases processed successfully. 0 cases flagged.\n",
      "14900 cases processed successfully. 0 cases flagged.\n",
      "15000 cases processed successfully. 0 cases flagged.\n",
      "15100 cases processed successfully. 0 cases flagged.\n",
      "15200 cases processed successfully. 0 cases flagged.\n",
      "15300 cases processed successfully. 0 cases flagged.\n",
      "15400 cases processed successfully. 0 cases flagged.\n",
      "15500 cases processed successfully. 0 cases flagged.\n",
      "15600 cases processed successfully. 0 cases flagged.\n",
      "15700 cases processed successfully. 0 cases flagged.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15800 cases processed successfully. 0 cases flagged.\n",
      "15900 cases processed successfully. 0 cases flagged.\n",
      "16000 cases processed successfully. 0 cases flagged.\n",
      "16100 cases processed successfully. 0 cases flagged.\n",
      "16200 cases processed successfully. 0 cases flagged.\n",
      "16300 cases processed successfully. 0 cases flagged.\n",
      "16400 cases processed successfully. 0 cases flagged.\n",
      "16500 cases processed successfully. 0 cases flagged.\n",
      "16600 cases processed successfully. 0 cases flagged.\n",
      "16700 cases processed successfully. 0 cases flagged.\n",
      "16800 cases processed successfully. 0 cases flagged.\n",
      "16900 cases processed successfully. 0 cases flagged.\n",
      "17000 cases processed successfully. 0 cases flagged.\n",
      "17100 cases processed successfully. 0 cases flagged.\n",
      "17200 cases processed successfully. 0 cases flagged.\n",
      "17300 cases processed successfully. 0 cases flagged.\n",
      "17400 cases processed successfully. 0 cases flagged.\n",
      "17500 cases processed successfully. 0 cases flagged.\n",
      "17600 cases processed successfully. 0 cases flagged.\n",
      "17700 cases processed successfully. 0 cases flagged.\n",
      "17800 cases processed successfully. 0 cases flagged.\n",
      "17900 cases processed successfully. 0 cases flagged.\n",
      "18000 cases processed successfully. 0 cases flagged.\n",
      "18100 cases processed successfully. 0 cases flagged.\n",
      "18200 cases processed successfully. 0 cases flagged.\n",
      "18300 cases processed successfully. 0 cases flagged.\n",
      "18400 cases processed successfully. 0 cases flagged.\n",
      "18500 cases processed successfully. 0 cases flagged.\n",
      "18600 cases processed successfully. 0 cases flagged.\n",
      "18700 cases processed successfully. 0 cases flagged.\n",
      "18800 cases processed successfully. 0 cases flagged.\n",
      "18900 cases processed successfully. 0 cases flagged.\n",
      "19000 cases processed successfully. 0 cases flagged.\n",
      "19100 cases processed successfully. 0 cases flagged.\n",
      "19200 cases processed successfully. 0 cases flagged.\n",
      "19300 cases processed successfully. 0 cases flagged.\n",
      "19400 cases processed successfully. 0 cases flagged.\n",
      "19500 cases processed successfully. 0 cases flagged.\n",
      "19600 cases processed successfully. 0 cases flagged.\n",
      "19700 cases processed successfully. 0 cases flagged.\n",
      "19800 cases processed successfully. 0 cases flagged.\n",
      "19900 cases processed successfully. 0 cases flagged.\n",
      "20000 cases processed successfully. 0 cases flagged.\n",
      "20100 cases processed successfully. 0 cases flagged.\n",
      "20200 cases processed successfully. 0 cases flagged.\n",
      "20300 cases processed successfully. 0 cases flagged.\n",
      "20400 cases processed successfully. 0 cases flagged.\n",
      "20500 cases processed successfully. 0 cases flagged.\n",
      "20600 cases processed successfully. 0 cases flagged.\n",
      "20700 cases processed successfully. 0 cases flagged.\n",
      "20800 cases processed successfully. 0 cases flagged.\n",
      "20900 cases processed successfully. 0 cases flagged.\n",
      "21000 cases processed successfully. 0 cases flagged.\n",
      "21100 cases processed successfully. 0 cases flagged.\n",
      "21200 cases processed successfully. 0 cases flagged.\n",
      "21300 cases processed successfully. 0 cases flagged.\n",
      "21400 cases processed successfully. 0 cases flagged.\n",
      "21500 cases processed successfully. 0 cases flagged.\n",
      "21600 cases processed successfully. 0 cases flagged.\n",
      "21700 cases processed successfully. 0 cases flagged.\n",
      "21800 cases processed successfully. 0 cases flagged.\n",
      "21900 cases processed successfully. 0 cases flagged.\n",
      "22000 cases processed successfully. 0 cases flagged.\n",
      "22100 cases processed successfully. 0 cases flagged.\n",
      "22200 cases processed successfully. 0 cases flagged.\n",
      "22300 cases processed successfully. 0 cases flagged.\n",
      "22400 cases processed successfully. 0 cases flagged.\n",
      "Flagged case at C:\\Users\\leodb\\Documents\\THESIS\\ldbw\\textopinions\\B294547.txt\n",
      "22500 cases processed successfully. 1 cases flagged.\n",
      "22600 cases processed successfully. 1 cases flagged.\n",
      "22700 cases processed successfully. 1 cases flagged.\n",
      "22800 cases processed successfully. 1 cases flagged.\n"
     ]
    }
   ],
   "source": [
    "# go thru the text opinions \n",
    "# make list of files \n",
    "text_file_list = []\n",
    "for root, dirs, files in os.walk(os.path.join(os.path.abspath(''),'textopinions'), topdown=False):\n",
    "    for file in files:\n",
    "        text_file_list.append(os.path.join(root, file))\n",
    "\n",
    "processed_cases = []\n",
    "flagged_cases = []\n",
    "processed_count = 0\n",
    "flagged_count = 0\n",
    "for filepath in text_file_list:\n",
    "    try: \n",
    "        case = Case(filepath)\n",
    "        case.make_corpus()\n",
    "        case.make_ngram_matrix(5)\n",
    "        processed_cases.append(case)\n",
    "        processed_count += 1\n",
    "        if processed_count % 100 == 0:\n",
    "            print('%d cases processed successfully. %d cases flagged.' % (processed_count, flagged_count))\n",
    "    except (KeyboardInterrupt,SystemExit): \n",
    "        raise\n",
    "    except:\n",
    "        flagged_cases.append(filepath)\n",
    "        flagged_count += 1\n",
    "        print('Flagged case at %s' % filepath)\n",
    "        continue\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pickle output for later use\n",
    "with open('processed_cases.txt', \"wb\") as file:\n",
    "    pickle.dump(processed_cases, file)\n",
    "with open('flagged_cases.txt', 'wb') as file: \n",
    "    pickle.dump(flagged_cases, file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}