Deep Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Audiobooks business case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocess the data. Balance the dataset. Standardize the data. Create 3 datasets: training, validation, and test. Save the newly created sets in a tensor friendly format (e.g. *.npz)\n",
    "\n",
    "Since we are dealing with real life data, we will need to preprocess it a bit.\n",
    "\n",
    "If you want to know how to do that, go through the code with comments. In any case, this should do the trick for most datasets organized in the way: many inputs, and then 1 cell containing the targets (supersized learning datasets). Keep in mind that a specific problem may require additional preprocessing.\n",
    "\n",
    "Note that we have removed the header row, which contains the names of the categories. We simply need the numerical data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract the data from the csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# We will use the StandardScaler module, so we can later deploy the model\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "import pickle\n",
    "\n",
    "# Load the data\n",
    "raw_csv_data = np.loadtxt('Audiobooks_data.csv',delimiter=',')\n",
    "\n",
    "# The inputs are all columns in the csv, except for the first one and the last one\n",
    "# The first column is the arbitrary ID, while the last contains the targets\n",
    "\n",
    "unscaled_inputs_all = raw_csv_data[:,1:-1]\n",
    "\n",
    "# The targets are in the last column. That's how datasets are conventionally organized.\n",
    "targets_all = raw_csv_data[:,-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Balance the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There are different Python packages that could be used for balancing\n",
    "# Here we approach the problem manually, so you can observe the inner workings of the balancing process\n",
    "\n",
    "# Count how many targets are 1 (meaning that the customer did convert)\n",
    "num_one_targets = int(np.sum(targets_all))\n",
    "\n",
    "# Set a counter for targets that are 0 (meaning that the customer did not convert)\n",
    "zero_targets_counter = 0\n",
    "\n",
    "# We want to create a \"balanced\" dataset, so we will have to remove some input/target pairs.\n",
    "# Declare a variable that will do that:\n",
    "indices_to_remove = []\n",
    "\n",
    "# Count the number of targets that are 0. \n",
    "# Once there are as many 0s as 1s, mark entries where the target is 0.\n",
    "for i in range(targets_all.shape[0]):\n",
    "    if targets_all[i] == 0:\n",
    "        zero_targets_counter += 1\n",
    "        if zero_targets_counter > num_one_targets:\n",
    "            indices_to_remove.append(i)\n",
    "\n",
    "# Create two new variables, one that will contain the inputs, and one that will contain the targets.\n",
    "# We delete all indices that we marked \"to remove\" in the loop above.\n",
    "unscaled_inputs_equal_priors = np.delete(unscaled_inputs_all, indices_to_remove, axis=0)\n",
    "targets_equal_priors = np.delete(targets_all, indices_to_remove, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standardize the inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Crete a standar scaler object\n",
    "scaler_deep_learning = StandardScaler()\n",
    "# Fit and transform the original data\n",
    "# Essentially, we calculate and STORE the mean and variance of the data in the scaler object\n",
    "# At the same time we standrdize the data using this information\n",
    "# Note that the mean and variance remain recorded in the scaler object\n",
    "scaled_inputs = scaler_deep_learning.fit_transform(unscaled_inputs_equal_priors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shuffle the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# When the data was collected it was actually arranged by date\n",
    "# Shuffle the indices of the data, so the data is not arranged in any way when we feed it.\n",
    "# Since we will be batching, we want the data to be as randomly spread out as possible\n",
    "shuffled_indices = np.arange(scaled_inputs.shape[0])\n",
    "np.random.shuffle(shuffled_indices)\n",
    "\n",
    "# Use the shuffled indices to shuffle the inputs and targets.\n",
    "shuffled_inputs = scaled_inputs[shuffled_indices]\n",
    "shuffled_targets = targets_equal_priors[shuffled_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the dataset into train, validation, and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1784.0 3579 0.4984632578932663\n",
      "216.0 447 0.48322147651006714\n",
      "237.0 448 0.5290178571428571\n"
     ]
    }
   ],
   "source": [
    "# Count the total number of samples\n",
    "samples_count = shuffled_inputs.shape[0]\n",
    "\n",
    "# Count the samples in each subset, assuming we want 80-10-10 distribution of training, validation, and test.\n",
    "# Naturally, the numbers are integers.\n",
    "train_samples_count = int(0.8 * samples_count)\n",
    "validation_samples_count = int(0.1 * samples_count)\n",
    "\n",
    "# The 'test' dataset contains all remaining data.\n",
    "test_samples_count = samples_count - train_samples_count - validation_samples_count\n",
    "\n",
    "# Create variables that record the inputs and targets for training\n",
    "# In our shuffled dataset, they are the first \"train_samples_count\" observations\n",
    "train_inputs = shuffled_inputs[:train_samples_count]\n",
    "train_targets = shuffled_targets[:train_samples_count]\n",
    "\n",
    "# Create variables that record the inputs and targets for validation.\n",
    "# They are the next \"validation_samples_count\" observations, folllowing the \"train_samples_count\" we already assigned\n",
    "validation_inputs = shuffled_inputs[train_samples_count:train_samples_count+validation_samples_count]\n",
    "validation_targets = shuffled_targets[train_samples_count:train_samples_count+validation_samples_count]\n",
    "\n",
    "# Create variables that record the inputs and targets for test.\n",
    "# They are everything that is remaining.\n",
    "test_inputs = shuffled_inputs[train_samples_count+validation_samples_count:]\n",
    "test_targets = shuffled_targets[train_samples_count+validation_samples_count:]\n",
    "\n",
    "# We balanced our dataset to be 50-50 (for targets 0 and 1), but the training, validation, and test were \n",
    "# taken from a shuffled dataset. Check if they are balanced, too. Note that each time you rerun this code, \n",
    "# you will get different values, as each time they are shuffled randomly.\n",
    "# Normally you preprocess ONCE, so you need not rerun this code once it is done.\n",
    "# If you rerun this whole sheet, the npzs will be overwritten with your newly preprocessed data.\n",
    "\n",
    "# Print the number of targets that are 1s, the total number of samples, and the proportion for training, validation, and test.\n",
    "print(np.sum(train_targets), train_samples_count, np.sum(train_targets) / train_samples_count)\n",
    "print(np.sum(validation_targets), validation_samples_count, np.sum(validation_targets) / validation_samples_count)\n",
    "print(np.sum(test_targets), test_samples_count, np.sum(test_targets) / test_samples_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the three datasets in *.npz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the three datasets in *.npz.\n",
    "# In the next lesson, you will see that it is extremely valuable to name them in such a coherent way!\n",
    "\n",
    "np.savez('Audiobooks_data_train', inputs=train_inputs, targets=train_targets)\n",
    "np.savez('Audiobooks_data_validation', inputs=validation_inputs, targets=validation_targets)\n",
    "np.savez('Audiobooks_data_test', inputs=test_inputs, targets=test_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the scaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Similar to how we have saved the scaler files before, we also save this scaler, so we can apply in on new data\n",
    "pickle.dump(scaler_deep_learning, open('scaler_deep_learning.pickle', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical example. Audiobooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem\n",
    "\n",
    "You are given data from an Audiobook app. Logically, it relates only to the audio versions of books. Each customer in the database has made a purchase at least once, that's why he/she is in the database. We want to create a machine learning algorithm based on our available data that can predict if a customer will buy again from the Audiobook company.\n",
    "\n",
    "The main idea is that if a customer has a low probability of coming back, there is no reason to spend any money on advertizing to him/her. If we can focus our efforts ONLY on customers that are likely to convert again, we can make great savings. Moreover, this model can identify the most important metrics for a customer to come back again. Identifying new customers creates value and growth opportunities.\n",
    "\n",
    "You have a .csv summarizing the data. There are several variables: Customer ID, Book length in mins_avg (average of all purchases), Book length in minutes_sum (sum of all purchases), Price Paid_avg (average of all purchases), Price paid_sum (sum of all purchases), Review (a Boolean variable), Review (out of 10), Total minutes listened, Completion (from 0 to 1), Support requests (number), and Last visited minus purchase date (in days).\n",
    "\n",
    "So these are the inputs (excluding customer ID, as it is completely arbitrary. It's more like a name, than a number).\n",
    "\n",
    "The targets are a Boolean variable (so 0, or 1). We are taking a period of 2 years in our inputs, and the next 6 months as targets. So, in fact, we are predicting if: based on the last 2 years of activity and engagement, a customer will convert in the next 6 months. 6 months sounds like a reasonable time. If they don't convert after 6 months, chances are they've gone to a competitor or didn't like the Audiobook way of digesting information. \n",
    "\n",
    "The task is simple: create a machine learning algorithm, which is able to predict if a customer will buy again. \n",
    "\n",
    "This is a classification problem with two classes: won't buy and will buy, represented by 0s and 1s. \n",
    "\n",
    "Good luck!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the machine learning algorithm\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the relevant libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we must import the libraries once again since we haven't imported them in this file\n",
    "import numpy as np\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's create a temporary variable npz, where we will store each of the three Audiobooks datasets\n",
    "npz = np.load('Audiobooks_data_train.npz')\n",
    "\n",
    "# we extract the inputs using the keyword under which we saved them\n",
    "# to ensure that they are all floats, let's also take care of that\n",
    "train_inputs = npz['inputs'].astype(np.float)\n",
    "# targets must be int because of sparse_categorical_crossentropy (we want to be able to smoothly one-hot encode them)\n",
    "train_targets = npz['targets'].astype(np.int)\n",
    "\n",
    "# we load the validation data in the temporary variable\n",
    "npz = np.load('Audiobooks_data_validation.npz')\n",
    "# we can load the inputs and the targets in the same line\n",
    "validation_inputs, validation_targets = npz['inputs'].astype(np.float), npz['targets'].astype(np.int)\n",
    "\n",
    "# we load the test data in the temporary variable\n",
    "npz = np.load('Audiobooks_data_test.npz')\n",
    "# we create 2 variables that will contain the test inputs and the test targets\n",
    "test_inputs, test_targets = npz['inputs'].astype(np.float), npz['targets'].astype(np.int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model\n",
    "Outline, optimizers, loss, early stopping and training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 3579 samples, validate on 447 samples\n",
      "Epoch 1/100\n",
      "3579/3579 - 1s - loss: 0.5996 - accuracy: 0.6921 - val_loss: 0.4299 - val_accuracy: 0.8725\n",
      "Epoch 2/100\n",
      "3579/3579 - 0s - loss: 0.3799 - accuracy: 0.8737 - val_loss: 0.3138 - val_accuracy: 0.8837\n",
      "Epoch 3/100\n",
      "3579/3579 - 0s - loss: 0.3205 - accuracy: 0.8846 - val_loss: 0.2926 - val_accuracy: 0.8881\n",
      "Epoch 4/100\n",
      "3579/3579 - 0s - loss: 0.2967 - accuracy: 0.8916 - val_loss: 0.2876 - val_accuracy: 0.8881\n",
      "Epoch 5/100\n",
      "3579/3579 - 0s - loss: 0.2828 - accuracy: 0.8969 - val_loss: 0.2776 - val_accuracy: 0.8949\n",
      "Epoch 6/100\n",
      "3579/3579 - 0s - loss: 0.2727 - accuracy: 0.8997 - val_loss: 0.2745 - val_accuracy: 0.8881\n",
      "Epoch 7/100\n",
      "3579/3579 - 0s - loss: 0.2669 - accuracy: 0.9014 - val_loss: 0.2683 - val_accuracy: 0.8971\n",
      "Epoch 8/100\n",
      "3579/3579 - 0s - loss: 0.2605 - accuracy: 0.9042 - val_loss: 0.2659 - val_accuracy: 0.8993\n",
      "Epoch 9/100\n",
      "3579/3579 - 0s - loss: 0.2573 - accuracy: 0.9042 - val_loss: 0.2654 - val_accuracy: 0.8993\n",
      "Epoch 10/100\n",
      "3579/3579 - 0s - loss: 0.2531 - accuracy: 0.9028 - val_loss: 0.2654 - val_accuracy: 0.8971\n",
      "Epoch 11/100\n",
      "3579/3579 - 0s - loss: 0.2509 - accuracy: 0.9039 - val_loss: 0.2627 - val_accuracy: 0.8926\n",
      "Epoch 12/100\n",
      "3579/3579 - 0s - loss: 0.2479 - accuracy: 0.9075 - val_loss: 0.2638 - val_accuracy: 0.9060\n",
      "Epoch 13/100\n",
      "3579/3579 - 0s - loss: 0.2466 - accuracy: 0.9072 - val_loss: 0.2604 - val_accuracy: 0.8993\n",
      "Epoch 14/100\n",
      "3579/3579 - 0s - loss: 0.2463 - accuracy: 0.9078 - val_loss: 0.2609 - val_accuracy: 0.8926\n",
      "Epoch 15/100\n",
      "3579/3579 - 0s - loss: 0.2464 - accuracy: 0.9064 - val_loss: 0.2563 - val_accuracy: 0.8971\n",
      "Epoch 16/100\n",
      "3579/3579 - 0s - loss: 0.2411 - accuracy: 0.9084 - val_loss: 0.2700 - val_accuracy: 0.8971\n",
      "Epoch 17/100\n",
      "3579/3579 - 0s - loss: 0.2410 - accuracy: 0.9089 - val_loss: 0.2564 - val_accuracy: 0.9016\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x2069dd3fb08>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Optionally set the input size. We won't be using it, but in some cases it is beneficial\n",
    "# input_size = 10\n",
    "# Set the output size\n",
    "output_size = 2\n",
    "# Use same hidden layer size for both hidden layers. Not a necessity.\n",
    "hidden_layer_size = 50\n",
    "    \n",
    "# define how the model will look like\n",
    "model = tf.keras.Sequential([\n",
    "    # tf.keras.layers.Dense is basically implementing: output = activation(dot(input, weight) + bias)\n",
    "    # it takes several arguments, but the most important ones for us are the hidden_layer_size and the activation function\n",
    "    tf.keras.layers.Dense(hidden_layer_size, activation='relu'), # 1st hidden layer\n",
    "    tf.keras.layers.Dense(hidden_layer_size, activation='relu'), # 2nd hidden layer\n",
    "    # the final layer is no different, we just make sure to activate it with softmax\n",
    "    tf.keras.layers.Dense(output_size, activation='softmax') # output layer\n",
    "])\n",
    "\n",
    "\n",
    "### Choose the optimizer and the loss function\n",
    "\n",
    "# we define the optimizer we'd like to use, \n",
    "# the loss function, \n",
    "# and the metrics we are interested in obtaining at each iteration\n",
    "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "\n",
    "### Training\n",
    "# That's where we train the model we have built.\n",
    "\n",
    "# set the batch size\n",
    "batch_size = 100\n",
    "\n",
    "# set a maximum number of training epochs\n",
    "max_epochs = 100\n",
    "\n",
    "# set an early stopping mechanism\n",
    "# let's set patience=2, to be a bit tolerant against random validation loss increases\n",
    "early_stopping = tf.keras.callbacks.EarlyStopping(patience=2)\n",
    "\n",
    "# fit the model\n",
    "# note that this time the train, validation and test data are not iterable\n",
    "model.fit(train_inputs, # train inputs\n",
    "          train_targets, # train targets\n",
    "          batch_size=batch_size, # batch size\n",
    "          epochs=max_epochs, # epochs that we will train for (assuming early stopping doesn't kick in)\n",
    "          # callbacks are functions called by a task when a task is completed\n",
    "          # task here is to check if val_loss is increasing\n",
    "          callbacks=[early_stopping], # early stopping\n",
    "          validation_data=(validation_inputs, validation_targets), # validation data\n",
    "          verbose = 2 # making sure we get enough information about the training process\n",
    "          )  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model\n",
    "\n",
    "As we discussed in the lectures, after training on the training data and validating on the validation data, we test the final prediction power of our model by running it on the test dataset that the algorithm has NEVER seen before.\n",
    "\n",
    "It is very important to realize that fiddling with the hyperparameters overfits the validation dataset. \n",
    "\n",
    "The test is the absolute final instance. You should not test before you are completely done with adjusting your model.\n",
    "\n",
    "If you adjust your model after testing, you will start overfitting the test dataset, which will defeat its purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "448/1 [================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 0s 432us/sample - loss: 0.2730 - accuracy: 0.8996\n"
     ]
    }
   ],
   "source": [
    "# declare two variables that are going to contain the two outputs from the evaluate function\n",
    "# they are the loss (which is there by default) and whatever was specified in the 'metrics' argument when fitting the model\n",
    "test_loss, test_accuracy = model.evaluate(test_inputs, test_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test loss: 0.25. Test accuracy: 89.96%\n"
     ]
    }
   ],
   "source": [
    "# Print the result in neatly formatted\n",
    "print('\\nTest loss: {0:.2f}. Test accuracy: {1:.2f}%'.format(test_loss, test_accuracy*100.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtain the probability for a customer to convert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[9.30453837e-01, 6.95461929e-02],\n",
       "       [9.99998569e-01, 1.41493751e-06],\n",
       "       [1.35837094e-04, 9.99864101e-01],\n",
       "       [1.16327487e-01, 8.83672476e-01],\n",
       "       [1.00000000e+00, 7.11777443e-11],\n",
       "       [1.07386224e-01, 8.92613769e-01],\n",
       "       [1.80177465e-02, 9.81982291e-01],\n",
       "       [8.69632065e-01, 1.30367935e-01],\n",
       "       [9.99988079e-01, 1.19029910e-05],\n",
       "       [9.99502897e-01, 4.97044064e-04],\n",
       "       [9.63217914e-02, 9.03678238e-01],\n",
       "       [1.04484417e-01, 8.95515621e-01],\n",
       "       [2.85309970e-01, 7.14689970e-01],\n",
       "       [8.80754173e-01, 1.19245805e-01],\n",
       "       [7.64859200e-01, 2.35140830e-01],\n",
       "       [8.41083407e-01, 1.58916652e-01],\n",
       "       [4.19977248e-01, 5.80022752e-01],\n",
       "       [7.40365982e-01, 2.59634018e-01],\n",
       "       [3.56689602e-01, 6.43310428e-01],\n",
       "       [1.04733994e-02, 9.89526629e-01],\n",
       "       [2.61641800e-01, 7.38358200e-01],\n",
       "       [1.66383058e-01, 8.33616972e-01],\n",
       "       [1.21992292e-04, 9.99878049e-01],\n",
       "       [9.25583839e-01, 7.44160935e-02],\n",
       "       [3.61022174e-01, 6.38977826e-01],\n",
       "       [9.83680725e-01, 1.63192172e-02],\n",
       "       [1.10430792e-01, 8.89569223e-01],\n",
       "       [2.38521963e-01, 7.61478007e-01],\n",
       "       [1.69038370e-01, 8.30961645e-01],\n",
       "       [9.99989748e-01, 1.03071916e-05],\n",
       "       [1.76587150e-01, 8.23412895e-01],\n",
       "       [9.57734138e-02, 9.04226661e-01],\n",
       "       [1.07815815e-03, 9.98921871e-01],\n",
       "       [3.10346877e-08, 1.00000000e+00],\n",
       "       [1.56920741e-03, 9.98430789e-01],\n",
       "       [9.32997227e-01, 6.70027509e-02],\n",
       "       [8.57020319e-01, 1.42979741e-01],\n",
       "       [1.39835253e-01, 8.60164762e-01],\n",
       "       [9.87669766e-01, 1.23302797e-02],\n",
       "       [8.06992948e-01, 1.93007082e-01],\n",
       "       [5.30556321e-01, 4.69443709e-01],\n",
       "       [9.99981403e-01, 1.85642402e-05],\n",
       "       [8.34989130e-01, 1.65010825e-01],\n",
       "       [9.25707936e-01, 7.42920786e-02],\n",
       "       [1.23402819e-01, 8.76597226e-01],\n",
       "       [8.45300257e-01, 1.54699728e-01],\n",
       "       [1.00000000e+00, 1.01467315e-11],\n",
       "       [1.74305364e-01, 8.25694621e-01],\n",
       "       [7.78605044e-01, 2.21394971e-01],\n",
       "       [1.00000000e+00, 2.01175990e-13],\n",
       "       [9.99641061e-01, 3.58952500e-04],\n",
       "       [9.70711350e-01, 2.92886607e-02],\n",
       "       [9.99999523e-01, 4.42310068e-07],\n",
       "       [2.18931824e-08, 1.00000000e+00],\n",
       "       [9.53518033e-01, 4.64820229e-02],\n",
       "       [9.23178077e-01, 7.68218935e-02],\n",
       "       [8.24134886e-01, 1.75865114e-01],\n",
       "       [1.00000000e+00, 4.05194012e-09],\n",
       "       [3.15806051e-06, 9.99996901e-01],\n",
       "       [3.71254755e-05, 9.99962926e-01],\n",
       "       [8.80546331e-01, 1.19453713e-01],\n",
       "       [9.52197850e-01, 4.78021391e-02],\n",
       "       [8.92399922e-02, 9.10760045e-01],\n",
       "       [7.11297116e-06, 9.99992847e-01],\n",
       "       [9.13227677e-01, 8.67723003e-02],\n",
       "       [9.24932361e-01, 7.50676468e-02],\n",
       "       [8.63835692e-01, 1.36164352e-01],\n",
       "       [8.07916294e-05, 9.99919176e-01],\n",
       "       [1.74417123e-01, 8.25582922e-01],\n",
       "       [1.00000000e+00, 2.91046326e-10],\n",
       "       [9.28210974e-01, 7.17890039e-02],\n",
       "       [2.08586464e-08, 1.00000000e+00],\n",
       "       [5.58222225e-03, 9.94417787e-01],\n",
       "       [1.08370706e-01, 8.91629338e-01],\n",
       "       [9.67722535e-01, 3.22774760e-02],\n",
       "       [9.99007761e-01, 9.92234563e-04],\n",
       "       [9.74311411e-01, 2.56885141e-02],\n",
       "       [5.87328136e-01, 4.12671864e-01],\n",
       "       [1.39383391e-01, 8.60616624e-01],\n",
       "       [1.34803988e-02, 9.86519575e-01],\n",
       "       [2.47419113e-03, 9.97525752e-01],\n",
       "       [6.53663883e-05, 9.99934673e-01],\n",
       "       [9.75138366e-01, 2.48616133e-02],\n",
       "       [1.21541068e-01, 8.78458917e-01],\n",
       "       [8.47659767e-01, 1.52340159e-01],\n",
       "       [7.69552439e-02, 9.23044801e-01],\n",
       "       [3.80294841e-05, 9.99961972e-01],\n",
       "       [9.45563078e-01, 5.44369034e-02],\n",
       "       [8.41524243e-01, 1.58475801e-01],\n",
       "       [9.99966860e-01, 3.31942465e-05],\n",
       "       [7.54062176e-01, 2.45937839e-01],\n",
       "       [1.71284556e-01, 8.28715444e-01],\n",
       "       [5.59141440e-03, 9.94408548e-01],\n",
       "       [1.00000000e+00, 4.50613540e-11],\n",
       "       [5.65095767e-02, 9.43490386e-01],\n",
       "       [9.99805272e-01, 1.94792519e-04],\n",
       "       [4.03621167e-01, 5.96378803e-01],\n",
       "       [1.71284556e-01, 8.28715444e-01],\n",
       "       [9.09985840e-01, 9.00141969e-02],\n",
       "       [1.03702515e-01, 8.96297455e-01],\n",
       "       [8.55304673e-02, 9.14469540e-01],\n",
       "       [1.88061014e-01, 8.11939001e-01],\n",
       "       [9.37709332e-01, 6.22906350e-02],\n",
       "       [9.45392326e-02, 9.05460835e-01],\n",
       "       [2.09250837e-04, 9.99790728e-01],\n",
       "       [9.98526037e-01, 1.47393311e-03],\n",
       "       [8.37306559e-01, 1.62693426e-01],\n",
       "       [1.55403644e-01, 8.44596386e-01],\n",
       "       [2.81001814e-03, 9.97189939e-01],\n",
       "       [1.32406756e-01, 8.67593288e-01],\n",
       "       [2.92227238e-01, 7.07772732e-01],\n",
       "       [8.14971328e-01, 1.85028687e-01],\n",
       "       [5.27299009e-02, 9.47270095e-01],\n",
       "       [8.88838172e-01, 1.11161895e-01],\n",
       "       [1.00000000e+00, 1.39967034e-08],\n",
       "       [8.18988502e-01, 1.81011498e-01],\n",
       "       [5.76405945e-08, 1.00000000e+00],\n",
       "       [9.99999881e-01, 6.33984811e-08],\n",
       "       [8.25964749e-01, 1.74035311e-01],\n",
       "       [5.20697653e-01, 4.79302317e-01],\n",
       "       [4.50477179e-04, 9.99549448e-01],\n",
       "       [8.41059208e-01, 1.58940822e-01],\n",
       "       [6.93646260e-03, 9.93063509e-01],\n",
       "       [8.15947175e-01, 1.84052870e-01],\n",
       "       [1.74305350e-01, 8.25694740e-01],\n",
       "       [9.16352510e-01, 8.36474672e-02],\n",
       "       [7.69623294e-02, 9.23037708e-01],\n",
       "       [9.99978900e-01, 2.11012702e-05],\n",
       "       [9.99534965e-01, 4.65101126e-04],\n",
       "       [9.98224318e-01, 1.77568465e-03],\n",
       "       [9.99996543e-01, 3.49951915e-06],\n",
       "       [9.37745810e-01, 6.22541681e-02],\n",
       "       [2.81408072e-01, 7.18591869e-01],\n",
       "       [9.99644160e-01, 3.55840049e-04],\n",
       "       [3.16137969e-01, 6.83861971e-01],\n",
       "       [9.99986649e-01, 1.34073871e-05],\n",
       "       [9.98374820e-01, 1.62519433e-03],\n",
       "       [1.28528357e-01, 8.71471643e-01],\n",
       "       [3.46778147e-02, 9.65322137e-01],\n",
       "       [1.32834837e-01, 8.67165208e-01],\n",
       "       [8.68563771e-01, 1.31436199e-01],\n",
       "       [8.82367611e-01, 1.17632322e-01],\n",
       "       [1.76587179e-01, 8.23412895e-01],\n",
       "       [3.88413355e-06, 9.99996066e-01],\n",
       "       [8.00376713e-01, 1.99623257e-01],\n",
       "       [9.99998808e-01, 1.19318554e-06],\n",
       "       [1.71407670e-01, 8.28592360e-01],\n",
       "       [1.68157861e-01, 8.31842184e-01],\n",
       "       [9.99999762e-01, 2.38276684e-07],\n",
       "       [9.18719709e-01, 8.12802687e-02],\n",
       "       [9.99612510e-01, 3.87538748e-04],\n",
       "       [9.02393639e-01, 9.76062939e-02],\n",
       "       [9.29223448e-02, 9.07077670e-01],\n",
       "       [9.99702275e-01, 2.97709950e-04],\n",
       "       [9.99375880e-01, 6.24199281e-04],\n",
       "       [7.49404132e-01, 2.50595868e-01],\n",
       "       [5.17917840e-07, 9.99999523e-01],\n",
       "       [1.00000000e+00, 5.01060491e-08],\n",
       "       [8.88756858e-06, 9.99991059e-01],\n",
       "       [2.69970536e-01, 7.30029464e-01],\n",
       "       [5.56286541e-04, 9.99443710e-01],\n",
       "       [1.00000000e+00, 4.00299016e-10],\n",
       "       [1.00000000e+00, 3.00082768e-08],\n",
       "       [9.99999881e-01, 1.21968000e-07],\n",
       "       [6.63487494e-01, 3.36512536e-01],\n",
       "       [9.34316039e-01, 6.56840056e-02],\n",
       "       [9.36600626e-01, 6.33994341e-02],\n",
       "       [9.99891520e-01, 1.08505708e-04],\n",
       "       [1.26619965e-01, 8.73380065e-01],\n",
       "       [7.03380443e-03, 9.92966175e-01],\n",
       "       [2.86356717e-05, 9.99971390e-01],\n",
       "       [7.74748921e-01, 2.25251094e-01],\n",
       "       [6.14274584e-04, 9.99385715e-01],\n",
       "       [1.36532653e-05, 9.99986291e-01],\n",
       "       [9.13227677e-01, 8.67723003e-02],\n",
       "       [8.84272624e-03, 9.91157293e-01],\n",
       "       [9.55431581e-01, 4.45683785e-02],\n",
       "       [7.32982066e-04, 9.99267042e-01],\n",
       "       [1.69238493e-01, 8.30761492e-01],\n",
       "       [8.79989803e-01, 1.20010212e-01],\n",
       "       [9.99999762e-01, 2.11871068e-07],\n",
       "       [1.00000000e+00, 2.30217165e-11],\n",
       "       [9.18692529e-01, 8.13074932e-02],\n",
       "       [8.80977154e-01, 1.19022824e-01],\n",
       "       [1.00000000e+00, 1.14745373e-10],\n",
       "       [2.14514613e-01, 7.85485446e-01],\n",
       "       [7.56156921e-01, 2.43843079e-01],\n",
       "       [9.66590405e-01, 3.34095880e-02],\n",
       "       [6.47373497e-01, 3.52626443e-01],\n",
       "       [9.96395409e-01, 3.60457040e-03],\n",
       "       [9.14106011e-01, 8.58939737e-02],\n",
       "       [9.16808307e-01, 8.31916705e-02],\n",
       "       [9.38398719e-01, 6.16013035e-02],\n",
       "       [4.07205516e-04, 9.99592841e-01],\n",
       "       [9.46162462e-01, 5.38375713e-02],\n",
       "       [1.31923123e-03, 9.98680770e-01],\n",
       "       [9.52331603e-01, 4.76683713e-02],\n",
       "       [2.21908813e-05, 9.99977827e-01],\n",
       "       [1.00000000e+00, 1.30825780e-14],\n",
       "       [1.00000000e+00, 2.16020951e-10],\n",
       "       [8.72159183e-01, 1.27840862e-01],\n",
       "       [8.47569704e-01, 1.52430311e-01],\n",
       "       [9.09379840e-01, 9.06201974e-02],\n",
       "       [9.43907499e-01, 5.60924374e-02],\n",
       "       [5.34471683e-02, 9.46552873e-01],\n",
       "       [1.31643876e-01, 8.68356109e-01],\n",
       "       [3.39597091e-06, 9.99996662e-01],\n",
       "       [3.71313916e-04, 9.99628663e-01],\n",
       "       [2.85309970e-01, 7.14689970e-01],\n",
       "       [3.76513781e-05, 9.99962330e-01],\n",
       "       [7.59656191e-01, 2.40343794e-01],\n",
       "       [1.49495915e-01, 8.50504160e-01],\n",
       "       [2.08162978e-01, 7.91837037e-01],\n",
       "       [8.61448720e-02, 9.13855135e-01],\n",
       "       [9.09586906e-01, 9.04130936e-02],\n",
       "       [1.71284616e-01, 8.28715444e-01],\n",
       "       [6.97678246e-04, 9.99302268e-01],\n",
       "       [7.36443639e-01, 2.63556361e-01],\n",
       "       [9.99994874e-01, 5.08381436e-06],\n",
       "       [9.99995828e-01, 4.17978208e-06],\n",
       "       [1.60264164e-01, 8.39735866e-01],\n",
       "       [9.36790764e-01, 6.32092059e-02],\n",
       "       [2.73407727e-01, 7.26592243e-01],\n",
       "       [8.74279201e-01, 1.25720859e-01],\n",
       "       [2.04252198e-01, 7.95747817e-01],\n",
       "       [9.74132776e-01, 2.58671995e-02],\n",
       "       [1.71376243e-01, 8.28623772e-01],\n",
       "       [1.43989384e-01, 8.56010675e-01],\n",
       "       [9.99999881e-01, 1.55079618e-07],\n",
       "       [1.65808007e-01, 8.34192038e-01],\n",
       "       [1.25677720e-01, 8.74322295e-01],\n",
       "       [8.99645925e-01, 1.00354031e-01],\n",
       "       [1.00000000e+00, 3.52108342e-10],\n",
       "       [7.38604227e-04, 9.99261439e-01],\n",
       "       [9.92180705e-01, 7.81933218e-03],\n",
       "       [8.34785640e-01, 1.65214345e-01],\n",
       "       [9.41073418e-01, 5.89265861e-02],\n",
       "       [9.90084291e-01, 9.91566759e-03],\n",
       "       [1.74589112e-01, 8.25410962e-01],\n",
       "       [9.94754195e-01, 5.24575915e-03],\n",
       "       [5.30556321e-01, 4.69443709e-01],\n",
       "       [8.42182815e-01, 1.57817215e-01],\n",
       "       [8.31935585e-01, 1.68064430e-01],\n",
       "       [9.51957345e-01, 4.80426401e-02],\n",
       "       [5.05510727e-08, 1.00000000e+00],\n",
       "       [1.00121259e-04, 9.99899864e-01],\n",
       "       [2.39628367e-03, 9.97603714e-01],\n",
       "       [1.71284616e-01, 8.28715444e-01],\n",
       "       [9.36790764e-01, 6.32092059e-02],\n",
       "       [2.44515851e-01, 7.55484164e-01],\n",
       "       [7.54532754e-01, 2.45467275e-01],\n",
       "       [7.91186154e-01, 2.08813816e-01],\n",
       "       [4.24072132e-05, 9.99957561e-01],\n",
       "       [1.20235386e-03, 9.98797655e-01],\n",
       "       [1.07137099e-01, 8.92862856e-01],\n",
       "       [9.99988437e-01, 1.15772400e-05],\n",
       "       [9.32825208e-01, 6.71747774e-02],\n",
       "       [9.35358286e-01, 6.46417812e-02],\n",
       "       [9.24001575e-01, 7.59984627e-02],\n",
       "       [9.47844326e-01, 5.21556251e-02],\n",
       "       [1.36363208e-12, 1.00000000e+00],\n",
       "       [2.23325486e-07, 9.99999762e-01],\n",
       "       [7.81031395e-06, 9.99992132e-01],\n",
       "       [1.74097627e-01, 8.25902402e-01],\n",
       "       [1.18625851e-03, 9.98813748e-01],\n",
       "       [1.94631735e-04, 9.99805391e-01],\n",
       "       [8.09906602e-01, 1.90093428e-01],\n",
       "       [8.50985587e-01, 1.49014384e-01],\n",
       "       [9.99899387e-01, 1.00584992e-04],\n",
       "       [8.21366787e-01, 1.78633213e-01],\n",
       "       [1.42644765e-02, 9.85735476e-01],\n",
       "       [1.33095583e-07, 9.99999881e-01],\n",
       "       [1.74305350e-01, 8.25694740e-01],\n",
       "       [6.21503532e-01, 3.78496379e-01],\n",
       "       [9.99951601e-01, 4.84345655e-05],\n",
       "       [9.11360860e-01, 8.86391550e-02],\n",
       "       [1.90611824e-01, 8.09388161e-01],\n",
       "       [2.31202096e-01, 7.68797934e-01],\n",
       "       [8.90160322e-01, 1.09839723e-01],\n",
       "       [9.37067688e-01, 6.29323572e-02],\n",
       "       [2.60908335e-01, 7.39091635e-01],\n",
       "       [8.59778702e-01, 1.40221283e-01],\n",
       "       [8.59778702e-01, 1.40221283e-01],\n",
       "       [3.50830938e-08, 1.00000000e+00],\n",
       "       [9.36553776e-01, 6.34462237e-02],\n",
       "       [9.53717589e-01, 4.62823622e-02],\n",
       "       [8.01161776e-11, 1.00000000e+00],\n",
       "       [9.86434698e-01, 1.35653317e-02],\n",
       "       [9.81017947e-01, 1.89820696e-02],\n",
       "       [9.93108511e-01, 6.89145084e-03],\n",
       "       [2.33168736e-01, 7.66831219e-01],\n",
       "       [8.62063885e-01, 1.37936145e-01],\n",
       "       [9.01352882e-01, 9.86470878e-02],\n",
       "       [7.39938617e-01, 2.60061294e-01],\n",
       "       [9.19232011e-01, 8.07679892e-02],\n",
       "       [8.72496884e-08, 9.99999881e-01],\n",
       "       [3.30992401e-01, 6.69007540e-01],\n",
       "       [4.91868705e-01, 5.08131266e-01],\n",
       "       [9.99992847e-01, 7.13251393e-06],\n",
       "       [1.41568109e-01, 8.58431876e-01],\n",
       "       [8.31749201e-01, 1.68250829e-01],\n",
       "       [9.87377107e-01, 1.26228491e-02],\n",
       "       [5.45529311e-08, 1.00000000e+00],\n",
       "       [2.06853159e-07, 9.99999762e-01],\n",
       "       [9.99995112e-01, 4.86372619e-06],\n",
       "       [2.75831446e-02, 9.72416818e-01],\n",
       "       [1.52037348e-04, 9.99847889e-01],\n",
       "       [1.40129566e-01, 8.59870493e-01],\n",
       "       [3.11273098e-01, 6.88726902e-01],\n",
       "       [8.92705023e-01, 1.07294954e-01],\n",
       "       [5.68282127e-01, 4.31717932e-01],\n",
       "       [1.35730729e-01, 8.64269316e-01],\n",
       "       [9.33038354e-01, 6.69616386e-02],\n",
       "       [9.25598741e-01, 7.44012892e-02],\n",
       "       [1.73813909e-01, 8.26186061e-01],\n",
       "       [8.74867201e-01, 1.25132769e-01],\n",
       "       [8.04417610e-01, 1.95582375e-01],\n",
       "       [1.71284556e-01, 8.28715444e-01],\n",
       "       [8.43911886e-01, 1.56088129e-01],\n",
       "       [2.05281414e-02, 9.79471803e-01],\n",
       "       [8.77863705e-01, 1.22136347e-01],\n",
       "       [2.28381887e-01, 7.71618128e-01],\n",
       "       [6.93938091e-07, 9.99999285e-01],\n",
       "       [1.17268376e-01, 8.82731616e-01],\n",
       "       [1.43600404e-01, 8.56399655e-01],\n",
       "       [9.88984466e-01, 1.10155325e-02],\n",
       "       [1.00000000e+00, 7.92314714e-10],\n",
       "       [9.36790764e-01, 6.32092431e-02],\n",
       "       [1.22792069e-02, 9.87720788e-01],\n",
       "       [8.63676429e-01, 1.36323571e-01],\n",
       "       [2.25643488e-03, 9.97743607e-01],\n",
       "       [8.97667825e-01, 1.02332182e-01],\n",
       "       [1.41593307e-01, 8.58406663e-01],\n",
       "       [2.90168705e-03, 9.97098327e-01],\n",
       "       [7.39795446e-01, 2.60204464e-01],\n",
       "       [1.91238545e-08, 1.00000000e+00],\n",
       "       [1.41568109e-01, 8.58431876e-01],\n",
       "       [1.89582065e-01, 8.10417950e-01],\n",
       "       [3.93285811e-01, 6.06714129e-01],\n",
       "       [9.55958009e-01, 4.40419391e-02],\n",
       "       [7.13050842e-01, 2.86949158e-01],\n",
       "       [7.34080374e-01, 2.65919596e-01],\n",
       "       [1.26480415e-01, 8.73519540e-01],\n",
       "       [1.69507638e-01, 8.30492377e-01],\n",
       "       [1.56560197e-01, 8.43439758e-01],\n",
       "       [1.25036850e-05, 9.99987483e-01],\n",
       "       [2.00748746e-03, 9.97992516e-01],\n",
       "       [2.74758339e-02, 9.72524107e-01],\n",
       "       [9.37067688e-01, 6.29323572e-02],\n",
       "       [9.15173352e-01, 8.48266110e-02],\n",
       "       [9.25598741e-01, 7.44012892e-02],\n",
       "       [6.54797614e-01, 3.45202386e-01],\n",
       "       [7.29332896e-05, 9.99927044e-01],\n",
       "       [8.99819076e-01, 1.00180909e-01],\n",
       "       [3.48577206e-03, 9.96514261e-01],\n",
       "       [3.22226524e-01, 6.77773416e-01],\n",
       "       [7.99715417e-06, 9.99992013e-01],\n",
       "       [3.79240811e-01, 6.20759130e-01],\n",
       "       [9.09140170e-01, 9.08598304e-02],\n",
       "       [1.06165580e-01, 8.93834412e-01],\n",
       "       [8.40836883e-01, 1.59163088e-01],\n",
       "       [9.89776850e-02, 9.01022375e-01],\n",
       "       [5.25425225e-02, 9.47457552e-01],\n",
       "       [9.24801290e-01, 7.51987174e-02],\n",
       "       [9.36790764e-01, 6.32092059e-02],\n",
       "       [6.21804595e-01, 3.78195375e-01],\n",
       "       [8.52901697e-01, 1.47098318e-01],\n",
       "       [9.25598741e-01, 7.44012892e-02],\n",
       "       [9.99989510e-01, 1.04551364e-05],\n",
       "       [8.41192901e-01, 1.58807129e-01],\n",
       "       [2.30323712e-04, 9.99769628e-01],\n",
       "       [1.56590834e-01, 8.43409181e-01],\n",
       "       [1.00000000e+00, 1.39406409e-09],\n",
       "       [1.20195061e-01, 8.79804909e-01],\n",
       "       [5.37021132e-03, 9.94629741e-01],\n",
       "       [1.69202313e-01, 8.30797672e-01],\n",
       "       [8.75004292e-01, 1.24995761e-01],\n",
       "       [9.76945921e-06, 9.99990225e-01],\n",
       "       [8.27317357e-01, 1.72682658e-01],\n",
       "       [8.48093629e-01, 1.51906416e-01],\n",
       "       [1.18448310e-01, 8.81551743e-01],\n",
       "       [2.60160392e-04, 9.99739826e-01],\n",
       "       [1.23865582e-01, 8.76134455e-01],\n",
       "       [1.00000000e+00, 5.15396950e-14],\n",
       "       [8.42468679e-01, 1.57531351e-01],\n",
       "       [6.21478597e-04, 9.99378443e-01],\n",
       "       [2.74535120e-01, 7.25464880e-01],\n",
       "       [1.61259174e-02, 9.83874083e-01],\n",
       "       [9.03143406e-01, 9.68565866e-02],\n",
       "       [3.70898512e-08, 1.00000000e+00],\n",
       "       [1.51780378e-05, 9.99984860e-01],\n",
       "       [9.08165995e-04, 9.99091864e-01],\n",
       "       [8.87050390e-01, 1.12949692e-01],\n",
       "       [8.90101373e-01, 1.09898664e-01],\n",
       "       [1.93728685e-01, 8.06271255e-01],\n",
       "       [6.35244787e-01, 3.64755183e-01],\n",
       "       [6.14356054e-07, 9.99999404e-01],\n",
       "       [2.46549726e-01, 7.53450334e-01],\n",
       "       [1.97427675e-01, 8.02572310e-01],\n",
       "       [9.99697685e-01, 3.02287255e-04],\n",
       "       [8.43857884e-01, 1.56142160e-01],\n",
       "       [1.27648652e-01, 8.72351348e-01],\n",
       "       [7.77191341e-01, 2.22808704e-01],\n",
       "       [8.07536900e-01, 1.92463160e-01],\n",
       "       [2.21615977e-04, 9.99778450e-01],\n",
       "       [1.39313594e-01, 8.60686481e-01],\n",
       "       [9.99995232e-01, 4.82098721e-06],\n",
       "       [8.37600827e-01, 1.62399158e-01],\n",
       "       [5.86213637e-03, 9.94137883e-01],\n",
       "       [7.70920336e-01, 2.29079649e-01],\n",
       "       [9.34634686e-01, 6.53652847e-02],\n",
       "       [3.47001645e-08, 1.00000000e+00],\n",
       "       [9.19890046e-01, 8.01099241e-02],\n",
       "       [8.85693431e-01, 1.14306547e-01],\n",
       "       [3.72759392e-03, 9.96272445e-01],\n",
       "       [1.06953248e-05, 9.99989271e-01],\n",
       "       [1.00000000e+00, 4.75204372e-15],\n",
       "       [2.86030968e-06, 9.99997139e-01],\n",
       "       [1.72597423e-01, 8.27402651e-01],\n",
       "       [1.67688251e-01, 8.32311809e-01],\n",
       "       [2.70823104e-04, 9.99729216e-01],\n",
       "       [1.69416800e-01, 8.30583274e-01],\n",
       "       [7.86765695e-01, 2.13234290e-01],\n",
       "       [7.28446603e-01, 2.71553367e-01],\n",
       "       [9.99983430e-01, 1.65945585e-05],\n",
       "       [1.00000000e+00, 7.07102300e-15],\n",
       "       [8.45604122e-01, 1.54395953e-01],\n",
       "       [8.77681732e-01, 1.22318231e-01],\n",
       "       [1.57402456e-03, 9.98425961e-01],\n",
       "       [1.86600886e-08, 1.00000000e+00],\n",
       "       [8.74941409e-01, 1.25058666e-01],\n",
       "       [9.36657906e-01, 6.33420795e-02],\n",
       "       [9.26619351e-01, 7.33806342e-02],\n",
       "       [1.93879619e-01, 8.06120396e-01],\n",
       "       [3.93285811e-01, 6.06714129e-01],\n",
       "       [1.22016273e-01, 8.77983689e-01],\n",
       "       [1.11964196e-01, 8.88035834e-01],\n",
       "       [7.09489048e-01, 2.90510952e-01],\n",
       "       [1.16130851e-01, 8.83869112e-01],\n",
       "       [1.00000000e+00, 7.22684987e-13],\n",
       "       [8.42460811e-01, 1.57539144e-01],\n",
       "       [6.22172887e-03, 9.93778348e-01],\n",
       "       [9.06338274e-01, 9.36618000e-02],\n",
       "       [9.18777645e-01, 8.12222958e-02],\n",
       "       [1.18067406e-01, 8.81932676e-01],\n",
       "       [1.00000000e+00, 2.22446700e-10],\n",
       "       [2.85309970e-01, 7.14689970e-01],\n",
       "       [1.76587150e-01, 8.23412895e-01]], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can predict the probability of each class using the 'predict' method\n",
    "# The output comes in a scientific format\n",
    "# Please uncomment the round() method to achive a rounded result (not scientific notation)\n",
    "model.predict(test_inputs)#.round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 1., 0., 1., 1., 0., 0., 0., 1., 1., 1., 0., 0., 0., 1.,\n",
       "       0., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 0., 1., 1., 1., 1.,\n",
       "       1., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0.,\n",
       "       0., 0., 1., 0., 0., 0., 0., 1., 1., 0., 0., 1., 1., 0., 0., 0., 1.,\n",
       "       1., 0., 0., 1., 1., 1., 0., 0., 0., 0., 1., 1., 1., 1., 0., 1., 0.,\n",
       "       1., 1., 0., 0., 0., 0., 1., 1., 0., 1., 0., 1., 1., 0., 1., 1., 1.,\n",
       "       0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 1., 0., 0., 0., 1., 0., 0.,\n",
       "       0., 1., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0., 1., 0., 1., 0.,\n",
       "       0., 1., 1., 1., 0., 0., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0., 1.,\n",
       "       0., 0., 0., 1., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 1., 1.,\n",
       "       1., 0., 1., 1., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0., 1., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 0., 0.,\n",
       "       1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1., 0., 0., 0., 1.,\n",
       "       0., 1., 0., 1., 0., 1., 1., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0.,\n",
       "       1., 0., 0., 0., 0., 0., 1., 1., 1., 1., 0., 1., 0., 0., 1., 1., 1.,\n",
       "       0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 1., 1.,\n",
       "       1., 0., 0., 0., 1., 1., 0., 0., 1., 0., 0., 1., 0., 0., 1., 0., 0.,\n",
       "       0., 1., 0., 0., 0., 0., 1., 1., 1., 0., 1., 0., 0., 1., 1., 0., 1.,\n",
       "       1., 1., 1., 0., 0., 1., 0., 0., 1., 0., 0., 1., 0., 1., 0., 1., 1.,\n",
       "       1., 1., 0., 0., 0., 1., 0., 1., 0., 1., 1., 0., 1., 1., 1., 1., 0.,\n",
       "       0., 0., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 1., 0., 1., 1., 1.,\n",
       "       1., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1.,\n",
       "       1., 1., 0., 1., 0., 0., 1., 1., 1., 0., 0., 1., 1., 1., 0., 1., 1.,\n",
       "       1., 0., 0., 1., 0., 1., 1., 1., 0., 0., 1., 0., 0., 1., 1., 0., 0.,\n",
       "       1., 0., 0., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1., 1., 0., 0., 0.,\n",
       "       0., 0., 0., 1., 1., 0., 0., 0., 1., 1., 1., 1., 0., 1., 0., 0., 1.,\n",
       "       0., 0., 1., 0., 1., 1.], dtype=float32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Alternatively, we can get only the second column\n",
    "# The main idea is that we are often interested in ONLY ONE of the two outcomes\n",
    "# In this case we would like to know if the customer will convert again\n",
    "# Once more, we can round to 0 digits, to achieve only 0s or 1s\n",
    "model.predict(test_inputs)[:,1].round(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1,\n",
       "       1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0,\n",
       "       1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0,\n",
       "       0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0,\n",
       "       0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1,\n",
       "       1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0,\n",
       "       1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0,\n",
       "       0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1,\n",
       "       0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1,\n",
       "       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0,\n",
       "       1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0,\n",
       "       0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1,\n",
       "       1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0,\n",
       "       1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1,\n",
       "       1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,\n",
       "       1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,\n",
       "       1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1,\n",
       "       1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0,\n",
       "       1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1,\n",
       "       1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0,\n",
       "       0, 1, 0, 0, 1, 0, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A much better approach here would be to use argmax (arguments of the maxima)\n",
    "# Argmax indicates the position of the highest argument row-wise or column-wise\n",
    "# In our case, we want ot know which COLUMN has the higher argument (probability), therefore we set axis=1 (for columns)\n",
    "# The output would be the column ID with the highest argument for each observation (row)\n",
    "# For instance, the first observation (in our output) was [0.93,0.07]\n",
    "# np.argmax([0.93,0.07], axis=1) would find that 0.91 is the higher argument (higher probability) and return 0\n",
    "# This method is great for multi-class problems as it is independent of the number of classes\n",
    "np.argmax(model.predict(test_inputs),axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally we save the model using the built-in method TensorFlow method\n",
    "# We choose the name and the file extension\n",
    "# Since the HDF format is optimal for large numerical objects, that's our preferred choice here (and the one recommended by TF)\n",
    "# The proper extension is .h5 to indicate HDF, version 5\n",
    "model.save('audiobooks_model.h5') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting on new data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the relevant libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As usual we are starting a new notebook and we need to import all relevant packages\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the scaler and the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Sequential models without an `input_shape` passed to the first layer cannot reload their optimizer state. As a result, your model isstarting with a freshly initialized optimizer.\n"
     ]
    }
   ],
   "source": [
    "# To load the scaler we use the pickle method load\n",
    "scaler_deep_learning = pickle.load(open('scaler_deep_learning.pickle', 'rb'))\n",
    "# To load the model, we use the TensorFlow (Keras) function relevant for the operation\n",
    "model = tf.keras.models.load_model('audiobooks_model.h5')\n",
    "\n",
    "# Note that since we did not specify the input shape of our inputs in the modeling part, we get a warning\n",
    "# For feed-forward neural networks such as ours that is not an issue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the new data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The new data is located in 'New_Audiobooks_Data.csv'\n",
    "# To keep everything as before, we must specify the delimiter explicitly\n",
    "raw_data = np.loadtxt('New_Audiobooks_Data.csv',delimiter=',')\n",
    "# We are interested in all data except for the first column (ID)\n",
    "# Note that there are no targets in this CSV file (we don't know the behavior of these clients, yet!)\n",
    "new_data_inputs = raw_data[:,1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict the probability of a customer to convert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale the new data in the same way we scaled the train data\n",
    "new_data_inputs_scaled = scaler_deep_learning.transform(new_data_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.02, 0.  , 0.09, 1.  , 0.  , 0.07, 0.06, 0.07, 0.05, 0.78, 0.  ,\n",
       "       0.82, 0.99, 0.  , 0.12, 0.17, 0.89, 0.73, 0.87, 0.97, 1.  , 1.  ,\n",
       "       1.  , 0.  , 0.  , 0.99, 0.41, 0.  , 1.  , 1.  ], dtype=float32)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Predict the probability of each customer to convert\n",
    "# Here we have also taken only the second column and rounded to 2 digits after the dot\n",
    "model.predict(new_data_inputs_scaled)[:,1].round(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1,\n",
       "       1, 0, 0, 1, 0, 0, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Implement the better approach which is independent of the number of classes\n",
    "np.argmax(model.predict(new_data_inputs_scaled),1)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}