VFD_NOMA_Evaluation(initializers)UsingGridSearchCV.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "VFD_NOMA_Evaluation(initializers)UsingGridSearchCV.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNSOC9/L+IgPRkjpzm40l9S",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/shivendr7/VFD_NOMA/blob/main/VFD_NOMA_Evaluation(initializers)UsingGridSearchCV.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m-PevXqHjQO5"
      },
      "source": [
        "Hyperparameters tunning is an essential part of any Machine Learning project and one of the most time consuming. Even for the simplest models it can take hours to find the optimal parameters not mentioning neural nets that can be optimized day, weeks or even longer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "errDglYvMORM"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from keras.layers import Dense, Dropout, BatchNormalization\n",
        "from keras.models import Sequential\n",
        "from keras.losses import MeanSquaredError, CosineSimilarity, MeanAbsoluteError, MeanSquaredLogarithmicError, MeanAbsolutePercentageError, Huber\n",
        "from keras.optimizers import Adam"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aPBcrTOGjR87"
      },
      "source": [
        "##GridSearch\n",
        "The first and the simplest method to try is GridSearchCV which is included in sklearn.model_selection This approach just trying all available parameters' combinations 1 by 1 and choose the one with the best cross validation results.\n",
        "\n",
        "This approach has several drawbacks:\n",
        "\n",
        "It is very slow - you just try ALL combinations of ALL parameters and it takes a lot of time. Any additional parameter to variate multiply the number of iterations you need to complete. Imagine that you add to the parameter grid a new parameter with 10 possible values, this parameter can turn out to be meaningless but the computational time will be increased 10 times.\n",
        "It can work only with discrete values. If the global optimum is on n_estimators=550, but you are doing GridSearchCV from 100 to 1000 with step 100, you will never reach the optimal point.\n",
        "You need to know / guess the approximate localization of the optimum to complete the search in a reasonable time.\n",
        "You can overcome some of this drawbacks: you can do grid search parameter by parameter, or use it several times starting from the broad grid with large steps and narrowing the boundaries and decreasing step sizes on any iterations. But is still will be very computationally intensive and long.\n",
        "\n",
        "Let's estimate the time to do the Grid Search in our case. Let's suppose we want our grid to consist of 20 possible values of 'n_estimators' (100 to 2000), 19 values of 'max_depth' (2 to 20), and 5 values of 'learning_rate' (10e-4 to 0.1).\n",
        "\n",
        "This means we need to compute cross_val_score 20*19*5 = 1 900 times. If 1 computation takes ~0.5-1.0 second, our grid search will last for ~15-30 minutes. It is too much for the dataset with ~400 data points."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_8p3T9mvMjWo",
        "outputId": "f4caf4ef-1722-40c8-eda1-3e1979558a89"
      },
      "source": [
        "!wget 'https://raw.githubusercontent.com/shivendr7/VFD_NOMA/main/Data_P1_100000samples.csv'"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-07-05 15:23:28--  https://raw.githubusercontent.com/shivendr7/VFD_NOMA/main/Data_P1_100000samples.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 5587697 (5.3M) [text/plain]\n",
            "Saving to: ‘Data_P1_100000samples.csv’\n",
            "\n",
            "Data_P1_100000sampl 100%[===================>]   5.33M  --.-KB/s    in 0.08s   \n",
            "\n",
            "2021-07-05 15:23:29 (66.1 MB/s) - ‘Data_P1_100000samples.csv’ saved [5587697/5587697]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CrZhsLeNOyvJ",
        "outputId": "087baa73-97c7-452b-b538-fca95a8129fe"
      },
      "source": [
        "import pandas as pd\n",
        "from sklearn.model_selection import train_test_split\n",
        "df=pd.read_csv('Data_P1_100000samples.csv')\n",
        "X=np.array(df[df.columns[:9]])\n",
        "y=np.array(df[df.columns[-1]])\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=30)\n",
        "X_train.shape, y_test.shape"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((67000, 9), (33000,))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "06KrjPUkRYUw"
      },
      "source": [
        "from keras.wrappers.scikit_learn import KerasClassifier\n",
        "from sklearn.model_selection import GridSearchCV"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2t2vRwcRde7"
      },
      "source": [
        "def create_model(init='he_uniform'):\n",
        "  mape=MeanAbsolutePercentageError()\n",
        "  model=Sequential()\n",
        "  model.add(Dense(128, activation='relu', kernel_initializer=init, input_shape=(9,), kernel_regularizer='l2'))\n",
        "  #model.add(Dropout(0.08))\n",
        "  #model.add(Dense(128, activation='relu', kernel_initializer='he_uniform', kernel_regularizer='l2')) #-0\n",
        "  #model.add(Dropout(0.04))\n",
        "  #model.add(BatchNormalization())\n",
        "  model.add(Dense(64, activation='relu', kernel_initializer=init, kernel_regularizer='l2'))\n",
        "  model.add(Dropout(0.02))\n",
        "  #model.add(BatchNormalization())\n",
        "  model.add(Dense(64, activation='relu', kernel_initializer=init, kernel_regularizer='l2'))\n",
        "  #model.add(Dropout(0.04))\n",
        "  model.add(BatchNormalization())  \n",
        "  model.add(Dense(32, activation='relu', kernel_initializer=init, kernel_regularizer='l2'))\n",
        "  model.add(Dropout(0.02))\n",
        "  #model.add(BatchNormalization())\n",
        "  model.add(Dense(32, activation='relu', kernel_initializer=init, kernel_regularizer='l2'))\n",
        "  #model.add(Dropout(0.02))\n",
        "  #model.add(BatchNormalization())\n",
        "  model.add(Dense(1))\n",
        "  h_loss=Huber(delta=6)\n",
        "  model.compile(loss=[mape, h_loss], optimizer=Adam(learning_rate=0.001))\n",
        "  return model"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YL7EuwMVSNX4"
      },
      "source": [
        "model=KerasClassifier(build_fn=create_model, verbose=0, batch_size=25, epochs=50, validation_split=0.3)\n",
        "init_mode=['uniform', 'he_uniform']\n",
        "param_grid=dict(init=init_mode)\n",
        "grid=GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3, scoring='neg_mean_absolute_error')"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2eSSjlCS__0"
      },
      "source": [
        "results=grid.fit(X_train, y_train)"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YnwIbLvVeJ88",
        "outputId": "65cd162e-c755-4c69-b053-658820b48e92"
      },
      "source": [
        "grid_result=results\n",
        "print(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\n",
        "means = grid_result.cv_results_['mean_test_score']\n",
        "stds = grid_result.cv_results_['std_test_score']\n",
        "params = grid_result.cv_results_['params']\n",
        "for mean, stdev, param in zip(means, stds, params):\n",
        "    print(\"%f (%f) with: %r\" % (mean, stdev, param))"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Best: -0.165893 using {'init': 'uniform'}\n",
            "-0.165893 (0.000139) with: {'init': 'uniform'}\n",
            "-0.165893 (0.000139) with: {'init': 'he_uniform'}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J53hFgT5jzdt"
      },
      "source": [
        "##Random Search \n",
        "is on average more effective than Grid Search.\n",
        "\n",
        "Main advantages:\n",
        "\n",
        "Don't spend time on meaningless parameters. On every step random search variate all parameters.\n",
        "\n",
        "On average finds ~optimal parameters much faster than Grid search.\n",
        "It is not limited by grid when we optimize continuous parameters.\n",
        "Disadvantages:\n",
        "\n",
        "It may not find the global optimal parameter on a grid.\n",
        "\n",
        "All steps are independent. On every particular step it does not use any information about the results gathered so far. But they can be useful. For example, if we found a good solution it can be useful to search around it to find even better point comparing to looking at other completely random variants."
      ]
    }
  ]
}