tutorials/explainers/RISE/rise_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dd073a9d",
   "metadata": {},
   "source": [
    "<img width=\"150\" alt=\"Logo_ER10\" src=\"https://user-images.githubusercontent.com/3244249/151994514-b584b984-a148-4ade-80ee-0f88b0aefa45.png\">\n",
    "\n",
    "### Model Interpretation for Binary MNIST Model using RISE\n",
    "\n",
    "This notebook demonstrates how to apply the RISE explainability method on a pretrained binary MNIST model using a hand-written digit image. It visualizes the relevance attributions for each pixel/super-pixel by displaying them on top of the input image.<br>\n",
    "\n",
    "[RISE](http://bmvc2018.org/contents/papers/1064.pdf) is short for Randomized Input Sampling for Explanation of Black-box Models. It estimates the relevance empirically by probing the model with randomly masked versions of the input image to obtain the corresponding outputs.<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "731728b1",
   "metadata": {},
   "source": [
    "### Colab Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b04da052",
   "metadata": {},
   "outputs": [],
   "source": [
    "running_in_colab = 'google.colab' in str(get_ipython())\n",
    "if running_in_colab:\n",
    "    # install dianna\n",
    "    !python3 -m pip install dianna[notebooks]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7516aee",
   "metadata": {},
   "source": [
    "### Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0975a61d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore') # disable warnings relateds to versions of tf\n",
    "import dianna\n",
    "import onnxruntime\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from scipy.special import softmax\n",
    "from dianna import visualization\n",
    "from dianna.utils.downloader import download"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43d79a04",
   "metadata": {},
   "source": [
    "#### 1 - Loading the model and the dataset\n",
    "Loads pretrained binary MNIST model and the image to be explained."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a05f279",
   "metadata": {},
   "source": [
    "Load saved binary MNIST data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "767a183f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load dataset\n",
    "data = np.load(download('binary-mnist.npz', 'data'))\n",
    "# load testing data and the related labels\n",
    "X_test = data['X_test'].astype(np.float32).reshape([-1, 1, 28, 28])\n",
    "y_test = data['y_test']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3d3f6e7",
   "metadata": {},
   "source": [
    "Load the pretrained binary MNIST model and define a model runner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "40de1f6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_model(data):\n",
    "    fname = download('mnist_model.onnx', 'model')\n",
    "    # get ONNX predictions\n",
    "    sess = onnxruntime.InferenceSession(fname)\n",
    "    input_name = sess.get_inputs()[0].name\n",
    "    output_name = sess.get_outputs()[0].name\n",
    "    \n",
    "    onnx_input = {input_name: data}\n",
    "    pred_onnx = sess.run([output_name], onnx_input)\n",
    "    \n",
    "    return softmax(pred_onnx[0], axis=1)\n",
    "\n",
    "pred_onnx = run_model(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "802b6fd3",
   "metadata": {},
   "source": [
    "Print class and image of a single instance in the test data for preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "de15bd4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The predicted class is: digit 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x121254400>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAANpklEQVR4nO3df+hVdZ7H8dcrV/+xojJWtImdioimaPshIayt1TBDW1L5jyk0tWTYjwlmaIUNVxohBmzZaemvQslyF7dhSIdkWnJa+zVmhPZj1bSZLIxRvmVipVIwa773j+9x+I597+d+vffce26+nw/4cu8973vueXPp1Tn3fM7x44gQgBPfSU03AKA/CDuQBGEHkiDsQBKEHUjir/q5Mduc+gd6LCI82vKu9uy2r7P9e9s7bT/QzWcB6C13Os5ue5ykP0j6gaTdkjZJmhcR2wvrsGcHeqwXe/YrJe2MiA8j4k+Sfinppi4+D0APdRP2syT9ccTr3dWyv2B7ge3Ntjd3sS0AXer5CbqIWCZpmcRhPNCkbvbseySdPeL1d6plAAZQN2HfJOl82+fYniBprqS19bQFoG4dH8ZHxGHb90laJ2mcpBUR8W5tnQGoVcdDbx1tjN/sQM/15KIaAN8ehB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4k0dcpm9EbM2bMaFl7/fXXi+tecMEFxfqsWbOK9RtuuKFYf+6554r1ko0bNxbrGzZs6PizM2LPDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMIvrADj11FOL9VWrVhXr1157bcvaV199VVx3woQJxfrJJ59crPdSu96//PLLYv2ee+5pWXvmmWc66unboNUsrl1dVGN7l6SDkr6WdDgipnXzeQB6p44r6K6JiH01fA6AHuI3O5BEt2EPSb+1/abtBaO9wfYC25ttb+5yWwC60O1h/IyI2GP7ryW9YPu9iHh15BsiYpmkZRIn6IAmdbVnj4g91eNeSb+WdGUdTQGoX8dhtz3R9ilHn0v6oaRtdTUGoF4dj7PbPlfDe3Np+OfAf0XEz9usw2H8KB577LFi/a677urZtnfs2FGsf/rpp8X6gQMHOt62Pepw8J+1u1e+nYMHD7asXXXVVcV1t2zZ0tW2m1T7OHtEfCjpbzvuCEBfMfQGJEHYgSQIO5AEYQeSIOxAEtzi2gcXXXRRsf7yyy8X65MmTSrWd+/e3bJ22223FdfduXNnsf75558X64cOHSrWS046qbyvefDBB4v1xYsXF+vjxo1rWVuzZk1x3TvvvLNY/+yzz4r1JrUaemPPDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMGVzH5xyyinFertx9HbXQjz88MMta+3G8Jt05MiRYn3JkiXFert/BnvhwoUta7Nnzy6uu2LFimK9m6mom8KeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4H72Ppg5c2ax/tJLLxXrTz31VLF+xx13HG9LKXzwwQcta+ecc05x3SeffLJYnz9/fkc99QP3swPJEXYgCcIOJEHYgSQIO5AEYQeSIOxAEtzP3gcPPfRQV+u/8cYbNXWSy7p161rW7r777uK606dPr7udxrXds9teYXuv7W0jlp1h+wXb71ePp/e2TQDdGsth/FOSrjtm2QOS1kfE+ZLWV68BDLC2YY+IVyXtP2bxTZJWVs9XSrq53rYA1K3T3+yTI2Koev6xpMmt3mh7gaQFHW4HQE26PkEXEVG6wSUilklaJuW9EQYYBJ0OvX1ie4okVY9762sJQC90Gva1km6vnt8u6dl62gHQK20P420/LelqSWfa3i3pZ5KWSvqV7fmSPpI0p5dNDrpzzz23WJ86dWqx/sUXXxTrW7duPe6eIL344osta+3G2U9EbcMeEfNalL5fcy8AeojLZYEkCDuQBGEHkiDsQBKEHUiCW1xrcOuttxbr7YbmVq9eXaxv3LjxuHsCjsWeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJy9BnPnzi3W293C+uijj9bZDjAq9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7H3w3nvvFesbNmzoUyfIjD07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOPsYTZw4sWVt/PjxfewE6EzbPbvtFbb32t42YtkS23tsv1P9Xd/bNgF0ayyH8U9Jum6U5f8eEZdWf/9db1sA6tY27BHxqqT9fegFQA91c4LuPttbqsP801u9yfYC25ttb+5iWwC61GnYH5N0nqRLJQ1J+kWrN0bEsoiYFhHTOtwWgBp0FPaI+CQivo6II5KWS7qy3rYA1K2jsNueMuLlbEnbWr0XwGBoO85u+2lJV0s60/ZuST+TdLXtSyWFpF2S7updi4Nhzpw5LWvnnXdecd19+/bV3Q7G4MYbb+x43cOHD9fYyWBoG/aImDfK4id60AuAHuJyWSAJwg4kQdiBJAg7kARhB5LgFld8a11xxRXF+qxZszr+7EWLFnW87qBizw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOjoHVbhz9/vvvL9ZPO+20lrXXXnutuO66deuK9W8j9uxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7GO0a9eulrWDBw/2r5ETyLhx44r1hQsXFuu33HJLsb5nz56OP/tE/Kek2bMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKOiP5tzO7fxvpo+/btxXq773jmzJnF+iBP+XzJJZcU6/fee2/L2uWXX15cd9q0aR31dNQ111zTsvbKK6909dmDLCI82vK2e3bbZ9t+yfZ22+/a/km1/AzbL9h+v3o8ve6mAdRnLIfxhyX9U0R8T9J0ST+2/T1JD0haHxHnS1pfvQYwoNqGPSKGIuKt6vlBSTsknSXpJkkrq7etlHRzj3oEUIPjujbe9nclXSbpDUmTI2KoKn0saXKLdRZIWtBFjwBqMOaz8bZPlrRa0k8j4sDIWgyfgRr1LFRELIuIaRHR3dkWAF0ZU9htj9dw0FdFxJpq8Se2p1T1KZL29qZFAHVoexhv25KekLQjIh4ZUVor6XZJS6vHZ3vS4QngwgsvLNaff/75Yn1oaKhYb9L06dOL9UmTJnX82e2GHNeuXVusb9q0qeNtn4jG8pv97yT9SNJW2+9UyxZpOOS/sj1f0keS5vSkQwC1aBv2iNggadRBeknfr7cdAL3C5bJAEoQdSIKwA0kQdiAJwg4kwS2uNZg9e3axvnjx4mL9sssuq7OdgXLkyJGWtf379xfXfeSRR4r1pUuXdtTTia7jW1wBnBgIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtn7YOrUqcV6u/vZL7744jrbqdXy5cuL9bfffrtl7fHHH6+7HYhxdiA9wg4kQdiBJAg7kARhB5Ig7EAShB1IgnF24ATDODuQHGEHkiDsQBKEHUiCsANJEHYgCcIOJNE27LbPtv2S7e2237X9k2r5Ett7bL9T/V3f+3YBdKrtRTW2p0iaEhFv2T5F0puSbtbwfOyHIuLfxrwxLqoBeq7VRTVjmZ99SNJQ9fyg7R2Szqq3PQC9dly/2W1/V9Jlkt6oFt1ne4vtFbZPb7HOAtubbW/urlUA3RjztfG2T5b0iqSfR8Qa25Ml7ZMUkh7S8KH+HW0+g8N4oMdaHcaPKey2x0v6jaR1EfGN2faqPf5vIqL4LyMSdqD3Or4RxrYlPSFpx8igVyfujpotaVu3TQLonbGcjZ8h6XeStko6Ov/uIknzJF2q4cP4XZLuqk7mlT6LPTvQY10dxteFsAO9x/3sQHKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJNr+g5M12yfpoxGvz6yWDaJB7W1Q+5LorVN19vY3rQp9vZ/9Gxu3N0fEtMYaKBjU3ga1L4neOtWv3jiMB5Ig7EASTYd9WcPbLxnU3ga1L4neOtWX3hr9zQ6gf5reswPoE8IOJNFI2G1fZ/v3tnfafqCJHlqxvcv21moa6kbnp6vm0Ntre9uIZWfYfsH2+9XjqHPsNdTbQEzjXZhmvNHvrunpz/v+m932OEl/kPQDSbslbZI0LyK297WRFmzvkjQtIhq/AMP230s6JOk/jk6tZftfJe2PiKXV/yhPj4h/HpDelug4p/HuUW+tphn/RzX43dU5/XknmtizXylpZ0R8GBF/kvRLSTc10MfAi4hXJe0/ZvFNklZWz1dq+D+WvmvR20CIiKGIeKt6flDS0WnGG/3uCn31RRNhP0vSH0e83q3Bmu89JP3W9pu2FzTdzCgmj5hm62NJk5tsZhRtp/Hup2OmGR+Y766T6c+7xQm6b5oREZdL+gdJP64OVwdSDP8GG6Sx08cknafhOQCHJP2iyWaqacZXS/ppRBwYWWvyuxulr758b02EfY+ks0e8/k61bCBExJ7qca+kX2v4Z8cg+eToDLrV496G+/mziPgkIr6OiCOSlqvB766aZny1pFURsaZa3Ph3N1pf/fremgj7Jknn2z7H9gRJcyWtbaCPb7A9sTpxItsTJf1QgzcV9VpJt1fPb5f0bIO9/IVBmca71TTjavi7a3z684jo+5+k6zV8Rv4DSf/SRA8t+jpX0v9Wf+823ZukpzV8WPd/Gj63MV/SJEnrJb0v6X8knTFAvf2nhqf23qLhYE1pqLcZGj5E3yLpnerv+qa/u0JfffneuFwWSIITdEAShB1IgrADSRB2IAnCDiRB2IEkCDuQxP8D0wdNenALPw0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# class name\n",
    "class_name = ['digit 0', 'digit 1']\n",
    "# instance index\n",
    "i_instance = 3\n",
    "# predicted class\n",
    "pred_class = class_name[np.argmax(pred_onnx[i_instance])]\n",
    "print(\"The predicted class is:\", pred_class)\n",
    "plt.imshow(X_test[i_instance][0], cmap='gray')  # 0 for channel"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "222af804",
   "metadata": {},
   "source": [
    "#### 2 - Compute and visualize the relevance attributions\n",
    "Compute the relevance attributions using RISE and visualize them overlayed on the image. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7224bb0d",
   "metadata": {},
   "source": [
    "RISE masks random portions of the input image and passes the masked image through the model — the portion that decreases the accuracy the most is the most “important” portion.<br>\n",
    "To call the explainer and generate the relevance scores, the user need to specified the number of masks being randomly generated (`n_masks`), the resolution of features in masks (`feature_res`) and for each mask and each feature in the image, the probability of being kept unmasked (`p_keep`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "87212eac",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Explaining: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 101.37it/s]\n"
     ]
    }
   ],
   "source": [
    "relevances = dianna.explain_image(run_model, X_test[i_instance], method=\"RISE\",\n",
    "                                labels=[i for i in range(2)],\n",
    "                                n_masks=5000, feature_res=8, p_keep=.1,\n",
    "                                axis_labels=('channels','height','width'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88278094",
   "metadata": {},
   "source": [
    "Visualize the relevance scores for the predicted class on top of the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "49ce231f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explanation for `digit 0`\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAASAAAADnCAYAAAC+GYs4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXxUlEQVR4nO3dbWxcZ5UH8P8/buzYTho7dhrSOE3dkrZJm90AIQgqoAJSsnwgvEgopVu1EtqCtO2HLnwoEipVUAWsQGglumjLKqLwgahqK9a7Mhv6uqSipXYUaEmKG9fQxG4odV6aNE6c2D77wWM6cXPPGftO8nRu/z9pFM+cee6duXNz/Mx9jp+HZgYRkRTmpH4BIvLOpQQkIskoAYlIMkpAIpKMEpCIJHOBF2xoaLLm5pbM+Jwc6YucfVsAiAbvvHietucjXlQp33d0vuU9H/M4evTAsJktnm37je9+tw2PjFT03J0HDmw3s42z3Ve1uQmoubkFGzb8U2a8sdHfeF2ds2N3z7GTJ/34xER27PRpv+3oqB+P2o+Nzb599J8073+kKH4uE3d0XM5lgqqv9+PR+ej9so2OqXcuAsD27Vte9p/hGx4ZQe+Xv1zRc3n33e159lVtOdOAiCRHpu3C5aAEJFIESkAikowSkIgkk2dEKCElIJFap2tAIpJUEROQmT90Gg0v5jkm0ZDs+Lgf9153NMyeNx4dl1OnsmN5h+HPZU88el/RZxKVL0Tb90TvO9r23Ll+3CspyVPaUDVFTEAiUiOUgEQkGSUgEUmC1CiYiCSkHpCIJKMEJCLJKAGJSBJFLUQ082teoikMvNqKvHUbeWp1oqlTorhXxwPEdR/ee/PqTYB4WonouEafmffa8kwzAsRTqHifWVRjFJk3z49HU8t4xz3vVB9VoYvQIpJMEXtAIlIjlIBEJImiXgMSkRpRowmoNq9ciciZpnpB0a2iTXEjyT6S/STvPEt8BcnHSD5H8kmSHWWxcZK/K926on2pByRSBFUaBSNZB+BeABsADALoIdllZnvKnvY9AD81s/tJfgzAtwHcVIqdMLO1le7PTUDj48Dx4zN5+WfyhifzDFUD8TC8N+QbDbMfO5Zv33l6ww0Nfjwa0s07jJ9nSoy85Q3ecc9b+tDU5McXLPDjzc3Zsej/fnTMc6vuNaD1APrNbGBy09wGYBOA8gS0GsC/lH5+AsAvZrszfQUTKYLqfQVbBmB/2f3B0mPlfg/gc6WfPwtgAcm20v15JHtJPkPyM9HO9BVMpAgq7wG1k+wtu3+fmd03w719DcAPSd4C4NcAhgBMlYquMLMhkpcBeJzk82b2UtaGlIBEiqDyBDRsZuuc+BCA5WX3O0qP/Y2ZvYJSD4jkfACfN7MjpdhQ6d8Bkk8CeA+AzASkr2AiRVC9r2A9AFaS7CRZD2AzgDNGs0i2k5zKHV8HsLX0eCvJhqnnALgWZ147egslIJFaNzUhWSW3gJmNAbgNwHYALwB4wMx2k9xC8tOlp10HoI/kiwCWALin9PgqAL0kf4/Ji9PfmTZ69hb6CiZSBFUsRDSzbgDd0x67q+znBwE8eJZ2vwGwZib7UgISKYIarYQOp+Pw6mmi9xxN3+CJ6lny1BFFtS5RPHpfUU+3s/OSzNjhw4Nu2+XL29z4e997pRtfs2alG9+zZ29mLKrj2bVrnxt/5RU/nqcOKBKdL1GtjleflXcppdz0t2AikpQSkIgkownJRCQZ9YBEJAldAxKRpJSARCQZJSARSaaoCcircYiWYfGc63lrvM8jGjCIXltzsz9pzxe+8Hk3fuWVnZmxxka/yKitzX9xbW3+gYuWp7nyyhWZsWiepPe9zz8hNmzw4/ff/z+ZsZ073Yr+3LU4eeJR23M+QKW14UUkqaL2gESkBigBiUgySkAikowSkIgkoUJEEUlKo2Aiksw7sQcUzYvj1WZE61tF8blzZ98+2nZUg7RhwwY3/v73X+HGvV9Wzc3+GxsZec2NDw35k/aMj/uLmnnz7kTzAbW2+u+7tdV/bzfeuCkzdvz4Qbftq6++6sbnz3fD4bpg3rpi0flyXnLDOzEBicjbgK4BiUhSSkAikowuQotIMuoBiUgSugYkIkkVMQGR/vQNeZaniaa88JZByRuPpm5obb3IjX/qU6vdeDTlxeHDRzNjDz/8sNv29dcPufHxcWcdJQCnT/vr23hTrIyO+if52rXXBfGPuPHGxuwDd+ON/rZ/+csuNw6ccKPRULp3vkb/96PzrSqKmIBEpEYoAYlIEjU8IVltvmoROdPUhejoVtGmuJFkH8l+kneeJb6C5GMknyP5JMmOstjNJPeWbjdH+1ICEimCKiUgknUA7gXwDwBWA7iB5PSLnt8D8FMz+zsAWwB8u9R2EYBvAvgAgPUAvkmy1dufEpBIEVSvB7QeQL+ZDZjZKQDbAEz/I73VAB4v/fxEWfyTAB4xs0NmdhjAIwA2ejtTAhIpgsoTUDvJ3rLbrdO2tAzA/rL7g6XHyv0ewOdKP38WwAKSbRW2PYMuQovUupkVIg6b2bqce/wagB+SvAXArwEMARifzYbcBDRnjl/TctIvOXGPSVR30djoxy+8cPbxaDqOjg7/xa1Z48zNAGB01C/8ePbZpzJjhw//2W0bHfPx4DSIljPy2k9M+O/rt799wo03NPjFX9dd96HM2OWXr3Lbnjq1y40PDLzoxqPjMurMYhJ9Jt4UJ1VTvVGwIQDLy+53lB77GzN7BaUeEMn5AD5vZkdIDgG4blrbJ72d6SuYSBFU7xpQD4CVJDtJ1gPYDOCMKk+S7SSncsfXAWwt/bwdwPUkW0sXn68vPZZJCUikCKqUgMxsDMBtmEwcLwB4wMx2k9xC8tOlp10HoI/kiwCWALin1PYQgG9hMon1ANhSeiyTrgGJ1Loq/zGqmXUD6J722F1lPz8I4MGMtlvxZo8opAQkUgT6UwwRSaZG/xRDCUikCNQDEpEkijohGenXzETv2esVRnPmRMukLGoJCjeGh7NjQeHGBcEELnUHBt34i3v3uvG9e5/NjEXLDb3xhh8/4U97k2sOp+i1RbVZe/c+6sa/+MWrM2Mdy1rctuMjl7nx//rv3W58dMJ/c4cPzy4GxLVZVVHEBCQiNUIJSESSUQISkSRqeEIyJSCRIlAPSESSUQISkWSUgEQkmaImIK8OKLru5a3NtXCh37atJSie2Lffjx84kB0LimU+9uEPB/ve54YHd/lz09Sd9uqQ/AKpaG6ZqA4oWqOqyZnqqLnZb9vW5scXNjmLjgH4S8+TmbGOQ5e7bTuiYpy+PjfcsHixG7/ooiWZsajO55zPB1TUQkQRqREaBRORZNQDEpFklIBEJAldAxKRpJSARCQZXYQWkWSK2AOK5gPy6nwAYP787NiiRX5bt44HAPbPvg6oNShoWXD8uBs/+eqrbvzVoE7IO3DR+lSn/VKaMB59Zt6cPkuX+m0bRw76T3hhyA0POMd13fXX+9sOPhO8/LIfDyZKmuMcuAULWty2UW1WbroGJCJJKQGJSDJKQCKSjBKQiCShCclEJCn1gEQkmaImIG9YOJrawVvGpeGCYA6DQ+6a9sBf/+rHnWV5/n7VKrdpa7D+zJ4//tGN749KBK66KjMUrBiE0VE/Hg3jR8sheeURjSeDKS9eesmPR8dlSfaUF2HbaDqO6HyJ6hOcdaIuXOHUmwA43Xoefs9XMQGR3Ajg3wDUAfhPM/vOtPglAO4H0FJ6zp1m1k3yUgAvAJia++QZM/uKty/1gESKoEoJiGQdgHsBbAAwCKCHZJeZ7Sl72jcAPGBmPyK5GkA3gEtLsZfMbG2l+6vNK1ci8qapi9CV3GLrAfSb2YCZnQKwDcCmac8xAFMlqwsBvDLbl64EJFIEU9XQ0Q1oJ9lbdrt12paWASj/vjtYeqzc3QD+keQgJns/t5fFOknuIvl/JIOpRfUVTKQYKv8KNmxm63Lu7QYAPzGz75P8IICfkbwGwAEAl5jZQZLvA/ALkleb2dGsDakHJFLrKu39VJakhgAsL7vfUXqs3JcAPAAAZvY0JicybzezUTM7WHp8J4CXAFzh7UwJSKQIqpeAegCsJNlJsh7AZgBd056zD8DHJ3fLVZhMQK+RXFy6iA2SlwFYCWDA25m+gokUQZVGwcxsjORtALZjcoh9q5ntJrkFQK+ZdQH4KoAfk7wDkxekbzEzI/kRAFtIngYwAeArZubW0+SajqOx0X8zTukEcOSI3ziq64jiRzO/duKaSy91m54MXtszO3b4+46KbZyTJarjidTX+3Fvug0AaGl25vP446DfeDCIR1OsePVXb7zht43mvBgZ8ePR9p3zKTqXW1rb/W1XQxX/FMPMujF5cbn8sbvKft4D4NqztHsIwEMz2Zd6QCK1TvMBiUhSSkAikowSkIgkowQkIskoAYlIEpqQTESSKmIPqK4OaGmZ/cZbW53gkFNXAcR1GdHEON4yK0GxzfBrr7nxfdHcNCtX+nFnIqXoF1lU5xO1j+qA3Pqqg8GyO1FtVvSZep9Z9B8seuNR+2BZHneipmAZpzpvfaoqMRQwAYlIbYgmB3y7UgISKQAlIBFJwiz/n/CkogQkUgDqAYlIMkpAIpKMEpCIJGFW0AQU1QGFNSunnfqIoHYCp0758aCuo765OTM2J1oDKrqiF9WUjAdrnjnvranJX5MsOixRnVBY1/XnY9mxaNGy6H9B9OK8epmwgCngnA8A8hdYeaLzoQoKmYBEpDZoFExEklEPSESSKOw1IBGpDUpAIpKMEpCIJKMEJCJJFPZvwebQ0Fjv1DCcdtaQAoBjTk1JzjofBLU8V69ZkxlbtHSp23YkmA8IdXV+PPp15NYB+fUq0aajtdrqRpzPBPA/s6gOKBLU2ly1dm12sKnJbTsRzQ8VHZhoLTevfbTtqO6sCtQDEpFklIBEJBklIBFJopbrgGpzKn0ROcPERGW3SpDcSLKPZD/JO88Sv4TkEyR3kXyO5KfKYl8vtesj+cloX+oBiRRAtXpAJOsA3AtgA4BBAD0ku8xsT9nTvgHgATP7EcnVALoBXFr6eTOAqwFcDOBRkleYWeZIlnpAIgUw9TUsulVgPYB+Mxsws1MAtgHYNH13AKamJ1gI4JXSz5sAbDOzUTP7E4D+0vYy+T2giQlgZCQ7Hg19esO20dGY609LEQ3LulM7RG3zDqtGUzc45QtNC/2mFwR91nn1QT97MMfSOdHnHUw7cfGyZW78itWrs4PB+fDYjh1uPPzMos/cO2eCZXdOTZzbLxozvAbUTrK37P59ZnZf2f1lAMrXnRoE8IFp27gbwK9I3g6gGcAnyto+M62t+6HrK5hIAcwgAQ2b2bqcu7sBwE/M7PskPwjgZySvmc2GlIBECqCKo2BDAJaX3e8oPVbuSwA2Tu7XniY5D0B7hW3PoGtAIgVQxVGwHgArSXaSrMfkReWuac/ZB+DjAEByFYB5AF4rPW8zyQaSnQBWAnjW25l6QCI1rpp1QGY2RvI2ANsB1AHYama7SW4B0GtmXQC+CuDHJO/A5AXpW8zMAOwm+QCAPQDGAPyzNwIGKAGJFEI1CxHNrBuTQ+vlj91V9vMeANdmtL0HwD2V7ksJSKQAarUSWglIpACKmYBIv6YlqnfxajeiuosFC/x4NF2Ht/5MtDZNNO1EtMRLNLWDc9zm+F+ZMa8hOOZHg+k2ouWQTpzIjgV1QBcvXuzGP/iB6eUkZ5rnTNexb98+t23/wIAbD8+n6Hx0an2s0a8rO3HU33Q1FDMBicjbXmEnJBOR2qAekIgkowQkIskoAYlIErU8IZkSkEgB6CK0iCRTyB6QcQ5GL8iueZnb6NfDzGl26kaipW0iwfwuR5w6odFoPqDWVj9+4YV+PNq+tzxNVN80NubHoxqmKO4sGTQnmO/nQ+vduadwzdVXu/GjR7MLZn716KNu24nofIo+k+gzdWrHDueYYqlaCpmAROTtT9eARCQpJSARSUYJSESS0SiYiCSha0AikpQSkIgkU8gENDYGHDky+403OLU6Le96l984mp/l9dfd8J+OZc+LcyyYr6fh4ovdeFMQHwkX73L2H82xlPPL/pKlS934eqeWZ2nw2i6OammCGqeHH3ooMzY4POxve2GwoFo0H1DQfnxu9mcWlVYF5VNVUcgEJCJvf5oPSESSUg9IRJJRAhKRZJSARCQZJSARSaKwhYjj4/5od3Tl3Zt1wpn1AQCwYIE/LNq4NMf0CsFQ9OL2djd+U0eHGz8WTZnhDNNHQ7Z1Fmw7qJvoCEoQmkZGsoMHD7ptRw4dcuN9fX1ufMgbz25rc9uG02ksWuTHg6WavNWMonP5fIxQaRRMRJKp1R5QUPUmIrVg6mtYdKsEyY0k+0j2k7zzLPEfkPxd6fYiySNlsfGyWFe0L/WARGpcNa8BkawDcC+ADQAGAfSQ7DKzPW/uz+4oe/7tAN5TtokTZra20v2pByRSAFXsAa0H0G9mA2Z2CsA2AJuc598A4Oezfd1KQCIFMIME1E6yt+x267RNLQOwv+z+YOmxtyC5AkAngMfLHp5X2u4zJD8TvW59BRMpgBmMgg2b2boq7XYzgAfNrHzsdoWZDZG8DMDjJJ83s5eyNqAekEiNq7T3U+FXsCEAy8vud5QeO5vNmPb1y8yGSv8OAHgSZ14fegu3BzQx4S8pEtWszJ2bHYtKZaLaivnznY0DaGpqyYw9+tzzbtuPfvQjbvyiy69w423BcfFOhOgkCcp4AGcaEgCAs/QNAJizxsyJUWeZJQBP79rlxnfs3OnGsXhxdqzZXwIqjAe1XcfH/GWevMMWTccRrbRUDVUchu8BsJJkJyYTz2YAX5z+JJJXAWgF8HTZY60ARsxslGQ7gGsB/Ku3M30FEymAaiUgMxsjeRuA7QDqAGw1s90ktwDoNbOpofXNALaZnbHnVQD+g+QEJr9dfad89OxslIBECqCahYhm1g2ge9pjd027f/dZ2v0GwJqZ7EsJSKTGaUIyEUmqVv8UQwlIpACUgEQkGSUgEUmmsAnIq2GI6hu8Wp+8S5lEdULetDZPPfWC2/YPfxh04zfddJMbX7r0IjeeR3TMG4LlZ3Y+79dA/WXv3sxYz44d/s6jJYUuu8yP19Vlx6JlmpwloADg5IQzORWAI/5UR7nmA4pWacqrsBOSiUht0CiYiCSjHpCIJKMEJCJJ6BqQiCSlBCQiySgBiUgyhRwFI/3SjKjsI09WPn3aj0fzCXl1RlFdxrFgTp3vfvff3Xg4Z48jz1prQPzeorhXbsMlS/zG3gRQiM8H7zMP194K6sq8ea2AcJoknDiRHYvOVdUBZVMPSKQAlIBEJBklIBFJRglIRJLQhGQikpR6QCKSjBKQiCTzjkxAUR2Q9700OmDRd9qoDsibNycoVwm3HdV1RMfFE9W7BEtzhfMFeXVd0fbr6/0DN+bMwQTEn6k3B1T0vqP5o7w6HiCuE/L2H50vec6HSqgOSESSUgISkWQ0CiYiyagHJCJJ6BqQiCRVqwnoHF+fF5HzYaoXFN0qQXIjyT6S/STvPEv8ByR/V7q9SPJIWexmkntLt5ujfbk9oOhFR0O+XtvoYERDm9FwtSe6YJf3t0k0PUOe4eZI9N6iEgLvuEfTjOQ9rnmWcYrOl2gY3lt2B/DPt+jzPtfD8ED1ekAk6wDcC2ADgEEAPSS7zGzPm/uyO8qefzuA95R+XgTgmwDWATAAO0ttD2ftTz0gkRo39bdgldwqsB5Av5kNmNkpANsAbHKefwOAn5d+/iSAR8zsUCnpPAJgo7czJSCRApjBV7B2kr1lt1unbWoZgP1l9wdLj70FyRUAOgE8PtO2U3QRWqQAZvAVbNjM1lVpt5sBPGhmQR16NvWARAqgihehhwAsL7vfUXrsbDbjza9fM20LQAlIpBCqmIB6AKwk2UmyHpNJpmv6k0heBaAVwNNlD28HcD3JVpKtAK4vPZZJX8FEalw1JyQzszGSt2EycdQB2Gpmu0luAdBrZlPJaDOAbWZvpjUzO0TyW5hMYgCwxcwOeftTAhIpgGoWIppZN4DuaY/dNe3+3RlttwLYWum+wgSUp5Ynz3Qc0fQKeWovovqlvPGo1sZ77XnrgKLjGr127zOLfsvmqQuL4tH5ENWFRXVAUdyrQ4rOxfOhViuh1QMSKQAlIBFJQn+MKiJJKQGJSDKakExEklEPSESS0DUgEUmqVhMQzXnlJF8D8PL5ezki70grzGzxbBuT/F8A7RU+fdjM3Ckyzic3AYmInEv6Y1QRSUYJSESSUQISkWSUgEQkGSUgEUnm/wGjrJEfN8XP/AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(f'Explanation for `{pred_class}`')\n",
    "fig, _ = visualization.plot_image(relevances[0], X_test[i_instance][0], data_cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1004be3",
   "metadata": {},
   "source": [
    "#### 3 - Conclusions\n",
    "The relevance scores are generated by passing multiple randomly masked inputs to the black-box model and averaging their pixel relevances. The idea behind this is that whenever a mask preserves important parts of the image it gets higher score. <br>\n",
    "\n",
    "The example here shows that the RISE method evaluates the relevance of each pixel/super pixel to the classification and highlights the lateral parts of the digit zero. This implies that the model determines whether the digit is 0 by checking the lateral parts. The results are reasonable compared to the human visual preception of the test image."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}