NEU 171 Connectivity Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align=\"center\">Introduction to basic statistical analysis using the Allen Brain Connectivity Atlas</h1>\n",
    "<p><center>This notebook will introduce you to the Allen Mouse Brain Connectivity Atlas tools and data and will guide you in conducting basic statistical analysis.</center></p>\n",
    "<img src='./images/connectivity_website.png' width=\"500px\"/>\n",
    "\n",
    "<h3 align=\"center\">Estimated Duration: 30 mins</h3>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Table of Contents:\n",
    "\n",
    "[Introduction](#section_intro)\n",
    " \n",
    "**Part I: Brain Connectivity Statistics**\n",
    "1. [Acquiring the Data](#section_data)\n",
    "2. [Most Projected Structure](#section_projection)\n",
    "3. [Projection Density Distribution](#section_distr)\n",
    "4. [Medial Mammilary Nucleus: Average Projection Density](#section_mean)\n",
    "5. [Average Projection Densities across all Structures](#section_avg_proj_density) \n",
    "\n",
    "**Part II: Bootstrap**\n",
    "1. [Aquiring the Data](#section_data2)\n",
    "2. [Medial Mammilary Nucleus DataFrame](#section_mmn)\n",
    "3. [Bootstrapping](#section_bootstrapping)\n",
    "4. [95% Confidence Interval](#section_ci)\n",
    "5. [The P-Value](#section_pval)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction<a id='section_intro'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "The <b>Allen Brain Institute for Brain Science</b> is a non-profit research institute that utilizes big science to understand specific aspects of brain function. The insitute has many publicly available atlases online that provide information on gene expression and neural connectivity. The atlas that we will be working with today is the Mouse Brain Connectivity Atlas, defined as \"a brain-wide map of neural projections, including cell class specific data.\" \n",
    "\n",
    "In this part of the lab, we will utilize information from the Mouse Brain Connectivity Atlas to make conclusions regarding structural connectivity and to further explore projection information.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS CELL FIRST or the notebook won't work\n",
    "\n",
    "# Import some necessary software\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "plt.style.use('fivethirtyeight')\n",
    "from allensdk.core.mouse_connectivity_cache import MouseConnectivityCache\n",
    "from allensdk.api.queries.ontologies_api import OntologiesApi\n",
    "\n",
    "%matplotlib inline \n",
    "import seaborn as sns\n",
    "\n",
    "# Activates the API and cache\n",
    "mcc = MouseConnectivityCache()\n",
    "structure_tree = mcc.get_structure_tree()\n",
    "\n",
    "# supress Allen SDK warnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "In this lab, we will obtain specific information about individual structures of the brain and the experiments done on them, information that we will later be able to use to conduct basic statistical analyses and make conclusions regarding connectivity. \n",
    "\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Brain Connectivity Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 1.1: Aquiring the Data<a id='section_data'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "Running the cell below, you will obtain the data that we will be using in this lab. This data contains a set of experiments done on the Subiculum (Structure ID: 502) in wild-type mice. You don't need to worry about what these variables mean, but if you would like to learn more, you can go to <a href=\"https://allensdk.readthedocs.io/en/latest/_static/examples/nb/mouse_connectivity.html\">Mouse Connectivity</a> for reference.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to load the data; you don't need to know all the details of the code\n",
    "# If you want to know generally what's happening, read the comments in this cell\n",
    "\n",
    "# Get the list of structures in the Brain - Summary Structures set\n",
    "summary_structures = structure_tree.get_structures_by_set_id([167587189])\n",
    "# Extract just the ID numbers for the Summary Structures into a list\n",
    "summary_structure_ids = [item['id'] for item in summary_structures]\n",
    "\n",
    "# Get the list of experiments done on the Subiculum (structure 502)\n",
    "experiments = mcc.get_experiments(injection_structure_ids=[502]) \n",
    "# Extract just the ID numbers of the Subiculum experiments into a list,\n",
    "# keeping only experiments where 'transgenic_line' doesn't have a value\n",
    "experiment_set = [exp['id'] for exp in experiments if exp['transgenic_line'] == None]\n",
    "\n",
    "# Get a dictionary of {structure ID # : structure name} pairs\n",
    "name_map = structure_tree.get_name_map()\n",
    "\n",
    "# Get a DataFrame of records that describe projection signal in the above structures and experiments\n",
    "unionizes_set = mcc.get_structure_unionizes(experiment_set,\n",
    "                                               is_injection = False,\n",
    "                                                structure_ids = summary_structure_ids,\n",
    "                                               hemisphere_ids = [3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "We have now gathered all the information needed to do statistical and data analysis. Our `unionizes_set` DataFrame (fancy word for a table) consists of a list of structures where the Subiculum projects to when it is injected. Some of the columns in this DataFrame that we will be analyzing are:\n",
    "\n",
    "</div>\n",
    "\n",
    "- `projection_density`: The projection density is defined as the sum of detected pixels in the image/ sum of all pixels in division. It represents the strength of the connection between two structures in the brain as the more colored pixels there are in the image, the stronger the connection.\n",
    "- `structure_id`: A unique number assigned to every individual structure in the brain by the Allen Insitute for Brain Science.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 1.2: Most Projected Structure<a id='section_projection'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Let's check to see which structure has the largest projection density. \n",
    "    \n",
    "To do this, we want use the <code>sort_values</code> function with the correct column label (projection density) and order (descending). This will output a DataFrame with projection densities sorted from largest to smallest. Run the cell below and  find the <code>structure_id</code> of the first row. Assign this number to <code>mystery_id</code> in the next cell. The function <code>.head()</code> will cause the table to display just the first five rows.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the data sorted from largest to smallest projection density\n",
    "index = unionizes_set.sort_values('projection_density', ascending=False)[['hemisphere_id', 'id', 'projection_density', 'projection_volume', 'experiment_id', 'structure_id']]\n",
    "index.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace the ... with the number of the structure_id with the largest projection density\n",
    "mystery_id = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "We can't immediately recognize what structure corresponds to this id. The function <code>name_map</code> provides us with the name of the structure that corresponds to the structure id. Run the cell below to find what the structure is!\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to see the name of the structure with the largest projection density\n",
    "name_map[mystery_id]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Does our result make sense? Write down in the cell below why or why not. You can write in a cell by double clicking in it, typing and then hitting shift-return. Although you do not need to submit the entire Jupyter notebook, writing your answers will be a helpful way for you to think about what you are doing and to use later as a study tool.\n",
    "\n",
    "<p><b>Hint:</b> Where should the subiculum project to?</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your Answer Here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Now that we know the structure id and name of where the highest projection density is, let's get a subset of that data. Run the cell below to get a DataFrame of structure ids that are equal to number we found in the previous question.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to show only rows with the mystery_id structure ID\n",
    "medial_mammillary = unionizes_set[unionizes_set['structure_id'] == mystery_id]\n",
    "medial_mammillary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 1.3: Projection Density Distribution<a id='section_distr'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "Let's observe the distribution of values in the Medial Mammilary Nucleus. Recall in statistcs the distribution of values in a normal distribution:\n",
    "\n",
    "\n",
    "<img src='./images/normal.png' width=\"400px\"/>\n",
    "<i>Image from Illinois State University</i>\n",
    "\n",
    "<p>First, we will set <code>proj_dens</code> to a list of projection density values from our medial_mammillary table using <code>.values</code>. Run the cell below to visualize the distribution of projection densities in the medial mammilary nucleus.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gives us a list of projection_densities in the medial mammilary nucleus\n",
    "proj_dens = medial_mammillary['projection_density'].values\n",
    "\n",
    "# Make a Kernel Density Estimation, or KDE, plot of the distribution of MMN projection density values\n",
    "# A KDE plot is like a smoothed histogram\n",
    "plt.title('Distribution of Projection Densities at the Medial Mammilary Nucleus')\n",
    "plt.xlabel(\"Projection Density\")\n",
    "sns.kdeplot(proj_dens);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "Based on the distribution above, what do you think the values of the mean, median, and standard deviation are? Type your answer in the cell below:\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your Answer Here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 1.4: Medial Mammilary Nucleus: Average Projection Density<a id='section_mean'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "We are going to calculate the average of the data in the <code>proj_dens</code> list we made above. Let's first just take a look at the numbers in this list. Recall that you can ask Python to show you the values in a list by using the <code>print</code> command.\n",
    "</div>    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Put in the name of the projection density list, so we can see the values\n",
    "print(...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "Let's calculate the <b>mean of the projection densities</b> from the subicculum to <b>just</b> the medial mammillary nucleus. In other words, of all the projections from the subiculum to the medial mammillary nucleus, what is the projection density on average? Assign this value to <code>mean_values</code>.\n",
    "\n",
    "<p><b>Hint:</b> Recall the functions <code>sum</code> and <code>len</code> that allow you to calculate the mean as discussed in the Pre-Lab.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO-DO: sum the projection densities\n",
    "sum_values = ...\n",
    "# TO-DO: get the length of the projection densities array\n",
    "num_values = ...\n",
    "\n",
    "# TO-DO: calculate the average projectiond density\n",
    "# Hinte: use sum_values and num_values\n",
    "mean_values = ... / ...\n",
    "mean_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 1.5: Average Projection Densities across all Structures<a id='section_avg_proj_density'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Using our statistical analysis, we can observe the average projection densities at the top 20 sites where the subiculum maps to for the 5 wildtype injections. In other words, for each of the 20 structures that the subiculum maps to, we will calculate the average projection density. Luckily, the Connectivity API provides an easy way to display these averages: <b>Simply run the cell below to visualize a bar chart representation of this.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "# You DON'T need to understand the code details, but you can read the comments to learn more\n",
    "\n",
    "# Find the average projection density across experiments for each summary structure\n",
    "unionizes_set_mean = (unionizes_set.groupby('structure_id', as_index = False)['projection_density'].mean())\n",
    "\n",
    "# Add a column with summary structure acronyms so we can interpret the unionizes more easily\n",
    "names = [name_map[strid] for strid in unionizes_set_mean['structure_id']]\n",
    "unionizes_set_mean['structure_name'] = names\n",
    "unionizes_set_mean.sort_values(by = 'projection_density', ascending = False, inplace = True)\n",
    "\n",
    "# Plot the 20 structures with the highest average projection density across all wild type Subiculum injections\n",
    "unionizes_set_mean[:20].plot('structure_name', 'projection_density', kind='barh')\n",
    "plt.gca().invert_yaxis()\n",
    "plt.title('Average Unionize Data for %i injections'%len(experiment_set));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "The cell below provides statistics (i.e. mean and standard deviation) of the projection densities from the subiculum to various structures.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the data\n",
    "# Re-generate means so they are in the same order as standard deviation\n",
    "unionizes_set_mean = (\n",
    "    unionizes_set.groupby('structure_id', as_index = False)[\n",
    "    'projection_density'].mean())\n",
    "\n",
    "# Generate a second data frame with standard deviations\n",
    "unionizes_set_std = (\n",
    "    unionizes_set.groupby('structure_id', as_index = False)[\n",
    "    'projection_density'].std())\n",
    "\n",
    "# Find names from structure ids\n",
    "names = [name_map[strid] for strid in unionizes_set_mean['structure_id']]\n",
    "\n",
    "# Combine names, mean, and standard deviation into one dataframe for easy plotting\n",
    "set_mean_std = pd.DataFrame({'structure_name': names, \n",
    "                            'mean_projection_density': unionizes_set_mean['projection_density'], \n",
    "                            'stdev':  unionizes_set_std['projection_density']\n",
    "                            })\n",
    "\n",
    "# Sort the dataframe to get the top 20 values on top\n",
    "set_mean_std.sort_values(by = 'mean_projection_density', ascending = False, inplace = True)\n",
    "\n",
    "set_mean_std.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Do your results for the medial mammilary nucleus match what you estimated above? Does the mean match what you calculated before? Explain in the cell below.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your Answer Here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Part 2: Bootstrap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 2.1: Aquiring the Data<a id='section_data2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "For this part of the lab, we will apply a technique called **bootstrapping** to analyze the probability of a structure being a part of a circuit. We will start off with looking at the *primary visual cortex* since it has more wildtype injections than the subiculum, with 33 wildtype samples. \n",
    "\n",
    "The <code>visual_unionizes_set</code> consists of all of the structures that the primary visual area projects to for every experiment done.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to get the primary visual area data\n",
    "# You don't need to understand the code details fully, but you can read the comments to learn more\n",
    "\n",
    "# Get the primary visual area information\n",
    "VISp = structure_tree.get_structures_by_name([\"Primary visual area\"])\n",
    "# Extract just the primary visual area ID\n",
    "VISp_id = VISp[0]['id']\n",
    "# Get a list of all experiments that had injections into the primary visual area \n",
    "visual_experiments = mcc.get_experiments(injection_structure_ids=[VISp_id]) \n",
    "# Extract just the IDs of the primary visual area experiments,\n",
    "# keeping only experiments that do not have a value for 'transgenic_line'\n",
    "visual_experiment_set = [exp['id'] for exp in visual_experiments if exp['transgenic_line'] == None]\n",
    "# Put the experiment data into a DataFrame\n",
    "visual_data = pd.DataFrame(visual_experiment_set)\n",
    "\n",
    "# Get a DataFrame of records that describe projection signal for primary visual area experiments\n",
    "visual_unionizes_set = mcc.get_structure_unionizes(visual_experiment_set,\n",
    "                                               is_injection = False,\n",
    "                                                structure_ids = summary_structure_ids,\n",
    "                                               hemisphere_ids = [3])\n",
    "\n",
    "# Add a column to the DataFrame containing the structure names\n",
    "visual_names = [name_map[strid] for strid in visual_unionizes_set['structure_id']]\n",
    "visual_unionizes_set['structure_name'] = visual_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to show the Primary Visual Area data\n",
    "visual_unionizes_set.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 2.2: Medial Mammilary Nucleus DataFrame<a id='section_mmn'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "In the next cell, we want to obtain the projection densities for the different injections to the primary visual area that project to the medial mammillary nucleus. We do this in the first line and we name it <code>mmn</code> to differentiate from the variable <code>medial_mammillary</code>. Next, we arrange the projection densities from biggest to smallest. We will place these projection densities in a DataFrame that we can then use for bootstrapping. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "In the next cell, use the <code>sort_values</code> function to sort the values for the <code>\"projection_density\"</code> column from greatest to least and assign this table to <code>mmn_sorted</code>.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a DataFrame called mmn with primary visual area -- medial mammillary nucleus data\n",
    "mmn = visual_unionizes_set[visual_unionizes_set['structure_name'] == 'Medial mammillary nucleus']\n",
    "\n",
    "# TO-DO: replace the ellipses\n",
    "# use the sort_values function on the mmn DataFrame to sort it from greatest to least projection_density\n",
    "mmn_sorted = ...\n",
    "mmn_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "In the next cell, use your knowledge of indexing columns with <code>[...]</code> and the <code>.values</code> function to obtain an list with all the values for the projection densities. Set this equal to <code>mmn_values</code>.\n",
    "    \n",
    "<p><b>Hint:</b> Indexing just means we are selecting a specific column in a table, using its name (put it in quotes). For example, if you wanted to select column 'exam 2' from the table neu171L_grades, you would write <code>neu171L_grades['exam 2']</code>. The function <code>.values</code> comes at the end of the line.   \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TO-DO: replace the ellipses with code to select the projection_density column from the sorted mmn table we just \n",
    "# made and get the values in an array\n",
    "mmn_values = ...\n",
    "mmn_values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Run the cells below to create a DataFrame with the <code>mmn_values</code>. We will use this table for bootstrapping. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to make a DataFrame with the mmn values\n",
    "mmn_data = {'Medial mammillary nucleus': mmn_values}\n",
    "mmn_dataframe = pd.DataFrame(data=mmn_data)\n",
    "mmn_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 2.3: Bootstrapping<a id='section_bootstrapping'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "The essence of bootstrapping is reliant on a population. In this case, the population is going to be that of the neurons that project to the medial mammillary nucleus from the primary visual area. We obtained a sample, the 33 wildtype experiments from the primary visual area. \n",
    "\n",
    "Run the code below to sample with replacement from the population 5000 times using the mean as our test statistic. This will make sure that our number is big enough to be accurate and representative of the population.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "means = []\n",
    "# We want a lot of resamples, so we resample 5000 times\n",
    "for i in np.arange(5000):\n",
    "    resampled = mmn_dataframe.sample(n = 33, replace = True)\n",
    "    # We had 33 wildtype experiments, so n = 33. We set replace = True\n",
    "    boot_mean = resampled.mean()\n",
    "    means = np.append(means, boot_mean)\n",
    "# We end with an array (list) of all the means of the 5000 resamples\n",
    "means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "We will now place the resampled means into a DataFrame and graph them into the histogram below. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "mmn = {'Projection Density Means': means}\n",
    "mmn_means = pd.DataFrame(data=mmn).hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Now, let's overlay the mean projection density from the subiculum to the medial mammillary nucleus onto the histogram shown above.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "mmn = {'Projection Density Means': means}\n",
    "mmn_means = pd.DataFrame(data=mmn).hist()\n",
    "plt.axvline(mean_values, color = 'k');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "What can we see about the difference between the mean projection densities from primary visual area to the medial mammillary nucleus and the mean projection density from the subiculum to the medial mammillary nucleus?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "*Your Answer Here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### What does this mean in terms of connectivity?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "  \n",
    "Since the subiculum and the medial mammillary nucleus are a part of the Papez Circuit, it makes sense why the mean projection density is large. In contrast, the mean projection density for the primary visual cortex to the medial mammillary nucleus is low, making this structure not a part of the Papez Circuit (even though there is still minimal connectivity).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 2.4: 95% Confidence Interval<a id='section_ci'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "\n",
    "Using the bootstrapped sample means calculated above, we will create a **95% confidence interval** using the <code>quantile</code> function. This will give us an interval with the true mean of all projection densities from the primary visual area to the medial mammillary nucleus. \n",
    "\n",
    "Run the cell below to obtain the 95% confidence interval. Run the cell below that to overlay the 95% confidence interval over the projection density mean distribution.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "mmn = {'Projection Density Means': means}\n",
    "mmn_means = pd.DataFrame(data=mmn)\n",
    "percentile = list(mmn_means.quantile([0.025, 0.975])['Projection Density Means'])\n",
    "percentile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell\n",
    "mmn_means.hist()\n",
    "plt.hlines(y=0, xmin=percentile[0], xmax=percentile[1], linewidth=10, color = 'y');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Part 2.5: The P-Value<a id='section_pval'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Below, we will calculate the <b>p-value</b> using the above distribution and a random mean named <code>mystery_mean</code> that will simulate a mean projection density obtained from one of the injections into the primary visual area that projects to the medial mammillary nucleus. We are wondering if the <code>mystery_mean</code> came from the same distribution as the primary visual cortex projections to the medial mammillary nucleus. We must come up with a null hypothesis and an alternative hypothesis in order to use the p-value.\n",
    "\n",
    "Write down your <b>Null Hypothesis</b> and your <b>Alternative Hypothesis</b> below. Run the cells and find the p-value for further analysis. \n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Null hypothesis**: *Your Answer Here.* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Alternative hypothesis**: *Your Answer Here.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to define the mystery_mean and put the means in a DataFrame\n",
    "mystery_mean = 0.0434\n",
    "mmn = {'Projection Density Means': means}\n",
    "mmn_means = pd.DataFrame(data=mmn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to plot the means on a histogram\n",
    "# The mystery mean is shown as a horizontal black line\n",
    "mmn_means.hist()\n",
    "plt.axvline(mystery_mean, color = 'k');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to calculate the p value\n",
    "p_value = np.average(mmn_means > mystery_mean)\n",
    "p_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "Based on the p-value calculated above, do we reject or fail to reject the null hypothesis? Explain.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your Answer Here.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<div style=\"border-left: 3px solid #003262; padding: 1px; padding-left: 10px; background: #ffffff; \">\n",
    "    \n",
    "We hoped that this lab served as a good introductions to the tools and the statistical analysis used in neuroinformatics. Neuroinformatics is a growing field and every year, new techniques are being invented for collection of large neural information. Because of this, it is important to know how to analyze and manipulate this data. \n",
    "\n",
    "If you liked the lab and would like to further your data science techniques, below are some introductory courses that may interest you:\n",
    "\n",
    "[Data 8: The Foundations of Data Science](http://www.data8.org)\n",
    "\n",
    "[Data 100: Principles and Techniques of Data Science](http://ds100.org)\n",
    "\n",
    "[Prob 140: Probability for Data Science](http://prob140.org)\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting your work\n",
    "\n",
    "To submit this lab and receive credit for the assignment:\n",
    "\n",
    "<ol>\n",
    "    <li> First, <b>save your work</b> by clicking the floppy disk icon at the top left or by going to the \"File\" menu and clicking \"Save and Checkpoint\"</li>\n",
    "    <li>To download as a pdf, you can do any of the following:\n",
    "        <ol>\n",
    "            <li>Go to the \"File\" menu and click \"Download as\", then click \"PDF via LaTeX\" or \"PDF via Chrome\"</li>\n",
    "            <li>Click Ctrl+P (Windows) or Command+P (Mac) to print, then select the option to print to PDF</li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li> Check that your answers show up on your downloaded pdf. If you're not seeing your answers, make sure you saved before downloading, or try another pdf conversion method (different methods work for different browser settings)</li>\n",
    "    </ol>\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feedback\n",
    "\n",
    "This notebook was developed as a part of the [UC Berkeley Data Science Modules Program](https://data.berkeley.edu/education/modules). Please take five minutes to share your feedback on the notebook. We will use your feedback to improve future modules.\n",
    "\n",
    "Your answers or identity with anyone, including instructors, and will in no way affect the grades you receive in any class.\n",
    "\n",
    "### [Leave your feedback here](https://forms.gle/HkCPvRG4gP6Rgh6j9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<i>Notebook Developed by: Elias Saravia & Daniel Lopez</i>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}