notebooks/Diet-Related-Disease-Exploratory.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing-Food-Insecurity-with-Pixie-Dust-and-Watson-Analytics\n",
    "_IBM Journey showing how to visualize US Food Insecurity with Pixie Dust and Watson Analytics._\n",
    "\n",
    "Often in data science we do a great deal of work to glean insights that have an impact on society or a subset of it and yet, often, we end up not communicating our findings or communicating them ineffectively to non data science audiences. That's where visualizations become the most powerful. By visualizing our insights and predictions, we, as data scientists and data lovers, can make a real impact and educate those around us that might not have had the same opportunity to work on a project of the same subject. By visualizing our findings and those insights that have the most power to do social good, we can bring awareness and maybe even change. This journey walks you through how to do just that, with IBM's Watson Studio, Pandas, Pixie Dust and Watson Analytics.\n",
    "\n",
    "For this particular journey, food insecurity throughout the US is focused on. Low access, diet-related diseases, race, poverty, geography and other factors are considered by using open government data. For some context, this problem is a more and more relevant problem for the United States as obesity and diabetes rise and two out of three adult Americans are considered obese, one third of American minors are considered obsese, nearly ten percent of Americans have diabetes and nearly fifty percent of the African American population have heart disease. Even more, cardiovascular disease is the leading global cause of death, accounting for 17.3 million deaths per year, and rising. Native American populations more often than not do not have grocery stores on their reservation... and all of these trends are on the rise. The problem lies not only in low access to fresh produce, but food culture, low education on healthy eating as well as racial and income inequality.\n",
    "\n",
    "The government data that I use in this journey has been conveniently combined into a dataset for our use, which you can find in this repo under combined_data.csv. You can find the original, government data from the US Bureau of Labor Statistics https://www.bls.gov/cex/ and The United States Department of Agriculture https://www.ers.usda.gov/data-products/food-environment-atlas/data-access-and-documentation-downloads/."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is Watson Studio, Pixie Dust and Watson Analytics and why should I care enough about them to use them for my visualizations?\n",
    "\n",
    "IBM's Watson Studio, is an online browser platform where you can use notebooks or R Studio for your data science projects. Watson Studio is unique in that it automatically starts up a Spark instance for you, allowing you to work in the cloud without any extra work. Watson Studio also has open data available to you, which you can connect to your notebook. There are also other projects available, in the form of notebooks, which you can follow along with and apply to your own use case. Watson Studio also lets you save your work, share it and collaborate with others, much like I'm doing now!\n",
    "\n",
    "Pixie Dust is a visualization library you can use on Watson Studio. It is already installed into Watson Studio and once it's imported, it only requires one line of code (two words) to use. With that same line of code, you can pick and choose different values to showcase and visualize in whichever way you want from matplotlib, seaborn and bokeh. If you have geographic data, you can also connect to google maps and Mapbox, depending on your preference. Check out a tutorial on Pixie Dust here: https://ibm-watson-data-lab.github.io/pixiedust/displayapi.html#introduction\n",
    "\n",
    "IBM's Watson Analytics is another browser platform which allows you to input your data, conduct analysis on it and then visualize your findings. If you're new to data science, Watson recommends connections and visualizations with the data it has been given. These visualizations range from bar and scatter plots to predictive spirals, decision trees, heatmaps, trend lines and more. The Watson platform then allows you to share your findings and visualizations with others, completing your pipeline. Check out my visualizations with the link further down in the notebook, or in the images in this repo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's start with Watson Studio.\n",
    "\n",
    "Here's a tutorial on getting started with Watson Studio: https://datascience.ibm.com/docs/content/analyze-data/creating-notebooks.html.\n",
    "\n",
    "To summarize the introduction, you must first make an account and log in. Then, you can create a project (I titled mine: \"Diet-Related Disease\"). From there, you'll be able to add data and start a notebook. To begin, I used the combined_data.csv as my data asset. You'll want to upload it as a data asset and once that is complete, go into your notebook in the edit mode (click on the pencil icon next to your notebook on the dashboard). To load your data in your notebook, you'll click on the \"1001\" data icon in the top right. The combined_data.csv should show up. Click on it and select \"Insert Pandas Data Frame\". Once you do that, a whole bunch of code will show up in your first cell. Once you see that, run the cell and follow along with my tutorial!\n",
    "\n",
    "_Quick Note: In Github you can view all of the visualizations by selecting the circle with the dash in the middle at the top right of the notebook!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import data and libraries (do this by using the 1001 button above to the right)\n",
    "\n",
    "from io import StringIO\n",
    "import requests\n",
    "import json\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Insert your data as a Pandas DataFrame\n",
    "\n",
    "* Click on the cell below to highlight it.\n",
    "* In the upper right of this notebook, click on `01/00` to open the Data tab, where you should see the `combined_data.csv` that you loaded earlier.\n",
    "* Click on `Insert to code` -> `Insert Pandas DataFrame`\n",
    "* Make sure that the data is called `df_data_1` and change it if it is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# @hidden_cell\n",
    "# This function accesses a file in your Object Storage. The definition contains your credentials.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning data and Exploring\n",
    "\n",
    "This notebook starts out as a typical data science pipeline: exploring what our data looks like and cleaning the data. Though this is often considered the boring part of the job, it is extremely important. Without clean data, our insights and visualizations could be inaccurate or unclear. \n",
    "\n",
    "To initially explore, I used matplotlib to see a correlation matrix of our original data. I also looked at some basic statistics to get a feel for what kind of data we are looking at. I also went ahead and plotted using pandas and seaborn to make bar plots, scatterplots and regression plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data_1.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data_1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#to see columns distinctly and evaluate their state\n",
    "df_data_1['PCT_LACCESS_POP10'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data_1['PCT_REDUCED_LUNCH10'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data_1['PCT_DIABETES_ADULTS10'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_data_1['FOODINSEC_10_12'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#looking at correlation in a table format\n",
    "df_data_1.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#checking out a correlation matrix with matplotlib\n",
    "plt.matshow(df_data_1.corr())\n",
    "#we notice that there is a great deal of variables which makes it hard to read!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#other stats\n",
    "df_data_1.max()\n",
    "df_data_1.min()\n",
    "df_data_1.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot counts of a specified column using Pandas\n",
    "df_data_1.FOODINSEC_10_12.value_counts().plot(kind='barh')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Bar plot example\n",
    "sns.factorplot(\"PCT_SNAP09\", \"PCT_OBESE_ADULTS10\", data=df_data_1,size=3,aspect=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Regression plot\n",
    "sns.regplot(\"FOODINSEC_10_12\", \"PCT_OBESE_ADULTS10\", data=df_data_1, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After looking at the data I realize that I'm only interested in seeing the connection between certain values and because the dataset is so large it's bringing in irrelevant information and creating noise. To change this, I created a smaller data frame, making sure to remove NaN and 0 values (0s in this dataset generally mean that a number was not recorded)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a dataframe of values that are most interesting to food insecurity\n",
    "df_focusedvalues = df_data_1[[\"State\", \"County\",\"PCT_REDUCED_LUNCH10\", \"PCT_DIABETES_ADULTS10\", \"PCT_OBESE_ADULTS10\", \"FOODINSEC_10_12\", \"PCT_OBESE_CHILD11\", \"PCT_LACCESS_POP10\", \"PCT_LACCESS_CHILD10\", \"PCT_LACCESS_SENIORS10\", \"SNAP_PART_RATE10\", \"PCT_LOCLFARM07\", \"FMRKT13\", \"PCT_FMRKT_SNAP13\", \"PCT_FMRKT_WIC13\", \"FMRKT_FRVEG13\", \"PCT_FRMKT_FRVEG13\", \"PCT_FRMKT_ANMLPROD13\", \"FOODHUB12\", \"FARM_TO_SCHOOL\", \"SODATAX_STORES11\", \"State_y\", \"GROC12\", \"SNAPS12\", \"WICS12\", \"PCT_NHWHITE10\", \"PCT_NHBLACK10\", \"PCT_HISP10\", \"PCT_NHASIAN10\", \"PCT_65OLDER10\", \"PCT_18YOUNGER10\", \"POVRATE10\", \"CHILDPOVRATE10\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove NaNs and 0s\n",
    "df_focusedvalues = df_focusedvalues[(df_focusedvalues != 0).all(1)]\n",
    "df_focusedvalues = df_focusedvalues.dropna(how='any')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before visualizing, a quick heatmap is created so that we can see what correlations we may want to visualize. I visualized a few of these relationships using seaborn, but I ultimately want to try out other visualizations. The quickest way to explore these is through Pixie Dust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#look at heatmap of correlations with the dataframe to see what we should visualize\n",
    "corr = df_focusedvalues.corr()\n",
    "sns.heatmap(corr, \n",
    "            xticklabels=corr.columns.values,\n",
    "            yticklabels=corr.columns.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can immediately see that a fair amount of strong correlations and relationships exist. Some of these include 18 and younger and Hispanic, an inverse relationship between Asian and obese, a correlation between sodatax and Hispanic, African American and obesity as well as food insecurity, sodatax and obese minors, farmers markets and aid such as WIC and SNAP, obese minors and reduced lunches and a few more.\n",
    "\n",
    "Let's try and plot some of these relationships with seaborn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Percent of the population that is white vs SNAP aid participation (positive relationship)\n",
    "sns.regplot(\"PCT_NHWHITE10\", \"SNAP_PART_RATE10\", data=df_focusedvalues, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Percent of the population that is Hispanic vs SNAP aid participation (negative relationship)\n",
    "sns.regplot(\"SNAP_PART_RATE10\", \"PCT_HISP10\", data=df_focusedvalues, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Eligibility and use of reduced lunches in schools vs percent of the population that is Hispanic (positive relationship)\n",
    "sns.regplot(\"PCT_REDUCED_LUNCH10\", \"PCT_HISP10\", data=df_focusedvalues, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Percent of the population that is black vs percent of the population with diabetes (positive relationship)\n",
    "sns.regplot(\"PCT_NHBLACK10\", \"PCT_DIABETES_ADULTS10\", data=df_focusedvalues, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Percent of population with diabetes vs percent of population with obesity (positive relationship)\n",
    "sns.regplot(\"PCT_DIABETES_ADULTS10\", \"PCT_OBESE_ADULTS10\", data=df_focusedvalues, robust=True, ci=95, color=\"seagreen\")\n",
    "sns.despine();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these simple regression plots we were able to glean from our data information such as in 2010, non-hispanic whites were highly correlated with the use of the SNAP program, or food stamps. We see that the hispanic population is not highly correlated in this time frame. This could be for a variety of reasons including eligibility, reporting, varying policies and use of the program. In our next graphs we see that in 2010, the percentage of the population who were black were highly correlated with diabetes. Next, we see that diabetes and obesity are highly correlated. These graphs do not represent any statistical significance, but they can help us understand and familiarize ourselves with the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now, let's visualize with Pixie Dust.\n",
    "\n",
    "Now that we've gained some initial insights, let's try out a different tool: Pixie Dust!\n",
    "\n",
    "As you can see in the notebook below, to activate Pixie Dust, we just import it and then write:\n",
    "\n",
    " ```display(your_dataframe_name)```\n",
    " \n",
    "After doing this your dataframe will show up in a column-row table format. To visualize your data, you can click the chart icon at the top left (looks like an arrow going up). From there you can choose from a variety of visuals. Once you select the type of chart you want, you can then select the variables you want to showcase. It's worth playing around with this to see how you can create the most effective visualizations for your audience. The notebook below showcases a couple options such as scatterplots, bar charts, line charts, and histograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pixiedust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user --upgrade pixiedust"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "handlerId": "pieChart",
      "keyFields": "State",
      "title": "Food Hub by State",
      "valueFields": "FOODHUB12"
     }
    }
   },
   "outputs": [],
   "source": [
    "#looking at the dataframe table. Pixie Dust does this automatically, but to find it again you can click the table icon.\n",
    "#Just to give some examples of what you can do with the data, I've created a pie chart of percent of food hubs in the country by state.\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "handlerId": "scatterPlot",
      "histoChartType": "subplots",
      "keyFields": "FOODINSEC_10_12",
      "kind": "reg",
      "lineChartType": "subplots",
      "mpld3": "false",
      "rendererId": "seaborn",
      "rowCount": "500",
      "title": "Food Insecurity vs Percent of the population that is black",
      "valueFields": "PCT_LACCESS_CHILD10"
     }
    }
   },
   "outputs": [],
   "source": [
    "#using seaborn in Pixie Dust to look at Food Insecurity and the Percent of the population that is black in a scatter plot\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "handlerId": "mapView",
      "keyFields": "State",
      "legend": "false",
      "mapboxtoken": "pk.eyJ1IjoibWFkaXNvbmpteWVycyIsImEiOiJjajk0aWliYnY0Nm5tMnhxemZzdTZqNTJ5In0.GRE2BIO7LYMSygKLhDy18Q",
      "mpld3": "false",
      "orientation": "vertical",
      "rendererId": "google",
      "rowCount": "500",
      "title": "Food Insecurity by State",
      "valueFields": "PCT_OBESE_ADULTS10"
     }
    }
   },
   "outputs": [],
   "source": [
    "#using matplotlib in Pixie Dust to view Food Insecurity by state in a bar chart\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "handlerId": "lineChart",
      "keyFields": "PCT_OBESE_ADULTS10",
      "rendererId": "bokeh",
      "rowCount": "500",
      "title": "Percent of Population that is Black vs Percent of Population that is Obese",
      "valueFields": "PCT_NHBLACK10"
     }
    }
   },
   "outputs": [],
   "source": [
    "#using bokeh in Pixie Dust to view the percent of the population that is black vs the percent of the population that is obese in a line chart\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "handlerId": "scatterPlot",
      "keyFields": "PCT_DIABETES_ADULTS10",
      "kind": "kde",
      "rendererId": "seaborn",
      "rowCount": "500",
      "title": "Obesity vs Diabetes",
      "valueFields": "PCT_OBESE_ADULTS10"
     }
    }
   },
   "outputs": [],
   "source": [
    "#using seaborn in Pixie Dust to view obesity vs diabetes in a scatterplot\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pixiedust": {
     "displayParams": {
      "aggregation": "SUM",
      "chartsize": "80",
      "clusterby": "CHILDPOVRATE10",
      "handlerId": "scatterPlot",
      "keyFields": "PCT_OBESE_CHILD11",
      "lineChartType": "subplots",
      "rendererId": "bokeh",
      "rowCount": "1000",
      "title": "Childhood Obesity vs Reduced Lunches",
      "valueFields": "PCT_REDUCED_LUNCH10"
     }
    }
   },
   "outputs": [],
   "source": [
    "#using matplotlib in Pixie Dust to view childhood obesity vs reduced school lunches in a scatterplot\n",
    "display(df_focusedvalues)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Watson Studio to visualize our insights.\n",
    "\n",
    "You can see that with these visualizations the user can see the impact of food insecurity by state, geographically distributed and used aid such as reduced school lunches, a map of diabetes by state, a predictive model for food insecurity and diabetes (showcasing the factors that, in combination, suggest a likelihood of food insecurity), drivers of adult diabetes, drivers of food insecurity, the relationship with the frequency of farmers market locations, food insecurity and adult obesity, as well as the relationship between farmers markets, the percent of the population that is Asian, food insecurity and poverty rates.\n",
    "\n",
    "By reviewing our visualizations in Watson Studio, we learn that obesity and diabetes almost go hand in hand, along with food insecurity. We can also learn that this seems to be an inequality issue, both in income and race, with Black and Hispanic populations being more heavily impacted by food insecurity and diet-related diseases than those of the White and Asian populations. We can also see that school-aged children who qualify for reduced lunch are more likely obese than not whereas those that have a farm-to-school program are more unlikely to be obese.\n",
    "\n",
    "Like many data science investigations, this analysis could have a big impact on policy and people's approach to food insecurity in the U.S. What's best is that we can create many projects much like this in a quick time period and share them with others by using Pandas, Pixie Dust as well as Watson's predictive and recommended visualizations."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}