Chapter4_ABTesting.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4e0a9a23",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "desktop=pd.read_csv('desktop.csv')\n",
    "laptop=pd.read_csv('laptop.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6820f45c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=-2.109853741030508, pvalue=0.03919630411621095)\n",
      "Ttest_indResult(statistic=-0.7101437106800108, pvalue=0.4804606394128761)\n",
      "Ttest_indResult(statistic=0.20626752311535543, pvalue=0.8373043059847984)\n"
     ]
    }
   ],
   "source": [
    "import scipy.stats\n",
    "print(scipy.stats.ttest_ind(desktop['spending'],laptop['spending']))\n",
    "print(scipy.stats.ttest_ind(desktop['age'],laptop['age']))\n",
    "print(scipy.stats.ttest_ind(desktop['visits'],laptop['visits']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2340223",
   "metadata": {},
   "source": [
    "## Running Experiments to Test New Hypotheses"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "679c2c16",
   "metadata": {},
   "source": [
    "Should we change the color of the text?\n",
    "\n",
    "Hypothesis 0  Changing the color of text in our emails from black to blue will have no effect on revenues.\n",
    "\n",
    "Hypothesis 1  Changing the color of text in our emails from black to blue will lead to a change in revenues (either an increase or a decrease)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "636b33b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "medianage=np.median(desktop['age'])\n",
    "groupa=desktop.loc[desktop['age']<=medianage,:]\n",
    "groupb=desktop.loc[desktop['age']>medianage,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "89c49ce3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   userid  revenue\n",
      "0       1      100\n",
      "1       2        0\n",
      "2       3       50\n",
      "3       4      550\n",
      "4       5      175\n"
     ]
    }
   ],
   "source": [
    "emailresults1=pd.read_csv('emailresults1.csv')\n",
    "\n",
    "print(emailresults1.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0d3a5e29",
   "metadata": {},
   "outputs": [],
   "source": [
    "groupa_withrevenue=groupa.merge(emailresults1,on='userid')\n",
    "groupb_withrevenue=groupb.merge(emailresults1,on='userid')\n",
    "# \"specify on='userid', meaning that we take the row of emailresults1 that corresponds to a particular userid and \n",
    "# merge it with the row of groupa that corresponds to that same userid”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a5ef9339",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=-2.186454851070545, pvalue=0.03730073920038287)\n"
     ]
    }
   ],
   "source": [
    "print(scipy.stats.ttest_ind(groupa_withrevenue['revenue'],groupb_withrevenue['revenue']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7775f4a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "125.0\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(groupb_withrevenue['revenue'])-np.mean(groupa_withrevenue['revenue']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e8e1a0",
   "metadata": {},
   "source": [
    "###### A/B Testing\n",
    "- (1) a split into two groups, application of a different treatment to each group\n",
    "- (2) statistical analysis to compare the groups' outcomes and draw conclusions about which treatment is better"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b9f1990",
   "metadata": {},
   "source": [
    "###### Understanding the Math of A/B Testing\n",
    "E(A's revenue with blk text) + E(effect of changing blk → blue on A) = E(A’s revenue with blue text)\n",
    "\n",
    "E(B’s revenue with blk text) + E(effect of changing blk → blue on B) = E(B’s revenue with blue text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8107da",
   "metadata": {},
   "source": [
    "Reject Hypothesis 0? \n",
    "\n",
    "(1) First calculate E(effect of changing blk → blue on A) and E(effect of changing blk → blue on B)\n",
    "\n",
    "(2) If either are not equal to 0, reject Hypothesis 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ddfeeb4",
   "metadata": {},
   "source": [
    "After experiment:\n",
    "    \n",
    "    E(A’s revenue with blk text) = 104\n",
    "    E(B’s revenue with blue text) = 229\n",
    "    \n",
    "    => 104 + E(effect of changing blk → blue on A) = E(A’s revenue with blue text)\n",
    "    => E(B’s revenue with blk text) + E(effect of changing blk → blue on B) = 229\n",
    "\n",
    "Simplified equation:\n",
    "    \n",
    "    104 + E(effect of changing blk → blue on everyone) = 229\n",
    " => Effect of blue text: $125 revenue increase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "99af3fcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Translating Math into Practice\n",
    "\n",
    "np.random.seed(18811015)\n",
    "laptop.loc[:,'groupassignment1']=1*(np.random.random(len(laptop.index))>0.5)\n",
    "groupc=laptop.loc[laptop['groupassignment1']==0,:].copy()\n",
    "groupd=laptop.loc[laptop['groupassignment1']==1,:].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a6d89167",
   "metadata": {},
   "outputs": [],
   "source": [
    "emailresults2=pd.read_csv('emailresults2.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "948fdd8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "groupc_withrevenue=groupc.merge(emailresults2,on='userid')\n",
    "groupd_withrevenue=groupd.merge(emailresults2,on='userid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8ff8f65c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=-2.381320497676198, pvalue=0.024288828555138562)\n"
     ]
    }
   ],
   "source": [
    "print(scipy.stats.ttest_ind(groupc_withrevenue['revenue'],groupd_withrevenue['revenue']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "36254b18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# including picture has a nonzero effect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c9c7d2f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "260.3333333333333\n"
     ]
    }
   ],
   "source": [
    "print(np.mean(groupd_withrevenue['revenue'])-np.mean(groupc_withrevenue['revenue']))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47dc9d6",
   "metadata": {},
   "source": [
    "## Optimizing with the Champion/Challenger Framework\n",
    "\n",
    "\"Since the new email is in direct competition with the champion email, we call it the challenger. If the champion performs better than the challenger, the champion retains its champion status. If the challenger performs better than the champion, that challenger becomes the new champion.\n",
    "\n",
    "This process can continue indefinitely: we have a champion that represents the state of the art of whatever we’re doing (marketing emails, in this case). We constantly test the champion by putting it in direct competition with a succession of challengers in A/B tests. Each challenger that leads to significantly better outcomes than the champion becomes the new champion and is, in turn, put into competition against new challengers later.\n",
    "\n",
    "This endless process is called the champion/challenger framework for A/B tests. It’s meant to lead to continuous improvement, continuous refinement, and asymptotic optimization to get to the best-possible performance in all aspects of business.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18624024",
   "metadata": {},
   "source": [
    "## Preventing Mistakes with Twyman's Law and A/A Testing\n",
    "\n",
    "“The inevitability of mistakes should lead us to be naturally suspicious of anything that seems too good, bad, interesting, or strange to be true. \n",
    "\n",
    "This natural suspicion is advocated by Twyman’s law, which states that “any figure that looks interesting or different is usually wrong.” \n",
    "\n",
    "This law has been restated in several ways, including “any statistic that appears interesting is almost certainly a mistake” and “the more unusual or interesting the data, the more likely it is to have been the result of an error.”\n",
    "\n",
    "“A/A testing is a type of testing is just what it sounds like; we go through the steps of randomization, treatment, and comparison of two groups just as in A/B testing, but instead of sending two different emails to our two randomized groups, we send the identical email to each group”\n",
    "\n",
    "“A/A testing can be a useful sanity check that can prevent us from getting carried away by the kind of unusual, interesting, too-good-to-be-true results that Twyman’s law warns us about.”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76871cc6",
   "metadata": {},
   "source": [
    "## Understanding Effect Sizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "0d19fff8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "158884197328.32672\n"
     ]
    }
   ],
   "source": [
    "# difference of $125 -> A/B test's effect size\n",
    "# is this a small effect, medium effect, or a large effect?\n",
    "\n",
    "# sonsider list of nominal GDPs for Malaysia, Myanmar, and Marshall Islands\n",
    "gdps=[365303000000,65994000000,220000000]\n",
    "print(np.std(gdps))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8e3daedd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.867365169217765e-10\n"
     ]
    }
   ],
   "source": [
    "print(125/np.std(gdps))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a74e94d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# this shows that the $125 effect size is a little more than 1 one billionth of the std of the GDP figures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4c471a29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6455394252341695\n"
     ]
    }
   ],
   "source": [
    "burgers=[9.0,12.99,10.50]\n",
    "print(np.std(burgers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c9d025c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "75.96293232671214\n"
     ]
    }
   ],
   "source": [
    "print(125/np.std(burgers))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6c360154",
   "metadata": {},
   "outputs": [],
   "source": [
    "# $125 is about 75.9 burger price standard deviations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd9b8e17",
   "metadata": {},
   "source": [
    "Cohen's d: an effect size divided by a relevant standard deviation\n",
    "- the number of standard deviations that two populations' means are apart from each other\n",
    "\n",
    "Cohen's d:\n",
    "- 0.2 or lower, small effect\n",
    "- about 0.5, medium effect\n",
    "- around 0.8 or higher, large effect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e7b39801",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.763769235188029\n"
     ]
    }
   ],
   "source": [
    "print(125/np.std(emailresults1['revenue']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "4cc474da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 0.76 ~ 0.8 => large effect size"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4787e80a",
   "metadata": {},
   "source": [
    "## Calculating the Significance of Data\n",
    "\n",
    "Mathematically, statistical significance depends on three things:\n",
    "- Size of the effect being studied; Bigger effects make statistical significance more likely.\n",
    "- Size of the sample being studied; Bigger samples make statistical significance more likely.\n",
    "- Significance threshold; A higher threshold makes statistical significance more likely."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db89dbf",
   "metadata": {},
   "source": [
    "stastical power: The probability that a correctly run A/B test will reject a false null hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "21ee3c69",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6501855020289931\n"
     ]
    }
   ],
   "source": [
    "from statsmodels.stats.power import TTestIndPower\n",
    "alpha=0.05\n",
    "nobs=45 # number of observations\n",
    "effectsize=0.5 #using Cohen's d\n",
    "\n",
    "analysis=TTestIndPower()\n",
    "power = analysis.solve_power(effect_size=effectsize, nobs1=nobs, alpha=alpha)\n",
    "print(power)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8b7c78d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 65 percent chance of detecting an effect from our A/B and about a 35 percent chance that even though a true effect \n",
    "# exists, our A/B test doesn't find it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "953a7d50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63.765611775409525\n"
     ]
    }
   ],
   "source": [
    "# solve_power() reverses power calculations\n",
    "analysis = TTestIndPower()\n",
    "alpha = 0.05\n",
    "effect = 0.5\n",
    "power = 0.8\n",
    "observations = analysis.solve_power(effect_size=effect, power=power, alpha=alpha)\n",
    "# calculates the nobs to achieve the power level\n",
    "print(observations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c1d64f19",
   "metadata": {},
   "outputs": [],
   "source": [
    "# “if we want to have 80 percent statistical power for our planned A/B test, we’ll need to recruit at least 64 participants for both groups”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02963674",
   "metadata": {},
   "source": [
    "## Applications and Advanced Considerations\n",
    "\n",
    "“By running an A/B test on pricing, you can measure what economists call the price elasticity of demand, meaning how much demand changes in response to price changes. \n",
    "- If your A/B test finds only a very small change in demand when you increase the price, you should increase the price for everyone and take advantage of their greater willingness to pay. \n",
    "- If your A/B test finds that demand drops off significantly when you increase the price slightly, you can conclude that customers are sensitive to price, and their purchase decisions depend heavily on price considerations.”\n",
    "\n",
    "“business-to-consumer (B2C) business models: businesses sell directly to consumers\"\n",
    "- \"B2C scenarios are a natural fit for A/B testing because the number of customers, products, and transactions tends to be higher for B2C businesses than for other businesses, so we can get large sample sizes and higher statistical power”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3938c899",
   "metadata": {},
   "source": [
    "###### “exploration/exploitation trade-off in A/B tests. \n",
    "In this trade-off, two goals are in constant tension: \n",
    "- to explore (for example, to run A/B tests with possibly bad email designs to learn which is best)\n",
    "    - Exploration can lead to missed opportunities if one of your challengers performs much worse than the champion; you would have been better off just sending out the champion to everyone.\n",
    "- to exploit (for example, to send out only the champion email because it seems to perform the best)\n",
    "    - Exploitation can lead to missed opportunities if your champion is not as good as another challenger that you haven’t tested yet because you’re too busy exploiting your champion to do the requisite exploration.”\n",
    "    \n",
    "multi-armed bandit problem: a mathematical formalization of the exploration/exploitation dilemma”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb113ee",
   "metadata": {},
   "source": [
    "## The Ethics of A/B Testing\n",
    "\n",
    "- Informed Consent\n",
    "- Risk\n",
    "    - Potential Downsides to participation as a human subject\n",
    "    - the probability of experiencing those downsides\n",
    "- Potential Benefits"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}