estimation.qmd

# Estimation

## Probabilistic models {.scrollable}

{{< include macros.qmd >}}

---

:::{#def-model}

##### Scientific models

**Scientific models** are attempts to describe *physical conditions or changes* that occur in the world and universe around us. 

:::

:::{#exm-model}

##### Scientific models in epidemiology {.scrollable}

Epidemiologists typically study *biological conditions and changes*, such as the spread of infectious diseases through populations, or the effects of environmental factors on individuals.

:::

### Statistical analysis of scientific models

When we perform statistical analyses, we use data to help us choose between models - specifically, to determine which models best explain that data.

However, physical processes do not produce data on their own. Data is only produced when scientists implement an *observation process* (i.e., a *scientific study*), which is distinct from the underlying *physical process*. In some cases, the observation process and the physical process interact with each other. This phenomenon is called the ["observer effect"](https://en.wikipedia.org/wiki/Observer_effect).

In order to learn about the physical processes we are ultimately interested in, we often need to make special considerations for the observation process that produced the data which we are analyzing. In particular, if some of the planned observations in the study design were not completed, we will likely need to account for the incompleteness of the resulting data set in our analysis. If we are not sure why some observations are incomplete, we may need to model the observation process in addition to the physical process we were originally interested in. For example, if some participants in a study dropped out part-way through the study, we may need investigate why those participants dropped out, as opposed to other participants who completed the study.

These kinds of *missing data* issues are outside of the scope of this course; see @van2018flexible for more details.

## Estimands, estimates, and estimators 

### Estimands

\index{estimand}

::: {#def-estimand}
##### Estimand

An **estimand** is an unknown quantity whose value we want to know [@pohl2021estimands; @lawrance2020estimand].
:::

::: {#exm-estimand}
##### Mean height of students

If we are trying to determine the mean height of students at our school,
then the *population mean* is our **estimand**.
:::

In statistical contexts, most estimands are parameters of probabilistic
models, or functions of model parameters.

::: callout-note
##### Notation for estimands

Model paramaters and other estimands are often symbolized using
lower-case Greek letters: $\alpha, \beta, \gamma, \delta$, etc.
:::

### Estimates

\index{estimate} \index{estimated value}

::: {#def-estimate}
##### Estimate/estimated value

In statistics, an **estimate** or **estimated value** is an informed
guess of an estimand's value, based on observed data.
:::

::: {#exm-estimate}
##### Mean height of students

Suppose we measure the heights of 50 random students from our school,
and the sample mean was 175cm. We might use 175cm as an *estimate* of
the population mean.
:::


### Estimators

\index{estimator}

::: {#def-estimator}
##### Estimator

An **estimator** is a function $\hth(x_1,...x_n)$ that transforms data $x_1,...x_n$ into an estimate. 

:::

::: callout-note
#### Estimators are random variables

When estimators are applied to random variables, the estimators are also random variables.
:::

::: callout-note
##### Notation for estimators

Estimators are often symbolized by placing a \^ ("hat") symbol on top of
the corresponding estimand; for example, $\hth$.

Usually, their dependence on the data is implicit:

$$\hth \eqdef \hth(x_1,...x_n)$$

:::

::: {#exm-estimator}
##### Mean height of students

If we want to estimate the mean height of students at our university,
which we will represent as $\mu$, we might measure the heights of
$n= 50$ randomly sampled students as random variables $X_1,...,X_n$.
Then we could use the function

$$\hat\mu(X_1,...,X_n) = \frac{1}{n} \sum_{i=1}^n X_i \eqdef \bar X$$

as an *estimator* to produce an *estimate* $\hat\mu = \bar x$ of $\mu$.

Another estimator would be just the height of the first student sampled:

$$\hat\mu^{(2)}(X_1,...,X_n) = X_1$$

A third possible estimator would be the mean of all sampled students'
heights, except for the two most extreme; that is, if we re-order the
observations $X_{(1)} = \min_{i\in 1:n} X_i$,
$X_{(2)} = \min_{i\in \{1:n\} - \arg X_{(1)}} X_i$, ...,
$X_{(n)} = \max_{i\in 1:n} X_i$, then we could define the estimator:

$$\hat\mu^{(3)}(X_1,...,X_n) = \frac{1}{n}\sum_{i=2}^{n-1} X_{(i)}$$

Which of these estimators is best? It depends on how we evaluate them
(see @sec-est-accuracy below).
:::

### Contrasting estimands, estimates, and estimators

It's helpful to keep in mind the mathematical type of each estimation
concept:

-   estimands are numbers (or vector of numbers)
-   estimates are also numbers (or vectors)
-   estimators are functions of random variables, so they are also random variables


## Accuracy of estimators {#sec-est-accuracy}

### Accuracy

To determine which estimator is best, we need to define *best*.
"Accuracy" is usually most important; easy computation is usually
secondary.

::: {#def-accuracy}
##### Accuracy

The **accuracy** of an estimator for a given estimand does not have a
consensus formal definition, but all of the usual candidates are related
to the distributions of the *errors* made by the resulting estimates.

:::

### Error

::: {#def-error}
##### Error

The **error** of an estimate $\hth$ of a true value $\th$, often denoted $\eps(\hth)$, or more completely $\eps(\hth, \th)$, is the
difference between the estimate and its estimand $\theta$; that is:

$$\eps(\hth) \eqdef \hth - \th$$
:::

Some frequently-used measures of accuracy include:

### Mean squared error

::: {#def-mse}
##### Mean squared error

The **mean squared error** of an estimator $\hth$, denoted
$\mselr{\hth}$, is the expectation of the square of the error[^1]:

$$\mselr{\hth} \eqdef \E{(\err(\hth))^2}$$
:::

### Mean absolute error

::: {#def-mae}
##### Mean absolute error

The **mean absolute error** of an estimator is the expectation of the
absolute value of the error:

$$
\maelr{\hth} \eqdef \E{\abs{\err(\hth)}}
$$
:::

### Bias

::: {#def-bias}
##### Bias

The **bias** of an estimator $\hth$ for an estimand $\theta$ is the
expected value of the error:

$$\bias{\hth} \eqdef \E{\err(\hth)}$$ {#eq-bias-def}
:::

---

::: {#thm-bias-exprs}
##### Bias equals Expectation minus Truth

$$\bias{\hth} =\E{\hth} - \theta$$

:::

:::: proof

$$
\begin{aligned}
\bias{\hth} 
&\eqdef \E{\err(\hth)}\\
&= \E{\hth - \theta}\\
&=\E{\hth} - \E{\theta}\\
&=\E{\hth} - \theta
\end{aligned}
$$

The third equality is by the linearity of expectation.

::::


[^1]: <https://en.wikipedia.org/wiki/Does_exactly_what_it_says_on_the_tin>

---

::: {#thm-mse-bias-variance}
##### Mean Squared Error equals Bias Squared plus Variance

For any one-dimensional estimator $\hth$:

$$\mselr{\hth} = \sqf{\bias{\hth}} + \var{\hth}$$ {#eq-mse}

:::

:::: proof
{{< include proof-mse-bias-variance.qmd >}}
::::

----

#### Unbiased estimators {#sec-unbiased-estimators}

::: {#def-unbiased}
##### unbiased estimator

An estimator $\hth$ is **unbiased** if $\bias{\hth} = 0$.
:::

::: {#thm-unbiased-props}
##### properties of unbiased estimators

If $\hth$ is unbiased, then:

$$\E{\hth} = \th$$ {#eq-unbiased-exp}
$$\mselr{\hth} = \var{\hth}$$ {#eq-unbiased-mse}

:::

:::: proof  
  
If $\hth$ is unbiased, then:

@eq-unbiased-exp: 

$$
\ba
\bias{\hth} &= 0\\
\E{\hth} - \theta &= 0\\
\E{\hth} &= \theta
\ea
$$

@eq-unbiased-mse:

$$
\ba
\mselr{\hth} 
&\eqdef \E{\sqf{\eps(\hth)}}\\
&= \E{\sqf{\hth - \th}}\\
&= \E{\sqf{\hth - \E{\hth}}}\\
&\eqdef \Var{\hth}
\ea
$$

(*Alternative proof of @eq-unbiased-mse*) We could have started from
@thm-mse-bias-variance instead:

$$
\ba
\mselr{\hth} 
&= \sqf{\bias{\hth}} + \var{\hth}\\
&= \sqf{0} + \var{\hth}\\
&= 0 + \var{\hth}\\
&= \var{\hth}\\
\ea
$$

::::

---

### Standard error

:::{#def-SE}

##### Standard error

The **standard error** of an estimator $\hth$ is just the [standard deviation](probability.qmd#def-sd) of $\hth$; that is:

$$\SE{\hth} \eqdef \SD{\hth}$$

:::

"Standard error" is a confusing concept in a few ways. First of all, it isn't even defined as a characteristic of the [error](#def-error), $\eps(\hth)$! Moreover, it is just a synonym for standard deviation, so it seems like a redundant concept. However, standard errors help us construct p-values and confidence intervals, so they come up a lot - often enough to give them their own name.

We can relate standard error to actual error, by rearranging the result from @thm-mse-bias-variance:

$$
\ba
\Var{\hth} &= \Var{\hth - \th}\\
&= \Var{\eps(\hth)}\\
\ea
$$
So the variance of the estimator is equal to the variance of the error, and the standard error is equal to the standard deviation of the error:

$$\SE{\hth} = \SD{\eps(\hth)}$$

---

:::{#cor-var-mse-bias}

##### Standard error squared equals MSE minus squared bias
  
standard error is what is left over of MSE after bias is removed:

$$\sqf{\SE{\hth}} = \mselr{\hth} - \sqf{\bias{\hth}}$$

:::

::::{.proof}

$$
\ba
\mselr{\hth} &= \sqf{\bias{\hth}} + \Var{\hth}\\
\tf \Var{\hth} &= \mselr{\hth} - \sqf{\bias{\hth}}\\
\tf \sqf{\SE{\hth}} &= \mselr{\hth} - \sqf{\bias{\hth}}\\
\ea
$$

::::


:::{#cor-se-rmse-unbiased}
##### For unbiased estimators, SE = RMSE

If $\E{\erf{\hth}} = 0$, then:

$$\SE{\hth} = \sqrt{\mselr{\hth}}$$

(this result is equivalent to @eq-unbiased-mse)
:::

---