What is Statistics

Statistics gives meaning to data, or better, it is the science of drawing conclusions from data. It is used in a great many parts of life:

1. Predicting. Using what has already happened to say something about what has not: elections, weather, the price of an asset.
2. Contextualising. Placing a number beside the numbers that make it meaningful, which is most of what economics, politics and business reporting consists of.
3. Gathering. Designing the collection of new data so that it can support a prediction at all. Medicine, marketing, finance and the social sciences all depend on experimental design, and without well-gathered data the other two uses are worth little.

Scrutinising a Claim

Consider a study of microplastics in bubble tea sold in the Bay Area, which reported dangerously high levels of BPA and microplastics in the drinks.

Faced with a result like this we already have ideas about where the plastic came from. Perhaps it is the straws. Perhaps it is the containers the drink is served in. Whatever the guess, it has to be scrutinised, because one possibility is always that the effect arose by chance rather than from the cause we have in mind.

In this case the tap water used to prepare the drinks was itself already carrying a high concentration of microplastics, and tap water is used at every stage, from boiling the tapioca to making the syrup. Once that is known, the cup and the straw stop looking like the main source: the drink was made from contaminated water before it ever met a container.

Gathering New Data

How does one gather new information to answer a hard question? In medicine two designs are standard.

1. In an observational study the researchers watch what happens without intervening. Its aim is to find a cause and its effect as they occur naturally.
2. In a randomised double-blind trial the researchers assign the treatment themselves, at random, and neither the patient nor the researcher knows who is receiving the drug and who the placebo. Its aim is to remove bias.

The difference in aim matters. An observational study can tell us what happens in the world as it is; a randomised trial can tell us what the treatment does, because randomisation removes the link between receiving the treatment and everything else about the patient.

A third design, familiar outside medicine, is A/B testing: users are assigned at random to group A or group B, each group is shown a different version of something, and the versions are compared on whatever response we care about. Advertising a new phone one way or another way is decided this way.

Prediction

The third use is prediction. Given the data available to us, we want the likely outcome: how risky a stock is, what it is likely to be worth, how much of a portfolio to put in it. Much of the rest of this course is about doing that carefully.

How Statistics Goes Wrong

A statistic can be true and still mislead, if it answers a different question from the one being asked.

The second failure is one of procedure rather than arithmetic.

The order matters. Form the theory, then collect data to test it. Collecting data first and looking for patterns afterwards risks finding a pattern that is not there.

Numbers in Context

A number on its own is not informative. It becomes informative when we compare it with a baseline, and choosing the baseline is the hard part.

Is $10 billion a lot of money? As a person’s net worth, yes. As a country’s annual output, no.

To talk about a number at all we put it through three filters.

1. What kind of number is it? Is it a rate, a percentage, a nominal amount, an average? How was it calculated, and who is reporting it?
2. What can it be compared with? Is there a natural baseline or denominator? Can it be compared with the past? What is missing that the comparison would need?
3. Is it plausible? Is it surprising, or is it about what we should have expected?

Ball-park Estimates

Most of the time the number we want for the baseline is not to hand. The fix is to estimate it: split the quantity into smaller pieces we can each guess at, guess each piece, and multiply. Working to the nearest power of ten keeps the arithmetic easy and is usually accurate enough to settle the question.

Cost-Benefit Analysis

Ball-park estimates are most useful when a decision depends on them. Faced with a choice between $X$ and $Y$:

1. estimate the cost of each;
2. estimate the benefit of each;
3. choose the more attractive;
4. if the stakes are high, ask how robust the analysis is.

The first two steps are Fermi problems. The fourth asks which inputs we are least sure of, and whether the conclusion survives varying them.

Exploratory Data Analysis

Much of what probability is for is saying something about a whole group when we have only seen part of it.

Returns

Take a financial example. The monthly equity returns for Canada were recorded over $107$ months, from February 1988 to December 1996, one number per month.

 0.07  0.05  0.02 -0.04  0.08 -0.02 -0.05  0.02  0.03
 0.00  0.03  0.08 -0.03  0.01  0.03  0.01  0.02  0.08
 0.02 -0.02  0.00  0.01  0.02 -0.09  0.00  0.01 -0.07
 0.07  0.00  0.02 -0.05 -0.04 -0.03  0.03  0.04  0.00
 0.07  0.00  0.01  0.04 -0.02  0.02  0.01 -0.03  0.05
-0.02  0.00  0.01 -0.01 -0.05 -0.01  0.01  0.00  0.02
-0.02 -0.07  0.03 -0.04  0.03 -0.02  0.06  0.03  0.04
 0.01 -0.01 -0.01  0.01 -0.05  0.09 -0.02  0.05  0.06
-0.05 -0.04 -0.01  0.01 -0.06  0.05  0.06  0.02 -0.01
-0.06  0.02 -0.05  0.06  0.04  0.02  0.04  0.02  0.02
 0.00  0.00 -0.01  0.04  0.01  0.05 -0.01  0.02  0.04
 0.02 -0.03 -0.03  0.05  0.04  0.08  0.07 -0.03

A first question to ask of a column of numbers like this is where the middle is.

Scanning it, the values look as though they tend positive, and that is what we should expect. An asset whose returns averaged zero would go nowhere over the nine years, and one whose returns averaged negative would end the period worth less than it started, which is not what these equities did. So the middle is somewhere above zero. Reading a column one entry at a time is a poor way to locate it, though, and the first thing to do with data like this is draw it.

Histograms

To see a column of numbers we plot it. The values are sorted into bins, and the number of values falling in each bin is drawn as a bar.

Reading it: the centre sits somewhere between $0$ and $2\%$, the shape is a single mound with the two tails dying away at roughly the same rate, and nothing sits far out on its own. The most common outcome is a small positive month.

Not every set of data makes a symmetric mound. A distribution is skewed to the left when its left tail is drawn out, which puts the bulk of the values on the right, and skewed to the right when the right tail is the long one.

The spread of the values about the centre is the variation. In the Canadian returns it could come from almost anything: credit conditions, commodity prices, a change of policy, a single piece of news. Naming the sources is not the same as measuring the spread, and measuring it is what we do later.

Data, Units and Variables

A histogram is not the only way to draw a distribution. A dot plot puts one dot per observation above its value, stacking them where values repeat, which keeps every individual observation visible in a way binning does not.

For the example above, the observational units are the individual months, the variable is the return $r$, which is quantitative, and Figures 1.1 and 1.3 are two pictures of its distribution.

Comparing Distributions

Put a second country beside the first. Japanese equity returns were recorded over the same $107$ months, and here is each country drawn on the axes that suit it.

Set side by side like this the two pictures say almost nothing. Both show a mound near zero of roughly the same width on the page, so the eye reports that the two countries behave alike. They do not: the horizontal axes differ by a factor of two, and the bins differ with them, so equal-looking widths on the page stand for quite different widths in returns. The vertical axes can play the same trick.

Always check the axes before comparing two histograms, and if they differ, redraw. Here are the two on one scale.

Read together, the two show something a single number would not. The Canadian returns run from $-9\%$ to $+9\%$; the Japanese run from $-19\%$ to $+24\%$. Japan’s tails reach much further in both directions, so it offers both the larger gains and the larger losses, while Canada clusters more tightly about its centre.

Outliers

An outlier is a value lying far from the bulk of the data. Two cohorts of sixty students were each asked how many hours a week they spend on problem sheets; here are the answers, on one pair of axes. The figures are made up for the purpose, so that the shape being discussed is unmistakable.

The lone bar near $56$ hours in cohort B is an outlier. Everything else in that cohort lies below $25$, so a single student sits more than twice as far out as anyone around them.

Comparing the two, the general claim is available: cohort A works more, because its centre sits at larger values, near $22$ hours against cohort B’s $14$. The outlier does not disturb that claim, and it does move a summary. Cohort B’s mean is $15.0$ hours with the outlier and $14.3$ without it, so one student out of sixty shifts the average of the whole cohort by nearly three quarters of an hour.

An outlier is not automatically an error to be removed. It may be a mistake in recording, in which case it should go; it may be a student who genuinely works that way, in which case it is the most informative observation in the set. Deciding which requires knowing where the number came from, and that is a question about the data collection rather than about the histogram. The same applies to Japan’s two months below $-16\%$ in Figure 1.7: they sit clear of the mound, and any summary of the Japanese returns which throws them away will understate how badly a month can go.

Time Series

The returns above were collected over time, and they came in an order. A histogram throws that order away.

The vertical axis carries the return and the horizontal axis the time, so each point is one month and the line joins them in the order they occurred.

It is tempting to read this plot as returning towards zero after each excursion, or as being about to turn after a run in one direction. Neither is visible here. The series is close to what repeated coin tossing looks like when it is plotted, and one of the things this course does is give us the means to say what “close to” means, rather than deciding by eye.

Not every series is like that.

Source: monthly beer production in Australia, $476$ monthly observations running from January 1956 to August 1995. The figure shows the last four and a half years.

The pattern repeats once a year, and knowing it changes the question to ask. The interesting thing about any single month is no longer how large it is, but how large it is beside the same month in other years, since comparing a peak month with a trough month tells us about the calendar rather than about the brewing.

Measures of Central Tendency

Suppose we collect $n$ pieces of data. We need some way of describing them, and we write them as

Here $x_1$ is the first number and $x_n$ the last, and $n$ is the number of numbers, the number of observations, also called the sample size. These are the values we observe. The letter $x$ is just a name for the set of numbers; we could just as easily use $y$. If the data are $5, 2, 8, 6, 2$, for example, then $n = 5$, $x_1 = 5$ and $x_3 = 8$.

Sometimes the order of the observations means something. In the monthly returns of the last lesson the first observation corresponds to the first month. Sometimes it does not: in a survey each number corresponds to one respondent, and the list could be written in any order.

The Sample Mean

There is a more compact way to write the sum $x_1 + x_2 + \cdots + x_n$, called summation notation:

It means that for each value of $i$ from $1$ to $n$ we add to the sum the value indicated, in this case $x_i$. Using it,

For the data $5, 2, 8, 6, 2$ the mean is $\bar{x} = \dfrac{5 + 2 + 8 + 6 + 2}{5} = \dfrac{23}{5} = 4.6$.

We think of the mean as the centre of the data. In a class survey of how many beers students claim they can drink, one answer was deleted because nobody believed it. Among the rest, the women claimed on average that they could drink $4.2$ beers and the men $7.9$, so in this sense the men claim to drink more.

For the equity returns of the last lesson the means are

This is a big difference: about $0.9\%$ a month against about $0.2\%$. It was hard to see in the histograms, because it is small compared with the variation of the returns.

The Median

So the median of $1, 2, 3, 4, 5$ is $3$, and the median of $1, 1, 2, 3, 4, 5$ is $\dfrac{2 + 3}{2} = 2.5$.

Mean versus Median

The mean and the median are both good measures of the centre of a distribution, but the median is much less sensitive to extreme values. It depends only on which observations sit in the middle of the ordering, so making the largest value larger, or the smallest smaller, leaves it unchanged.

Extreme values like the $100$ are the outliers of the last lesson. We used to call them funny points, but outliers sounds more scientific. Outliers are sometimes the most interesting aspect of a data set, and sometimes they are just coding errors.

The Mode

People also use the word more loosely, calling any lump in a histogram a mode.

Measures of Dispersion

The mean and the median locate the centre of a set of observations, for different notions of centre. They do not measure its dispersion, or spread. Are most of the observations close to the centre, or not? What is the typical range the data fall into? The data sets $5, 5, 5, 5, 5$ and $1, 3, 5, 8, 8$ both have mean $5$, but the second is clearly the more variable.

Often we care about the variability as well as the centre.

1. Medicine: we may know the average life expectancy given a diagnosis, but what are the best and the worst cases?
2. Exams: we may know the class average and our own score, but how do we really compare with the rest of the class?
3. Investments: we may know the historical average annual return of a stock or a mutual fund, but is a big loss possible? Beside the average return we need a measure of risk.

The Variance

The basic idea is to view variability in terms of the distance between each observation and the mean, $x_i - \bar{x}$. Figure 2.3 shows these distances for the two small sets of returns in the table above: overall, the distances for $y$ are bigger than those for $x$.

We need an overall measure of how big the distances are, one number as the mean is one number. We cannot just add the distances up, because the negative distances cancel the positive ones and the sum is always zero:

Instead we square the distances, which makes them all positive, and average them.

The Sample Variance

We use $n - 1$ instead of $n$ for technical reasons, to do with using a sample to estimate the variability of the population it came from, and the intuition does not change: we still think of $s_x^2$ as the average squared distance of the observations from their mean. For large $n$ the two versions are almost equal. In this lesson $\sigma_x^2$, with divisor $n$, is called the variance and $s_x^2$, with divisor $n - 1$, the sample variance. Statistical software usually reports the sample version.

The Standard Deviation

It is helpful to have a measure of spread in the original units of the data. If the data have units $u$, the variance has units $u^2$, which cannot be compared with the data or with the mean. The standard deviation solves this problem.

So if we want a measure of variability that can be compared with the mean, we should use the standard deviation rather than the variance. For the rowers, $\sigma_x = \sqrt{20.5} = 4.53$ cm.

For the equity returns of the last lesson,

and the standard deviations measure the greater spread in the Japanese returns which the histograms showed.

We now have two numerical summaries of a set of data: $\bar{x}$, which says where the data are, and $s_x$, which says how spread out, how variable, they are. The mean is easy to interpret, as a sort of centre of the data. We know that the bigger $s_x$ is, the more variable the data are, but how do we really interpret the number? What is a big $s_x$, and what is a small one?

Variability and Risk

For the temperatures of Figure 2.1,

and the standard deviation of the temperature in Chicago is about twice that in Seattle. For the share prices of Figure 2.2,

The standard deviation of the Tesla price is about $27\%$ of its mean, and that of the Visa price about $19\%$.

The Standard Deviation and Outliers

Sometimes the standard deviation is large because of one outlier.

Because it squares the distances, the standard deviation can be made large by the influence of a few outliers, so it can be a pessimistic notion of variability.

Chebyshev’s Inequality

There is a guarantee which says how most observations sit relative to the standard deviation. It is called Chebyshev’s inequality: for any $k > 1$,

Equivalently, at least a fraction $1 - \dfrac{1}{k^2}$ of them lie within $k$ standard deviations of the mean. With $k = 2$, for example, at least $1 - \frac14 = 75\%$ of the observations are no more than two standard deviations from the mean, whatever the data.

The reason is short. Suppose that a fraction $f$ of the $n$ observations lie more than $k\sigma_x$ from the mean, and that there is at least one of them. Each of these $fn$ observations contributes more than $k^2\sigma_x^2$ to the sum of the squared distances, and no term of the sum is negative, so

Dividing both sides by $nk^2\sigma_x^2$ gives $f < \dfrac{1}{k^2}$. The sample standard deviation is at least as large as $\sigma_x$, so the guarantee holds for $s_x$ as well.

The Standard Deviation Can Overestimate Variability

We know for sure that at least $75\%$ of the observations lie in the window $\bar{x} \pm 2\sigma_x$. Very often the truth is far better than that. In Figure 2.4 almost all the days in each city, about $97\%$, are within two standard deviations of the mean, and about $64\%$ are within one standard deviation. In such cases the width of the window $\bar{x} \pm 2\sigma_x$ gives, qualitatively, an overestimate of the variability.

The Empirical Rule

For data whose histogram is a single roughly symmetric mound, a much sharper statement is usually close to the truth.

The empirical rule helps us understand $s_x$ and relate the two summaries back to the histogram. For the Canadian returns, $\bar{x} = 0.00907$ and $s_x = 0.03833$, so

The empirical rule says that roughly $68\%$ of the observations are between the dotted lines of Figure 2.5 and roughly $95\%$ between the dashed lines. Counting, $67$ of the $107$ returns, $63\%$, lie within one standard deviation of the mean and $103$, $96\%$, lie within two. That looks reasonable.

The same thing can be seen from the time series plot in Figure 2.6. Five per cent of the $107$ observations outside the dashed lines would be about $5$ points; there are $4$ points outside, which is pretty close.

Quantiles

Quantiles tell us what fraction of the data falls in each range, which gives a more complete picture of variability.

We can measure variability by the distance between two quantiles, the width of the window they make. For example, the distance from the $10$th percentile to the $90$th:

The quantiles tell us exactly what percentage of the data fall in a range. Here $80\%$ of the days fall between the $10$th and $90$th percentiles, because $10\%$ of the days lie below the $10$th percentile and $10\%$ above the $90$th. So in each city we can reasonably expect the temperature to be in the window on $80\%$ of days, and the width of the window is a measure of the variability. The quantiles also give us a better sense of the shape of the distribution, because they show where the data are packed tightly and where they are thin.

Comparing Assets

We can use means and standard deviations to compare investments. For nine mutual funds and other assets, over the same $180$ months, the means and sample standard deviations of the monthly returns are these.

Plotting the means against the standard deviations puts all nine on one picture, Figure 2.8. It is considered good to have a large mean return and a small standard deviation, so the best assets are towards the top left. Treasury bills have almost no variability, and keystne has the largest standard deviation without a larger mean to show for it.

The same picture compares countries, here from the monthly returns of $1988$ to $1996$.

When the Mean Misleads

The Mean Is Often Good

The mean is one proxy for the centre of a distribution, and it is common to treat the centre as the typical value. In some cases this is reasonable.

But many kinds of data we meet are not like this. The features of a data set which can throw off the mean as a measure of the typical value are

1. multimodal data;
2. skewed and heavy-tailed distributions;
3. outliers;
4. high variability.

Multimodal Data

A common intuition is that the mean is roughly where the mode is. In some cases, like the repeated experiments, this is true. But sometimes a data set is naturally multimodal.

In a multimodal data set the mean of the whole data set lies between the means of the different modes, and then it is not representative of a typical observation. The mean mass of all $342$ penguins is $4{,}202$ g, which is heavier than $85\%$ of the Adélie and Chinstrap penguins and lighter than $95\%$ of the Gentoos: a typical penguin of neither kind.

The same happens when we split the penguins by sex, in Figure 2.11. The mean for the entire data set is on the large end for a female, heavier than $68\%$ of the females, and on the small end for a male, lighter than $55\%$ of the males.

All three are influenced, in the sense that the median, quantiles and standard deviation of the aggregate data do not reflect what is going on within the separate modes.

Why It Matters

Many times the mean and other summary statistics are used to make decisions. When the decisions are based on statistics of all the data, instead of mode by mode, they might not be the right decisions for everyone.

Misleading Means in the Media

The American Medical Association reports that the average American consumes about $3{,}400$ milligrams of sodium a day, against a recommended limit of $2{,}300$ milligrams.

A closer look comes from a national study which estimated sodium intake from $24$-hour urine samples. It found a mean of $4{,}205$ mg a day for men and $3{,}039$ mg for women, and that on a given day $75.1\%$ of adults, $83.5\%$ of men and $67.1\%$ of women, consumed at least the recommended $2{,}300$ mg. So

1. the data are multimodal: men and women consume quite different amounts on average;
2. the variability is limited, in the sense that three adults in four are above the recommended amount.

The conclusion that Americans eat more salt than the medical community suggests they should is still valid, but a single figure of $3{,}400$ mg describes neither men nor women well.

Source: American Medical Association, What doctors wish patients knew about sodium consumption; Cogswell et al., Estimated 24-hour urinary sodium and potassium excretion in US adults, JAMA (2018).

Skew

Even when a distribution has only one mode, it can be skewed: its histogram stretches out to the left or to the right. There are several quantitative measures of skewness, and only the concept matters here. The stretched-out part looks like a tail, so a skewed distribution is sometimes also said to be heavy-tailed.

Right Skew

Types of data which tend to be skewed to the right are

1. incomes, GDP, the number of awards won, and views or clicks online, where the rich get richer;
2. waiting times, such as the time between earthquakes or volcanic eruptions;
3. the sizes of some natural objects, such as lakes, rocks and mineral deposits.

If a distribution is skewed to the right, then typically the median and the mode are smaller than the mean.

Left Skew

Types of data which tend to be skewed to the left are the age at death and exam scores. If a distribution is skewed to the left, then typically the median and the mode are larger than the mean.

Measuring Skew

The gap between the mean and the median is one quantitative measure of skew. It is positive for the GDP data, which are skewed to the right, and negative for the exam scores, which are skewed to the left.

Outliers

Outliers, abnormally large or small values, can move the mean away from the median or the mode. Outliers and skew are related, since outliers do skew the distribution; the distinction is usually that we think of outliers as anomalies, whereas skew can be a natural property of the data.

The tutorial attendances earlier showed the same effect of an outlier on the standard deviation.

High Variability

If the variability of a data set is high, then neither the mean nor the median is representative of a typical observation.