← All self-study courses

Know when a difference is real (10 sessions, 15 minutes each, v1.0)

Goal of this course: See for yourself why a difference between averages can be tested.

This course is for undergraduates in medicine and related fields. Starting from the mean and the standard deviation, you will verify the central limit theorem with your own hands and then work through the t-test, statistical power, and common pitfalls. By the end, you will be able to explain in your own words what papers mean when they write "mean ± ..." or "p < 0.05".

A Japanese version of this course (講座_中心極限定理と検定_v1.0.md) is also available; to study in Japanese, use that file or tell the AI "Please teach this course in Japanese."

Introduction (How to Use This Course)

What this is

This is a teaching material published for students' self-study by the Division of Neurophysiology, Kobe University Graduate School of Medicine. It takes the format we have used in our laboratory's practical training, "a course you work through in dialogue, 15 minutes a day", and makes it freely available to people who do not take part in the training.

All the numbers used as examples in this course are made up for the course. They are not data from real people.

Who it is for (expected background)

• If you know high-school mathematics (means, squares, square roots, basic probability), you can start.
• It is meant both for people who have not yet taken a statistics class and for people who have taken one but "still don't really understand p-values".
• Formulas do appear, but each one is introduced only after its meaning has been put into words.

What you need

• A conversational AI (one that answers questions in a chat). You load this course file into it.
• Something to take notes on (paper and pen are enough; there are hand-calculation exercises).
• A calculator (the one on your smartphone is enough).
• (If available) spreadsheet software or Python. For the sessions that use Python, a free environment that runs in your browser is enough (one in which the two packages numpy and scipy are available). Python installed on your own computer is also fine. If you cannot use Python, you can complete the whole course with the hand-calculation and spreadsheet exercises alone.

How to proceed

1. Load this file into a conversational AI (attach the file, or paste in the full text).
2. Type "Please start Session 1."
3. The AI takes the role of the teacher and asks you questions. Answer in your own words. It is fine to make mistakes; the course is designed so that you learn from them.
4. Each session takes about 15 minutes. When 15 minutes are up, you are done for the day. The next day, type "Please start Session 2." If you continue on another day in a new conversation, load this file again before typing.

Cautions

• The AI can make mistakes. Check important points (the conditions of a theorem, formulas, numerical values) in a textbook or the original source. Textbooks for checking are listed at the end of this course.
• For code results, trust the numbers you get when you run the code yourself. Do not trust a number the AI says "will probably come out" until you have checked it.
• Do not enter your own or anyone else's personal information, clinical information, or other people's data into the AI. All practice uses the made-up data in the course.
• This course is for learning statistics and is not intended as medical advice.

Request for feedback (please be sure to send it)

This course is revised every time based on what its users tell us. If you have used it, please be sure to send us feedback.

• Send to: the address listed on the "Contact" page of the laboratory homepage
• Subject: "Course feedback: Central limit theorem and statistical tests"
• Please tell us these four things: 1. How many sessions you completed 2. What was hard to understand 3. What was good 4. How long each session took

Feedback from people who stopped partway is also very helpful.

Terms of use

• You are free to use it for personal study or in classes.
• If you distribute a modified version, please contact the laboratory (via the Contact page).
• The content is current as of October 2026.

Instructions for the AI (How to Act as the Teacher)

Everything below is addressed to the AI that has loaded this file. Learners may read it too.

You (the AI) are the teacher for this course. Follow these rules.

1. Keep each session to 15 minutes. When the learner says "Please start Session N.", conduct that session only. When 15 minutes' worth (explanation, exercises, checks) is finished, close the session.
2. Ask one question at a time and wait for the answer. Do not bundle two or more questions together.
3. Do not give the answer first. Use the "Answers and reasoning" section of each session only after the learner has answered. Do not show its contents before the learner answers.
4. If the learner seems lost, rephrase one step more simply. If they get stuck on a formula, go back to concrete numbers (dice, hours of sleep).
5. At the end of each session, give three key points and two mini-check questions. For the mini-check questions too, wait for the learner's answer before showing the answer.
6. Do not move on to the next session on your own. Wait until the learner says "Please start Session N."
7. If the learner writes about a medical consultation or personal information, decline to deal with it. Reply "This course cannot deal with that. Let's practice with the course's made-up data," and return to the course.
8. Do not state code results from imagination. Have the learner actually run the code and paste the numbers it produces. If they differ slightly from the "Example output" in the text but show the same tendency, tell the learner the code is working correctly (the random numbers generated can differ between package versions; where a theoretical value [such as σ ÷ √n] is available, have the learner compare with it). If the learner cannot run the code, you may use the "Example output" in the text, but say "These are the numbers given as an example."
9. For learners who cannot use Python, proceed with the hand-calculation and spreadsheet exercises only. The Python exercises may be skipped.
10. Take the way theorems and formulas are written in the text as the reference. When you phrase something differently from the text, explain where the difference lies.
11. If the learner says "in Japanese", continue in Japanese from then on.

Overall Map

Session 1 Variability and the Mean

Aims of this session

1. Understand that the mean is "just one central point" and that variability is separate information.
2. Be able to calculate deviations, the variance, and the standard deviation (SD) by hand.
3. Be able to say in a word why we "divide by n − 1" when computing the standard deviation.

Explanation

Question: Person A and Person B both slept an average of 6.5 hours over one week. Do the two sleep in the same way?

Answer: You cannot tell from the mean alone. Person A may have had nights of 5 hours and nights of 8 hours, while Person B may have slept almost exactly 6.5 hours every night.

Turning variability into a single number takes the following four steps.

1. Deviation = value − mean. It is the distance from the mean.
2. Square the deviations. If you simply add the deviations, the pluses and minuses cancel out and the total is always 0.
3. Add up the squares and divide by n − 1. This is the variance (its unit is "hours squared").
4. Take the square root of the variance. This is the standard deviation (SD). The unit returns to "hours".

Why divide by n − 1 rather than n? Because the mean is calculated from the same data you have in hand. Your data tend to sit closer to their own mean than to the true mean. So dividing by n gives a value a little smaller than the true population variability. Dividing by n − 1 compensates for that. If n is large, it makes almost no difference which one you divide by.

Here is one rule of thumb to remember. If the data are close to a bell shape (a normal distribution), about 68% of them fall within mean ± 1 SD, and about 95% within mean ± 2 SD (strictly, 95% falls within ± 1.96 SD).

Hands-on exercises

By hand (required): Fill in the following table for Person A's seven nights.

Work them out in this order: mean → deviations → sum of squares → ÷(7 − 1) → square root.

Python (if you can): Check your hand calculation.

ddof=1 is what specifies "divide by n − 1". Note that if you omit it, NumPy divides by n.

Check questions

1. If you simply add up all the deviations, what do you get? Why?
2. Suppose Person A had slept 30 minutes longer on every night. How would the mean and the SD change?
3. What is the unit of the SD? What is the unit of the variance?

Answers and reasoning (for the AI: do not show until the learner answers)

• Hand calculation: the mean is 6.5. The deviations are −0.5, 1.0, −1.0, 0.5, 0, 1.5, −1.5. The squares are 0.25, 1, 1, 0.25, 0, 2.25, 2.25, totaling 7.0. The variance is 7.0 ÷ 6 ≈ 1.167, and the SD ≈ 1.08 hours. Person B's SD is about 0.29 hours.
• Q1: 0. The mean is defined as "the point at which the deviations sum to 0".
• Q2: The mean rises to 7.0 hours. The SD stays at 1.08 hours. If every value shifts by the same amount, the distances from the mean (the deviations) do not change.
• Q3: The SD is in "hours"; the variance is in "hours squared". We use the SD because it lets us talk in the original unit.

Summary

• Two data sets with the same mean are different data if their variability differs.
• The SD is "the square root of the mean of the squared deviations (dividing by n − 1)"; it is the variability expressed in the original unit.
• For a bell shape, about 95% of the data fall within mean ± 2 SD.

Session 2 Standard Deviation and Standard Error of the Mean

Aims of this session

1. Understand that the standard deviation (SD) and the standard error of the mean (SEM; written SE in this course) represent different things.
2. Be able to calculate using SE = SD ÷ √n.
3. Be able to choose, according to your purpose, whether to attach the SD or the SE to a mean in a paper or presentation.

Explanation

Question: In a survey, the mean sleep duration of 16 people was 6.5 hours. One person writes "6.5 ± 0.8" in the table, and another writes "6.5 ± 0.2". Is one of them wrong?

Answer: Both can be correct, because the number after ± represents different things.

• The standard deviation (SD) describes how much individual people vary. It is a property of the data themselves. It does not get smaller when you add more people (it only gets closer to the true variability of the population).
• The standard error of the mean (SE) describes how much the single number called the mean would vary from one repetition of the survey to the next (its sampling variability). It expresses how certain the mean is.

SE = SD ÷ √n

Increasing the sample size n makes the SE smaller (we will check why it is √n in Session 5).

A student once asked exactly this question: "Should I attach the standard deviation or the standard error of the mean (SEM)?" The answer depends on what you want to convey.

A 95% confidence interval shows how certain the mean is as a "range". It means the following: if the same survey were repeated many times and this range were constructed each time, about 95% of the ranges constructed would contain the true mean (the mean of the whole population). Rephrasing it as "there is a 95% probability that the true mean lies in this range" is not strictly correct, but at first it is enough to read it as "the range in which the mean is likely to lie". It is calculated roughly as "mean ± 2 SE" (strictly, you use a t value that depends on the number of people; this appears in Session 7).

You might think, "The SE is smaller and looks nicer, so let's use the SE." But choosing by appearance makes the variability look smaller to the reader. Always state clearly which one you wrote. A table that just says "± 0.2" is unhelpful, because the reader cannot tell whether it is the SD or the SE.

Hands-on exercises

By hand (required):

1. The SD of sleep duration for 16 people was 0.8 hours. What is the SE?
2. With the same SD and 64 people, what is the SE?
3. By what factor must you increase the number of people to halve the SE?

Python (if you can): See what happens to the SD and the SE as the number of people increases.

As the sample size increases, the SD stays around the true population value (1.0). The SE roughly halves each time the number of people quadruples. Notice also that when the sample size is small (n = 4), both the mean and the SD can be far from their true values.

Check questions

1. If the SD of sleep duration for 100 people is 1.0 hour, what is the SE?
2. If you quadruple the number of people, what happens to the SD and to the SE?
3. What is missing from a table that just says "6.5 ± 0.2"?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercises: 1. 0.8 ÷ √16 = 0.2 hours. 2. 0.8 ÷ 8 = 0.1 hours. 3. 4 times.
• Q1: 1.0 ÷ 10 = 0.1 hours.
• Q2: The SD hardly changes (it only gets closer to the population variability). The SE becomes about half.
• Q3: A clear statement of whether the number after ± is the SD or the SE (or a confidence interval), and the number of people n.

Summary

• The SD is the variability of individuals; the SE is the sampling variability of the mean.
• SE = SD ÷ √n. Increasing the number of people makes only the SE smaller.
• Always state what you attached to the mean. For comparisons, the 95% confidence interval is an easy-to-understand choice.

Session 3 Taking the Mean Again and Again Gives a Bell Shape (Hands-on)

Aims of this session

1. Build the distribution of dice "means" with your own hands.
2. Be able to say how the shape and width of the distribution of means change as you increase the number n of values averaged.
3. Understand that what became bell-shaped is "the mean", not "the face of a die".

Explanation

Question: Each face of a single die, 1 to 6, comes up with the same probability. As a graph, it is flat. Now suppose you repeat "roll 5 dice and compute the mean" many times and collect those means. What shape will they form?

Answer: It will not be flat. It approaches a bell shape with a high peak in the middle (3.5).

The reason is the number of combinations. Consider the case of 2 dice. There are 6 × 6 = 36 ways the two faces can come up. A mean of 1 arises in only one way, (1, 1). A mean of 3.5 arises in six ways: (1, 6) (2, 5) (3, 4) (4, 3) (5, 2) (6, 1). An extreme mean appears only if all the faces happen to be extreme together, so it is rare. A mean in the middle arises from many different combinations, so it appears often. The more dice you use, the clearer this difference becomes.

Hands-on exercises

Choose one of the following three that you can do.

A. By hand (with a die, or something that can stand in for one): Roll a die 5 times and compute the mean. Repeat this 10 times and plot the 10 means as dots on a number line. Compare them with the faces from single rolls (10 rolls' worth).

B. With spreadsheet software:

1. Enter =RANDBETWEEN(1,6) in the five cells A1 to E1 (5 dice).
2. Enter =AVERAGE(A1:E1) in F1 (the mean of the 5).
3. Copy row 1 down to row 500 (500 repetitions).
4. Make a histogram of column F (a bar chart of counts per interval). Compare it with a histogram of column A (a single die).
5. Change 5 dice to 2 or 30, and compare the shape and width.

C. With Python: For n = 1, 2, 4, and 30, collect 10,000 means each and look at them as text histograms.

With n = 1, every other bin is empty because a single die can only show whole numbers.

Look at the figure you made and write the following three things in your own words.

• As n increases, how did the shape of the distribution of means change (compared with the flat shape for a single die)?
• As n increases, how did the width of the distribution of means change?
• Did the "mean of means" change with n?

The means you made here are made-up practice values. Do not count them as research results or as numbers of participants.

Check questions

1. With 2 dice, what is the probability that the mean is exactly 1? What is the probability that the mean is 3.5?
2. The SD of the means for n = 30 was about what fraction of that for n = 1? How might that number be related to 30?
3. What became bell-shaped: "the faces of the die" or "the mean"?

Answers and reasoning (for the AI: do not show until the learner answers)

• Q1: 1/36 and 6/36 = 1/6.
• Q2: 1.715 ÷ 0.307 ≈ 5.6, which is close to √30 ≈ 5.5. The SD becomes 1/√n as large (the same form as SE = SD ÷ √n in Session 2; the reason is checked in Session 5).
• Q3: The mean. The distribution of the faces of a single die stays flat no matter how many times you roll. Not confusing these two is the preparation for Session 4.
• Shape: flat → triangular → approaching a bell. Width: becomes narrower. Mean of means: around 3.5 in every case, unchanged.

Summary

• As you increase the number n of values averaged, the distribution of means approaches a bell shape.
• Its width (the SD of the means) becomes narrower, roughly 1/√n as large.
• What becomes bell-shaped is "the distribution of means". The shape of the original data does not change.

Session 4 What the Central Limit Theorem States, and Its Conditions

Aims of this session

1. Be able to state the central limit theorem both in its textbook form and as one sentence in your own words.
2. Be able to state the three conditions for the theorem to hold (independence, identical distribution, finite variance).
3. Be able to correct two misconceptions: "the data become normally distributed" and "30 values are always enough".

Explanation

Question: You may have heard that "the t-test cannot be used unless the data are normally distributed". In practice, however, the t-test is widely used even on data that are not symmetric (skewed), such as the time it takes to fall asleep. Why is this allowed?

Answer: Because what the statistical test looks at is not the data themselves but "the mean". Thanks to the central limit theorem, the sampling distribution of the mean approaches a normal distribution regardless of the shape of the original data.

Theorem (textbook form): Suppose X₁, X₂, …, Xₙ are mutually independent and follow the same distribution (written IID), and that this distribution has mean μ and variance σ² (finite). μ and σ are the true mean and SD of the whole population (that distribution), and are to be distinguished from the mean and SD calculated from the data in hand. Let the sample mean be X̄ = (X₁ + … + Xₙ) ÷ n, and set

Z = (X̄ − μ) ÷ (σ ÷ √n)

Then, as n increases, the distribution of Z approaches the standard normal distribution N(0, 1) (the normal distribution with mean 0 and SD 1). In symbols, for every z, P(Z ≤ z) → Φ(z) (Φ is the cumulative distribution function of the standard normal distribution: the probability of a value of z or less).

(This is the same content as Theorem 5.8 in Wasserman, All of Statistics (2004), and Chapter 14 of Dekking et al., A Modern Introduction to Probability and Statistics (2005).)

In your own words: "If you repeatedly take n data values that have the same properties and do not influence one another, and compute their mean, then the way that mean varies can be approximated, regardless of the shape of the original data, by 'a normal distribution with mean μ and SD σ ÷ √n', and the approximation gets better as n gets larger."

The three conditions

1. Independence: one value does not influence the next. The same person's sleep on consecutive nights resembles the previous night, so it is not independent (Sessions 5 and 8).
2. Identical distribution: every value comes from the same population (the same mechanism). The basic textbook form assumes this (there are extended forms that hold even without identical distributions, but this course does not cover them).
3. Finite variance: the variability is not infinitely large. With a heavy-tailed distribution that very occasionally produces extremely large values, the mean takes a long time to settle down.

Two common misconceptions

• Misconception 1: "Increasing n makes the data normally distributed." It does not. What approaches a normal distribution is the distribution of the mean. Even if you measure the time to fall asleep in 1,000 people, the data themselves remain skewed. Wasserman's book also stresses that what is being approximated is a probability about the mean, not the data themselves.
• Misconception 2: "30 values are always enough." 30 is a rule of thumb from experience, not part of the theorem. The theorem is about "where things end up as n is made larger and larger". The more strongly skewed the original data, the larger the n needed to approach a normal distribution.

Let us also distinguish a similarly named result. The law of large numbers says that "as n increases, the mean approaches the true mean μ". The central limit theorem describes, on the way there, "the shape (normal distribution) and size (σ ÷ √n) of the fluctuations".

Hands-on exercises

Think it through (required): The following ways of collecting data each break one of the three conditions. Say which condition each one breaks. Some break none.

• (a) One student's sleep duration over 14 consecutive nights.
• (b) The sleep duration of 10 people, collected by deciding that the first 5 come from first-year students and the next 5 from sixth-year students.
• (c) The "time to fall asleep" (a skewed distribution) of 40 people chosen by lottery from the whole year group.

Python (if you can): Measure how close the mean of a skewed distribution gets to symmetry as n increases. "Skewness" is a number that expresses left–right asymmetry; 0 means symmetric. For an exponential distribution with a mean of 15 minutes (usually short, occasionally long), the skewness is 2, and the skewness of the sample mean of n values is known to be 2 ÷ √n. In the second half, we also look at how the theorem fails for the Cauchy distribution, a distribution with infinitely large variance.

The skewness roughly halves each time n quadruples, and the distribution of the mean gets closer to symmetric (a bell). For the Cauchy distribution, on the other hand, the width does not shrink at all even when you average 2,500 values. When condition 3 (finite variance) is broken, the central limit theorem does not hold.

Check questions

1. When the central limit theorem says "approaches a normal distribution", the distribution of what is it talking about?
2. State the three conditions of the theorem.
3. What is the approximate skewness of means of 16 values each, drawn from a distribution with skewness 2?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercise: (a) Independence is broken (each night resembles the previous one). (b) Identical distribution is broken (if the sleep distribution differs by year, the first and second halves come from different distributions). However, if people were chosen by lottery from all years, the values would be independent values from a single distribution, "the population with years mixed". (c) Breaks none. Even though it is skewed, it satisfies independence, identical distribution, and finite variance. A mean of 40 people can be approximated quite well by a normal distribution.
• Q1: The distribution of the sample mean (in standardized form). Not the data themselves.
• Q2: Independence, identical distribution, finite variance.
• Q3: 2 ÷ √16 = 0.5.
• Supplement (if the learner shows interest): the Berry–Esseen inequality gives an upper bound on the error of the normal approximation. The bound is proportional to the "mean of the cubed deviations from the mean" of the original distribution divided by σ³ (larger when the skewness or the heaviness of the tails is larger), and it shrinks at a rate of 1/√n (Wasserman, Theorem 5.11).

Summary

• Central limit theorem: for IID data with finite variance, the distribution of the sample mean can be approximated by a normal distribution with mean μ and SD σ ÷ √n, increasingly well as n grows.
• What approaches a normal distribution is the distribution of the mean, not the data.
• 30 is only a rule of thumb; the stronger the skew, the larger the n needed. If the variance is infinite, the theorem does not hold.

Session 5 Why the Standard Error of the Mean Shrinks with √n

Aims of this session

1. Be able to explain, from the addition of variances, why the SD of the mean is σ ÷ √n.
2. Understand that when values are not independent (repeated measurements of the same person), the variability of the mean does not shrink as much as you might expect.
3. Understand that a bias in the way you measure does not decrease when you increase n.

Explanation

Question: If you double the number of people, will the variability of the mean be halved?

Answer: No. To halve it you need four times as many. To reduce it to one-tenth you need 100 times as many.

The reason follows from two rules.

1. When you add independent quantities, their variances add. If X₁, …, Xₙ are independent and each has variance σ², the variance of their sum is nσ². SDs (the square roots of variances) do not add, because positive and negative deviations partly cancel each other out.

2. Multiplying by c multiplies the variance by c². The mean is "the sum ÷ n", so its variance is multiplied by (1/n)².

Putting these two together,

variance of the mean = nσ² × (1/n)² = σ² ÷ n SD of the mean = σ ÷ √n

This is what lies behind "SE = SD ÷ √n" in Session 2.

When values are not independent, this does not hold. A student actually once made this plan: "I want to know the mean sleep duration of all students, so I will measure myself for 8 nights." The mean of the 8 nights is pulled toward your own "usual sleep". When values resemble one another and the degree of resemblance is expressed by the correlation coefficient ρ (read "rho"; a number from −1 to 1 that is closer to 1 the more two values move together, and 0 if they are unrelated; it comes up again in Session 9), the variance of the mean is

σ² ÷ n × (1 + (n − 1)ρ)

(when every pair has the same correlation ρ). When the differences between people account for 30% of the total variability (variance), the correlation coefficient between two nights of the same person is ρ = 0.3. In that case, the mean of 8 nights measured on one person carries only as much information as 8 ÷ (1 + 7 × 0.3) ≈ 2.6 independent people. However many nights you measure, the upper limit is 1 ÷ ρ ≈ 3.3 people's worth. If you want to know about other people, the only way is to increase the number of people.

Bias does not decrease with n either. If a clock measures 20 minutes too long every night, the mean stays 20 minutes off however many nights you measure. Only "random variation" shrinks with √n.

Hands-on exercises

By hand (required):

1. Find the variance of the face of a single die. The mean is 3.5. The mean of the squared faces is (1 + 4 + 9 + 16 + 25 + 36) ÷ 6. The variance can be obtained as "mean of the squares" − "square of the mean".
2. Predict the variance and SD of the mean of 2 dice from the formula above. Compare with the result for n = 2 in Session 3 (SD of means 1.211).
3. Predict the SD of the mean of 9 dice.

Python (if you can):

The simulated "SD of means" agrees well with the formula σ ÷ √n. On the other hand, the variability of the mean when one person is measured for 8 nights (0.621) is about 1.8 times as large as when 8 people are measured for one night each (0.354).

Check questions

1. By what factor must you increase n to reduce the SD of the mean to one-third?
2. When differences between people account for 30% (ρ = 0.3), how many independent people's worth of information do you get by measuring one person for 100 nights?
3. What happens to a bias in measurement (such as always measuring too long) when you increase n?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercises: 1. The mean of the squares is 91 ÷ 6 ≈ 15.17. The variance is 15.17 − 12.25 ≈ 2.917 (= 35/12), and the SD ≈ 1.708. 2. Variance 2.917 ÷ 2 ≈ 1.458, SD ≈ 1.208. This agrees well with 1.211 in Session 3 (counting all 36 combinations also gives 35/24 ≈ 1.458). 3. 1.708 ÷ 3 ≈ 0.569.
• Q1: 9 times.
• Q2: 100 ÷ (1 + 99 × 0.3) = 100 ÷ 30.7 ≈ 3.3 people's worth.
• Q3: It does not decrease. A bias shifts the values in the same direction every time, so averaging does not cancel it out.
• If you want to know about that one person's record itself (that person's sleep), adding more nights does make sense. However, if the nights resemble one another, even then the information is less than that of the same number of independent nights (Session 8).

Summary

• If values are independent, variances add, and from this the SD of the mean = σ ÷ √n follows. You need 4 times as many measurements just to halve it.
• If values resemble one another (are not independent), the variability of the mean is larger than σ ÷ √n. Repeated measurements of the same person must not be counted as numbers of people.
• Only random variation decreases with √n; bias in the way you measure does not.

Session 6 The Null Hypothesis and the p-Value (Getting an Intuition with a Permutation Test)

Aims of this session

1. Understand why we set up a null hypothesis (the assumption that "there is no difference").
2. Be able to count a p-value yourself with a permutation test.
3. Be able to state the correct meaning of the p-value in one sentence and to correct three common misreadings.

Explanation

Question: When we compared the sleep duration of 8 members of a club with early-morning practice and 8 members of a club without it, the means differed by about 0.5 hours (the data are made up). Is this difference due to "the difference between the clubs"? Or did the 8 people who happened to be chosen just turn out that way?

Answer: First, we deliberately assume the opposite: "In reality, sleep does not differ between the clubs." This is called the null hypothesis. Under that assumption, we count how often a difference of 0.5 hours or more would arise by chance alone. If it hardly ever arises, we conclude that the assumption "there is no difference" is the one that looks doubtful.

Differences due to chance alone can be generated by reshuffling the labels. If there is no difference, the labels "morning practice" and "no morning practice" have nothing to do with sleep. So we keep the 16 people's sleep durations as they are, randomly reassign only the labels into 8 and 8, and recalculate the difference. Repeating this thousands of times gives the distribution of "differences that arise when there is no difference". The proportion of reshuffles that give a difference at least as large as the actual difference (0.5 hours) is the p-value. This method is called a permutation test.

The meaning of the p-value (this one sentence is the correct answer): the probability, assuming the null hypothesis is true, of obtaining a result as extreme as the one observed or more extreme.

In this course, "extreme" is judged by the size of the difference regardless of its direction (a two-sided statistical test). Whether "the club with morning practice sleeps longer" or "shorter", differences of the same size count as equally extreme.

Common misreadings

• × "p = 0.03, so the probability that the null hypothesis is true is 3%": the p-value is a probability calculated by assuming the null hypothesis is true. It is not the probability that the hypothesis is true.
• × "The smaller p is, the larger the difference": the p-value does not express the size of the difference. Even a small difference gives a small p if there are many people. Always report the difference itself (how many hours) along with it.
• × "It is not significant (p > 0.05), so there is no difference": this only means "there is not enough evidence for a difference". With few people, even a real difference is missed (Session 9).

The 0.05 significance level was set by convention. There is no essential difference between p = 0.049 and p = 0.051.

One last point. Even if p were small, this comparison would not let us say that "morning practice reduces sleep". The clubs with morning practice may also differ in, for example, the amount of training or the proportion of students living away from home (Session 10).

Hands-on exercises

Count by hand (required): a small example with 3 people per group.

• Morning practice: 5.8, 6.0, 6.4 hours (mean about 6.07)
• No morning practice: 6.6, 7.0, 7.4 hours (mean 7.00)
• The difference is about 0.93 hours.

How many ways are there in total to split the 6 numbers into "3 and 3"? Of those, how many splits give a difference whose size (regardless of direction) is 0.93 hours or more? What is the p-value?

Python (if you can): With data from 8 people per group, reshuffle the labels 10,000 times.

In about 2.8% of the 10,000 reshuffles, a difference as large as the actual one or larger appeared. In other words, if we assume "there is no difference", this difference is fairly rare.

Check questions

1. State the meaning of p = 0.03 in one sentence.
2. Is "p = 0.03, so the probability that the null hypothesis is true is 3%" correct?
3. With 4 people per group (splitting 8 numbers into 4 and 4), what is the smallest two-sided p-value you can get?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercise: there are 6C3 = 20 ways to split. The difference is 0.93 or more in 2 of them: the actual split and the same split with the two sides swapped. p = 2 ÷ 20 = 0.10. With 3 people per group, the two-sided p-value cannot go below 0.10, however clear the difference. This is one reason why small studies tend to come out "not significant".
• Q1: "If we assume there is no difference between the clubs, the probability of a difference as large as this one or larger is 3%."
• Q2: No. The p-value is a probability calculated by assuming the hypothesis is true; it is not the probability that the hypothesis is true.
• Q3: 8C4 = 70 ways. 2 ÷ 70 ≈ 0.029.
• Supplement: there are 16C8 = 12,870 ways to split 16 people into 8 and 8, so it is also possible to count them all. The 10,000 random reshuffles are an approximation of that.

Summary

• The p-value is obtained by assuming the null hypothesis "there is no difference" and counting the probability that chance alone produces a difference at least as large as the observed one.
• A permutation test can produce a p-value just by reshuffling labels.
• The p-value is neither "the probability that the hypothesis is true" nor "the size of the difference". Not being significant is not proof that there is no difference.

Session 7 The t-Test Stands on the Central Limit Theorem (z and t)

Aims of this session

1. Understand that the z-test is a statistical test that uses the central limit theorem directly.
2. Understand why using the SD in hand in place of the true σ gives a t distribution.
3. Confirm by simulation that it is acceptable to use the t-test on skewed data "when n is reasonably large".

Explanation

Question: The permutation test in Session 6 needed 10,000 reshuffles. How did people obtain p-values in the days before computers?

Answer: Thanks to the central limit theorem, we know that "the mean (or a difference of means) varies in a shape close to a normal distribution". So they could get by with a single look-up in a table of the normal distribution.

z-test: suppose the null hypothesis is "the true mean is μ₀".

z = (X̄ − μ₀) ÷ (σ ÷ √n)

According to the central limit theorem (Session 4), if the null hypothesis is true, z approximately follows N(0, 1). Under N(0, 1), the probability that |z| exceeds 1.96 is 5%, so if |z| > 1.96, then p < 0.05 (two-sided). The denominator σ ÷ √n is exactly the standard error of the mean from Session 2.

The problem: σ is unknown. In practice, the true σ is not known. We substitute the SD of the data in hand (written s).

t = (X̄ − μ₀) ÷ (s ÷ √n)

Because s also varies by chance, t is slightly more likely than z to take values far from 0 (such as values beyond ±2). That is, the distribution has heavier tails. This distribution is called the t distribution with n − 1 degrees of freedom. The degrees of freedom are "the number of values that are free to vary" when variability is calculated. The n deviations always sum to 0 (Session 1), so once n − 1 of them are fixed, the last one is determined automatically. This is the same reason we divided the SD by n − 1 in Session 1. The smaller the degrees of freedom (the fewer the people), the more s varies and the heavier the tails of the t distribution. It was published in 1908 by Gosset, who worked at a brewery, under the pen name "Student", so it is also called Student's t distribution.

• If the data are normally distributed, the t distribution holds exactly.
• Even if the data are not normally distributed, when n is large, the distribution of t approaches N(0, 1), by the central limit theorem and because s approaches σ (Wasserman, Theorem 5.10). This is why the t-test can also be used on skewed data. However, when n is small, this approximation breaks down.

When comparing two groups (Welch's t-test):

t = (X̄₁ − X̄₂) ÷ √(s₁²/n₁ + s₂²/n₂)

The denominator is "the standard error of the difference between the means". If the two means are independent, the variance of their difference is the sum of their variances (Session 5), so we add the squared SEs and take the square root. This form allows the two groups to have different variability, so use it when in doubt (the degrees of freedom in this case are given by a somewhat complicated formula, which the computer works out for you).

Make it a habit to report, along with the p-value, the difference itself (how many hours) and its 95% confidence interval (difference ± t critical value × SE of the difference).

Hands-on exercises

By hand (required): You measured the sleep duration of 9 people and got a mean of 6.4 hours and an SD of 0.9 hours. Test the null hypothesis "the true mean is 7 hours".

1. Find the SE.
2. Find t.
3. The critical value of t with 8 degrees of freedom is 2.306. Is the result significant at the two-sided 5% level?
4. If you mistakenly used the z critical value of 1.96, what would the conclusion be?

Python (if you can): First make a table of t critical values, then run a t-test on the same data as in Session 6 and compare its p-value with that of the permutation test.

The permutation test's p ≈ 0.028 and the t-test's p ≈ 0.034 are almost the same. The two are completely different calculations, but they measure the same thing.

Finally, with skewed data (time to fall asleep, mean 15 minutes), count the proportion of cases in which p < 0.05 even though the null hypothesis is true. For a statistical test that works correctly, this proportion should be 5%.

With n = 5, false positives, which should occur 5% of the time, occur about 12% of the time. As n increases, it approaches 5%. This is the central limit theorem taking effect before your eyes. When you analyze skewed data from a small sample, consider measures such as using a permutation test.

Check questions

1. What does the denominator of the t-test formula represent?
2. Why do the z critical value (1.96) and the t critical value (2.306 with 8 degrees of freedom) differ?
3. What happened when you ran a t-test on skewed data collected from only 5 people?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercises: 1. SE = 0.9 ÷ √9 = 0.3. 2. t = (6.4 − 7) ÷ 0.3 = −2.0. 3. |−2.0| < 2.306, so it is not significant (p ≈ 0.081). 4. |−2.0| > 1.96, so it would wrongly become "significant" (p ≈ 0.046). Using the z critical value when there are few people makes the decision too lenient.
• Q1: The standard error of the mean (or of the difference between means).
• Q2: The s in hand is used in place of σ, and s itself varies, so the t distribution has heavier tails. The larger the degrees of freedom (the larger n), the closer s gets to σ and the closer the t critical value gets to 1.96.
• Q3: Even though the null hypothesis was true, p < 0.05 occurred about 12% of the time (it should be 5%). With small n, the normal approximation given by the central limit theorem is not yet accurate.

Summary

• The z-test is a statistical test that uses the central limit theorem (the mean varies roughly as a normal distribution) directly.
• Using s in place of σ gives the t distribution. When n is large, the t-test works almost correctly even if the data are skewed.
• With small n and skewed data, the t-test becomes too lenient. Report the size of the difference and its confidence interval along with the p-value.

Session 8 Paired Comparisons, and Cautions for Before–After Comparisons

Aims of this session

1. Understand that in a comparison where the same people are measured twice, taking "each person's difference" reduces the chance of missing an effect.
2. Understand that the paired t-test is a one-sample t-test on the differences.
3. Be able to state the four cautions for comparing one person's "8 nights before" with their "8 nights after".

Explanation

Question: We compared the sleep duration of 10 people in an ordinary week and in an exam week (the data are made up). The mean of each person's difference was 0.4 hours, yet an unpaired t-test treating them as two groups gave p ≈ 0.41. Can we say that "sleep does not change even in exam week"?

Answer: No. A "paired t-test", which takes the differences within the same person, gives p ≈ 0.0006 on the same data.

The reason is the differences between people. Some people manage on 6 hours; others sleep 7.5 hours. When we compare two groups, these differences between people (which are large) enter the variability in the denominator, and the effect of the exams (which is small) is buried. If we take "ordinary week − exam week" for the same person, the differences between people disappear in the subtraction.

Paired t-test: form each person's difference d₁, …, dₙ, and from their mean d̄ and SD s_d calculate

t = d̄ ÷ (s_d ÷ √n), with n − 1 degrees of freedom

This is simply the one-sample t-test of Session 7 applied to the differences.

Match pairs by name (ID number). Do not pair two tables by the order of their rows. If only one of the tables has been re-sorted, you will be calculating differences between different people. Always pair by the person's ID number.

Cautions for a before–after comparison in one person. A student once asked, "Can I do a t-test on my own 8 nights before I started something and 8 nights after I started it?" The calculation can be done, but watch out for the following four points.

1. Nights are not independent. Each night resembles the previous one, so 16 nights do not carry 16 people's worth of information (Session 5). If you run a t-test as it is, false positives become more likely (see the simulation below).
2. Other things change between before and after. Season, exams, club schedules, getting used to things. Even if a difference appears, you cannot tell whether it is due to what you started or to some other change.
3. Regression to the mean. People often start something during a bad patch. After a bad patch, values tend to return to their usual level even if nothing is done, so it looks as if "it worked".
4. The conclusion applies to that one person. This comparison cannot tell you whether it applies to other people.

Ways to improve: alternate between periods with and without it (for example, switching every week), decide by lottery on which days to do it, keep records for longer, and model the time trend in the analysis. Then write the conclusion honestly as "what could be seen from one person's records".

Hands-on exercises

By hand (required): mean sleep duration of 4 people in an ordinary week and an exam week (made up).

1. Fill in the differences, and find the mean and SD of the differences.
2. Find the paired t. The critical value of t with 3 degrees of freedom is 3.182. Is it significant at the two-sided 5% level?
3. If you compare them as two groups, the SD of each group is about 0.88 hours. Compared with the mean difference (0.375 hours), explain in words why the two-group comparison tends to miss the effect.

Python (if you can):

Next, take 16 nights in which nothing was done, split them into the first 8 and the last 8, run a t-test, and count the proportion of cases that come out as "a difference between before and after". Compare the case where nights are independent (similarity 0) with the case where each night resembles the previous one (similarity 0.6). "Similarity" here is the autocorrelation: the correlation coefficient between the sleep duration on one night and on the previous night (the same kind of number as ρ in Session 5); 0.6 indicates a fairly strong resemblance.

If the nights are independent, false positives ("differences") occur about 5% of the time (as expected from the 5% significance level). If each night resembles the previous one, "a difference" comes out about 28% of the time even though nothing was done.

Check questions

1. The paired t-test is really which t-test, applied to what?
2. When pairing two tables, what do you match them by?
3. In a before–after comparison in one person, name two things other than what you started that can produce a difference.

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercises: 1. The differences are 0.4, 0.3, 0.5, 0.3. Mean 0.375, SD ≈ 0.096. 2. SE ≈ 0.096 ÷ 2 ≈ 0.048, t ≈ 7.83 > 3.182, so it is significant (p ≈ 0.004). 3. When compared as two groups, the differences between people (SD about 0.88 hours) enter the denominator, and the 0.375-hour difference is buried (Welch's p ≈ 0.57). Taking differences removes the differences between people, leaving variability of only about 0.1 hours.
• Q1: A one-sample t-test applied to each person's differences (testing whether the mean difference is 0).
• Q2: The person's ID number (name). Not the order of the rows.
• Q3: Things that changed during the same period, such as the season, exams, club schedules, or getting used to things; regression to the mean. It is good if the learner can also name chance "differences" caused by nights resembling one another.

Summary

• Two values from the same person are compared by taking each person's difference (the paired t-test). Because the differences between people disappear, effects are missed less often.
• Match pairs by ID number. Do not pair by row order.
• For a before–after comparison in one person, watch for four things: nights are not independent, other changes, regression to the mean, and the conclusion applies only to that one person.

Session 9 Statistical Power and the Required Sample Size

Aims of this session

1. Understand the two kinds of error (saying "there is" when there is not; saying "there is not" when there is) and what statistical power means.
2. Be able to calculate with a formula the sample size needed to detect a correlation.
3. Confirm by simulation that the calculated number really gives that power.

Explanation

Question: Can a statistical test come out "not significant" even when there really is a relationship?

Answer: Often. If there are not enough data, even a relationship that really exists is missed.

There are two kinds of error in a statistical test.

Statistical power = the probability of detecting a difference (relationship) as "significant" when one really exists. The usual target is 80%. Power is determined by the following four things.

• The size of the difference (relationship): the larger, the easier to detect.
• The number of data n: the more, the easier to detect (because the SE shrinks with √n; Session 5).
• The variability: the smaller, the easier to detect.
• α (the significance level): the more lenient, the easier to detect, but type I errors also increase.

From a student's question: "How many times do I need to measure to detect a correlation of r = 0.7?" For the correlation coefficient r (how closely two quantities move together along a straight line; from −1 to 1), a rough guide to the required number can be obtained with the following formula (an approximation based on a method called Fisher's z-transformation).

n ≈ ((z_α + z_β) ÷ atanh(r))² + 3

• z_α = 1.960 (the two-sided 5% critical value), z_β = 0.842 (the value corresponding to 80% power).
• atanh(r) is a function that converts r into "a quantity that varies in a shape close to a normal distribution, with a spread (SD) of about 1 ÷ √(n − 3) regardless of the value of r" (it is not a function taught in high school, so it is enough to look up its values with a calculator or a table). atanh(0.7) ≈ 0.867, atanh(0.5) ≈ 0.549, atanh(0.3) ≈ 0.310.
• 0.842 is the 80th percentile of the standard normal distribution. We use this value because the target is 80% power.
• The idea behind the formula: choose n so that the size of the relationship you want to detect, atanh(r), is at least (1.960 + 0.842) times the chance variation 1 ÷ √(n − 3). The "+ 3" in the formula comes from the n − 3 in this formula for the variation.

For r = 0.7, ((1.960 + 0.842) ÷ 0.867)² + 3 ≈ 10.4 + 3 ≈ 13.4, so rounding up gives 14 pairs.

There are two things to watch out for. First, these "14 pairs" are a count of independent pairs (14 people, for example). One person's 14 nights resemble one another, so they do not carry 14 pairs' worth of information. Second, r = 0.7 is a strong correlation that rarely appears in real research. Detecting a weak correlation requires far larger numbers.

The idea is the same when comparing the means of two groups. As a guide for 80% power, if the difference is half the SD (the person-to-person standard deviation, assumed equal in both groups) (difference ÷ SD = 0.5), about 64 people per group are needed; if the difference equals the SD (1.0), about 17 people per group.

One more thing: a result that "happened to be significant" with small numbers tends to overestimate the true effect, because it becomes significant only when it happens to come out large. Read the striking results of small studies with a discount.

Hands-on exercises

By hand (required): Using the formula above, calculate the number needed for r = 0.5 (atanh(0.5) ≈ 0.549). Also calculate it for r = 0.3 (atanh(0.3) ≈ 0.310).

Python (if you can): Get the answer from the formula; then generate two quantities whose true correlation is 0.7 and count by simulation the proportion of cases in which p < 0.05 with n pairs (the power).

With the formula's guide of 14 pairs, the power came out at about 85% (because the formula is an approximation, it comes out a little above the 80% target). With 8 pairs, nearly half the time the result is "not significant" even though a really strong correlation exists.

Check questions

1. What does "80% power" mean?
2. Roughly how many pairs are needed to detect a correlation of r = 0.3?
3. Why can you not say "it was not significant, so there is no relationship"?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercise: for r = 0.5, (2.802 ÷ 0.549)² + 3 ≈ 26.0 + 3 ≈ 29.0; rounding up gives 30 pairs. For r = 0.3, (2.802 ÷ 0.310)² + 3 ≈ 81.7 + 3 ≈ 84.7; with the computer's value (atanh(0.3) = 0.3095) it is 84.9, so 85 pairs.
• Q1: When a difference (relationship) really exists, it is detected as "significant" 8 times out of 10.
• Q2: About 85 pairs.
• Q3: Because it may simply be that the power was insufficient and a real relationship was missed (a type II error).
• Supplement: the two-group figures "about 64 people, about 17 people" are values from an exact calculation using the t distribution (the formula based on the normal approximation gives about 63 and about 16).

Summary

• Statistical power is the probability of detecting a difference (relationship) that really exists. The usual target is 80%.
• For r = 0.7 you need about 14 independent pairs, and for r = 0.3 about 85. Repeated measurements of the same person do not count toward the number of pairs.
• Not being significant is not proof that there is no relationship.

Session 10 Common Pitfalls (Multiple Comparisons, p-Hacking, Correlation and Causation)

Aims of this session

1. Be able to show with numbers that running many statistical tests produces "significant" results even when there is nothing there.
2. Understand that p-hacking, such as "continuing until it becomes significant", increases chance significance.
3. Be able to explain with examples why causation cannot be inferred from correlation (confounding, time trends).

Explanation

Question: Is it hard to get a "significant result" out of data in which there is no relationship at all?

Answer: It is very easy. In this session you will experience three ways of doing it. Having experienced them makes you harder to fool.

Pitfall 1: Multiple comparisons. Suppose you collect 20 kinds of records that are unrelated to how you feel, and test the correlation of each with how you feel. The probability that a single statistical test gives p < 0.05 by chance is 5%. With 20 statistical tests, the probability that at least one is "significant" by chance is

1 − 0.95²⁰ ≈ 0.64

Even when there is nothing there, there is a better than 60% chance of a "discovery".

• Countermeasures: before you look, decide on one measure to examine and write it down (the primary outcome). If you examined many, report the full number you examined. Use a stricter significance level (for example, with the Bonferroni method, for 20 statistical tests the significance level is 0.05 ÷ 20 = 0.0025. If each statistical test has a 0.0025 chance of being significant by chance, the probability that any of the 20 is significant by chance is at most 20 × 0.0025 = 0.05 in total).

Pitfall 2: p-hacking. This means tinkering with the data or the analysis until the result becomes significant. Common forms are as follows.

• Running the statistical test every day and stopping measurement on the day it becomes significant.
• Removing inconvenient values as "outliers".
• Trying many analysis methods and reporting only the ones that came out significant.
• Countermeasure: decide on the number of measurements and the analysis method before measuring, and write them down (preregistration).

In fact, the made-up data in this course were also generated by choosing random seeds that produce numbers that are easy to explain (the data with 8 people per group in Sessions 6 and 7, and the time-trend example in this session). This is acceptable because it is teaching material; doing the same in research would be p-hacking itself.

Pitfall 3: Confusing correlation with causation. Even if there is a correlation, one of the two is not necessarily the cause.

• Confounding: a third factor that affects both. Example: before a competition, nervousness makes sleep lighter, and at the same time fatigue lowers performance. Even if sleep and performance are correlated, you cannot say that sleeping more would improve performance.
• Time trend: this is a pattern that has actually occurred in a student's self-recorded data. A strong correlation appeared between training performance and sleep duration, but on closer inspection, performance was simply improving little by little with training, and over the same period sleep was also increasing little by little. When both move together with time, a correlation appears even if they are unrelated to each other. Time is the confounder.
• Countermeasures: include time in the analysis (look at the correlation after removing the time trend), analyze day-to-day changes (changes from the previous day), and if you want to establish a cause, run an experiment in which conditions are assigned by lottery.

Hands-on exercises

By hand (required):

1. Calculate the probability that at least one of 5 statistical tests is "significant" by chance (1 − 0.95⁵). Also calculate it for 10.
2. For 10 statistical tests, what is the Bonferroni-corrected significance level?

Python (if you can): Confirm the three pitfalls with numbers.

Even though the true effect was 0, simply "stopping once it is significant" increased chance significance from 5% to about 23%.

Sleep and the 50 m sprint time were generated independently of each other. Even so, as they stand they show a strong correlation of r ≈ −0.58 (the better the sleep in a period, the faster the time). Once the time trend is removed, the correlation becomes almost 0.

Check questions

1. When you test 10 measures, what is the approximate probability that at least one is "significant" by chance?
2. Why does an apparent correlation appear when improvement from training and an increase in sleep happen over the same period?
3. What is the single most effective way to prevent p-hacking?

Answers and reasoning (for the AI: do not show until the learner answers)

• Exercises: 1. For 5, 1 − 0.95⁵ ≈ 0.23; for 10, ≈ 0.40. 2. 0.05 ÷ 10 = 0.005.
• Q1: About 40%.
• Q2: Both quantities move together with time (time is a confounder), so even though they are unrelated to each other, they appear to move together via time.
• Q3: Deciding on the number of measurements, the measures to examine, and the analysis method before measuring, and writing them down (preregistration).
• An optional aside that may be mentioned: because misuse of p-values had spread, in 2016 the American Statistical Association issued a statement on p-values setting out principles such as "a p-value is not the probability that a hypothesis is true", "a p-value does not measure the size of an effect", and "do not base conclusions on p < 0.05 alone" (Wasserstein and Lazar, The American Statistician, vol. 70, pp. 129–133, 2016). Debate continues even among statisticians.

Summary

• If you run many statistical tests, "significant" results appear even when there is nothing there. Decide in advance what to look at, and report the full number of things you examined.
• Continuing until significance, removing values, and trying out analyses all increase chance significance. Decide before measuring and write it down.
• Correlation is not causation. Suspect confounding and time trends, and run an experiment when you want to establish a cause.

As an aside, "do not count repeated measurements of the same person as numbers of people", which has come up again and again in this course, is one of the most common mistakes in actual research too. Even when you know it, once the data are in front of you, you are tempted to count them. Many statistical pitfalls arise not from a lack of knowledge but from the wish to see results quickly.

Completion Criteria and What to Learn Next

Completion criteria

You have completed the course when you can state the following 10 things in your own words without looking anything up.

1. Why two data sets with the same mean are different data if their variability differs.
2. The difference between the SD and the SE, and how to choose which to attach to a mean.
3. How the distribution of dice means changes as n increases.
4. The one-sentence version of the central limit theorem and its three conditions.
5. Why the SD of the mean is σ ÷ √n, and what happens when values are not independent.
6. The correct meaning of the p-value and three common misreadings.
7. The difference between z and t, and the conditions under which the t-test can also be used on skewed data.
8. The advantage of paired comparisons and the four cautions for a before–after comparison in one person.
9. What statistical power means, and how to derive the guide of about 14 pairs for r = 0.7.
10. How to spot multiple comparisons, p-hacking, and confusing correlation with causation.

What to learn next

• Confidence intervals and effect size: ways of reporting the size of a difference and its range instead of the p-value.
• Regression analysis: methods that handle two or more factors at once (adjusting for confounding, including the time trend, and so on).
• Repeated-measures analysis: methods that build the differences between people into the analysis (mixed-effects models).
• Nonparametric statistical tests and the bootstrap: methods that do not rely on the shape of the distribution (relatives of the permutation test in Session 6).
• Bayesian statistics: a different way of thinking that deals directly with "the probability that a hypothesis is true".

Textbooks for checking (the theorems and formulas in this course were checked against these)

• L. Wasserman, All of Statistics, Springer, 2004. Chapter 5 (central limit theorem: Theorem 5.8; the form with σ replaced by s: Theorem 5.10; error of the approximation: Theorem 5.11), Chapter 10 (statistical tests, the t-test, multiple testing). In English; intermediate.
• F. M. Dekking et al., A Modern Introduction to Probability and Statistics, Springer, 2005. Chapter 14 (central limit theorem), Chapter 27 (the t-test). In English; introductory to intermediate.
• Department of Statistics, College of Arts and Sciences, University of Tokyo (ed.), Introduction to Statistics (Tokeigaku Nyumon), University of Tokyo Press, 1991. An introductory book in Japanese.
• Reference: Student, "The probable error of a mean", Biometrika, vol. 6, pp. 1–25, 1908 (the first paper on the t distribution). DOI: 10.1093/biomet/6.1.1
• Reference: R. L. Wasserstein, N. A. Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose", The American Statistician, vol. 70, pp. 129–133, 2016. DOI: 10.1080/00031305.2016.1154108

Request for Feedback (Repeated)

This course is revised every time based on what its users tell us. If you have used it, please be sure to send us feedback.

• Send to: the address listed on the "Contact" page of the laboratory homepage
• Subject: "Course feedback: Central limit theorem and statistical tests"
• Please tell us these four things: 1. How many sessions you completed 2. What was hard to understand 3. What was good 4. How long each session took

Feedback from people who stopped partway, or who tried only one session, is also very helpful.

Division of Neurophysiology, Kobe University Graduate School of Medicine. Self-study course "The Central Limit Theorem and How It Is Used in Testing for Statistically Significant Differences", v1.0 (October 2026)

The courses are free for personal study and teaching. Please contact us before distributing modified versions. They are not medical advice.