Lesson 14
Using Normal Distributions for Experiment Analysis
14.1: Notice and Wonder: Some Distributions (5 minutes)
Warmup
The purpose of this warmup is to elicit the idea that these distributions are symmetric and approximately normal, which will be useful when students study the origin of the distributions in a later activity. While students may notice and wonder many things about these images, the symmetry, shape, and center are the important discussion points.
Launch
Display the images for all to see. Ask students to think of at least one thing they notice and at least one thing they wonder. Give students 1 minute of quiet think time, and then 1 minute to discuss the things they notice and wonder with their partner, followed by a wholeclass discussion.
Student Facing
What do you notice? What do you wonder?
Student Response
Student responses to this activity are available at one of our IM Certified Partners
Activity Synthesis
Ask students to share the things they noticed and wondered. Record and display their responses for all to see. If possible, record the relevant reasoning on or near the images. After all responses have been recorded without commentary or editing, ask students, “Is there anything on this list that you are wondering about now?” Encourage students to respectfully disagree, ask for clarification, or point out contradicting information.
If the symmetry, shape, and center of the distributions does not come up during the conversation, ask students to discuss this idea.
14.2: A Theoretical Experiment (15 minutes)
Activity
In this lesson students begin to examine how their understanding of normal distributions can help to determine whether the results from an experiment are significant. For this activity, students recognize that the results from randomizing the groupings for data tend to be approximately normally distributed centered around 0. In the next activity, students will use the properties of normal distribution to determine the likelihood that the observed difference in means is due to the treatment rather than the groupings.
Students must reason abstractly and quantitatively (MP2) when the analysis leads them to look at the differences in means from possible groups created from the original experiment data.
Launch
Design Principle(s): Optimize output (for explanation); Maximize metaawareness
Student Facing
To see what might be happening when we regroup data, consider an experiment that takes 12 subjects and divides them into 2 groups at random. The control group contains 6 subjects and the treatment group contains 6 subjects. To explore what's possible, assume the control group results in the data: 1, 3, 4, 6, 8, and 10. The treatment group results in the data: 2, 5, 7, 9, 11, and 12.
 Find the difference in means for the original groups by subtracting the control group mean from the treatment group mean.

With a smaller data set like this, we can actually consider all of the different arrangements of the data. There are 924 distinct ways to separate the 12 values into 2 groups of 6. The frequency table shows all the possible differences in means and how often they occur. Notice that a difference in means of 4.33 occurs 7 times and a difference of 4.33 also occurs 7 times. The dot plot shows the same information.
What proportion of possible groupings have a difference at least as great as the difference in means for the original groups? Explain or show your reasoning.
difference in means \(\pm 6 \) \(\pm 5.67\) \(\pm 5.33\) \(\pm 5\) \(\pm 4.67\) \(\pm 4.33\) \(\pm 4\) frequency 1 1 2 3 5 7 11 difference in means \(\pm 3.67\) \(\pm 3.33\) \(\pm 3\) \(\pm 2.67\) \(\pm 2.33\) \(\pm 2\) frequency 13 18 22 28 32 39 difference in means \(\pm 1.67\) \(\pm 0.33\) \(\pm 1\) \(\pm 0.67\) \(\pm 0.33\) 0 frequency 42 48 51 55 55 58 
The proportion you calculate represents the probability that the original difference in means could be due to the groupings themselves. Based on the proportion you calculated for this situation, which description is most accurate? Explain your reasoning.

Because the proportion is so low, it is unlikely that the difference in means is due to the randomized groupings. This means that the difference in means is most likely caused by the treatment.

Because the proportion is not that low, it is still rather possible that the original difference in means is due to the random groupings. This means that there is not enough evidence to determine that the difference in means is likely caused by the treatment.

Student Response
Student responses to this activity are available at one of our IM Certified Partners
Anticipated Misconceptions
Students may be confused about how to interpret the frequency table with the \(\pm\) included. Remind students that the same data is displayed in the dot plot and to look particularly at the dots representing a difference of 6 and 6. The frequency of 1 in the table for the difference labeled \(\pm 6\) represents each of those differences.
Activity Synthesis
The purpose of the discussion is to help students understand that redistributing the data from an experiment into groups at random can help determine whether the original results are significant or due to the way the subjects were grouped.
Select students to share their solutions and reasoning.
Tell students that the use of statistics cannot prove that the results of an experiment are due to the treatment or groups. An analysis of this type can merely provide strong evidence that the treatment is the reason the original means are different or the analysis can tell us that there is not enough evidence to make that conclusion.
Consider asking:
 "Why do you think the distribution of differences in means is approximately symmetric? Why do you think the center of the distribution is 0?" (For every redistribution of data like 1, 2, 3 and 4, 5, 6 there is another that switches the two groups like 4, 5, 6 and 1, 2, 3. By a similar reasoning, for every difference of means like 3 there is another for the opposite like 3. So, the center of the distribution is 0.)
 "Suppose the original data from the control group was 1, 2, 3, 4, 5, 8 and the original data from the treatment group was 6, 7, 9, 10, 11, and 12. Based on the data alone, do you think there is a significant difference in the means from the group? The means are different by 5.33. Using the table, does the proportion of possible groupings with a difference of means that are at least 5.33 support your claim?" (I think there is a significant difference in the means. This is supported by using the table, since only 8 of the groupings area that extreme, so there is less than 1% chance (\(\frac{8}{924} < 0.01\)) that this grouping could have happened by chance.)
14.3: Simulating to Decide (15 minutes)
Activity
In this activity, students work through a case in which looking at all the possible combinations of regrouping experimental data is too much to consider. In many cases, even the simulations are too many to use to find exact proportions of mean differences to compare with the original difference in means. In these cases, we can use the approximately normal distribution of the simulations to estimate a proportion of redistributed data that would be at least as extreme as the original difference in means.
Students model with mathematics (MP4) when they use a normal distribution to model the simulated difference in means based on redistributing the data. Students must also select appropriate tools (MP5) to find the area under the normal curve for the analysis.
Launch
Provide access to devices that can run GeoGebra or other statistical technology.
Design Principle(s): Support sensemaking
Supports accessibility for: Conceptual processing; Visualspatial processing
Student Facing
Researchers want to know the effect of captively raising birds on the weight of the birds. The researchers begin with 100 birds divided into 2 groups of 50 each. One group of 50 will be raised in captivity and the other 50 are tagged and released into the wild. After 5 years, all 100 birds are collected and weighed.
There are more than \(10^{29}\) different ways to regroup the 100 birds into groups of 50 again, so looking at all the combinations would be too time consuming to reproduce. In this case, we can run simulations to determine how the original difference in means compares to those from regrouping the data.
The original groups have a difference of means of 0.27 grams. Researchers run 1,000 simulations regrouping the data into 2 groups at random and record the differences in means for the groups in each simulation. The histogram shows the differences in means from the simulations.
They determine that the mean of the differences of means from the simulations is 0.0021 grams and the standard deviation for the differences of means from the simulations is 0.112 grams.

What features of the distribution in the histogram let you know that modeling with a normal distribution is reasonable?

Model the simulations using a normal distribution with a mean of 0.0021 and a standard deviation of 0.112. What is the area under this normal curve that is more extreme than 0.27?

How can this area be used to compare the difference of means from the simulations to the difference of means from the original groups?

Based on the area under the normal curve, is there evidence that the original difference in means is likely due to where the birds spent the 5 years? Explain your reasoning.
Student Response
Student responses to this activity are available at one of our IM Certified Partners
Student Facing
Are you ready for more?
Suppose we decide that if the probability of observing our difference or a more extreme difference happens less than 5% of the time in simulations we will conclude that it was captivity that caused the difference, and if the probability is greater than 5% we will not draw any conclusions. In what ways could the conclusion we make (or decide not to make) be wrong?
Student Response
Student responses to this activity are available at one of our IM Certified Partners
Activity Synthesis
The purpose of this discussion is to show that areas under normal curves can be used to help decide whether the difference in means from an experiment are significant.
Select students to share their solutions and reasoning. To promote discussion, consider asking: "In this problem, a 1.5% chance of the difference in means from the original groups happening by random chance seems low enough to conclude that the difference in means is likely due to the treatment rather than the way the subjects were grouped. What percentage might make you change your opinion on whether the groupings were the cause of the original difference in means?"
Tell students that standard cutoffs used in statistics are 1%, 5%, or 10% before the results are called into question.
Ask students, "Why do you think it would be important to set your expectations for a cutoff before doing the experiment?" (Setting the expectations can help decide how many subjects to include in the study and does not unfairly bias the results. Some experimenters may continue to run new simulations until their results become significant.)
Lesson Synthesis
Lesson Synthesis
The purpose of the discussion is to help students understand what they can do to analyze the significance of results from an experiment. To promote discussion, consider asking:
 "What can you do to determine if the original difference in means is significant if the number of subjects in the experiment are large and finding all the different combinations of data is difficult?" (Use simulations to determine differences in mean for many possible groupings, then find the mean and standard deviation of those values to create a normal distribution. The normal distribution can then be used to estimate the likelihood of the original difference in means happening by chance.)
 "Why is a normal distribution a good model for differences of means from simulated regrouping of data?" (The differences in means should produce a distribution that is approximately symmetric and bellshaped which is a type of distribution that is usually represented fairly well by a normal distribution.)
 "What does the total area on either end of the normal distribution that is more extreme than the original difference in means represent?" (The area represents the approximate proportion of the regroupings that have a difference of means at least as large as the one from the original groups.)
 "If you want the analysis to show that there is evidence that the difference in means from the experiment is due to the treatment, would you want the areas on either end of the normal distribution to be large or small? Explain your reasoning." (Small, because the area represents the likelihood that the original difference in means is due to the way the subjects were grouped. If that likelihood is small, then the difference is likely due to another effect like the treatment.)
14.4: Cooldown  The Science Experiment (5 minutes)
CoolDown
Cooldowns for this lesson are available at one of our IM Certified Partners
Student Lesson Summary
Student Facing
To analyze the significance of the data collected from an experiment, a randomization distribution can be used. In some cases, where the number of subjects is small, all of the possible ways to regroup the data can be used to compare the original difference in means. When the difference in means is more extreme than most of the differences seen from the randomized regroupings (usually more than 90%, 95%, or 99% depending on the situation), we can say that we have evidence that the difference in means is due to the treatment rather than the way the subjects were originally grouped.
The more subjects included in the experiment, the greater number of possible regroupings. For example, 14 subjects divided into 2 groups of 7 can have their data redistributed into groups 3,432 different ways. When there are 60 subjects divided into 2 groups of 30, there are more than 118 quadrillion (\(1.18 \times 10^{17}\)) different ways to redistribute the data into groups of 30. This large number of ways to regroup the data makes looking at the distribution of every possible regrouping difficult.
In these cases, we often do a simulation and redistribute the data many times to get a sense of the true distribution of all possibilities. For example, this histogram shows the difference of means for 1,000 simulations of redistributing 60 data values into 2 groups of 30 each.
The simulations should produce approximately normal distributions with a center near 0. This allows us to use our understanding of normal distributions to estimate the proportion of regroupings that are at least as extreme as the original difference in means from the experiment. When the proportion is small enough, we should conclude that there is enough evidence to say that the difference in means from the original groups is most likely due to the treatment.
For example, using the values from the histogram, the mean is 0.04 and the standard deviation is 9.07. That provides enough information to create a normal distribution that models the data. In the image, we see the normal distribution and the regions for which the difference of means might be significant since there is only a 5% chance of the original difference in means being in the shaded region (less than 17.74 or greater than 17.82).
If the original difference in means is something like 20, then we can conclude that there is evidence to show that the difference in means is due to the treatment. On the other hand, if the original difference in means is something like 10, then we should say that there is not enough evidence to conclude that the difference in means is due to the treatment, since there is still a good chance that the difference in means is due to the way the subjects were originally grouped.