Chapter 5 Lab 5: Fundamentals of Hypothesis Testing

The null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation. Every experiment may be said to exist only to give the facts a chance of disproving the null hypothesis. —R. A. Fisher

5.1 Overview

From here on out in the labs we will be focusing on making sense of data from experiments. In all of this, we use experiments to ask a question about whether one thing causes change (influences) another thing. Then, we look at the data to help us answer that question. In general, we expect to find a difference in our measurement between the conditions of the experimental manipulation. We expect to find a difference when the manipulation works, and causes change in our measure. We expect not to find a difference when the manipulation does not work, and does not cause change.

However, as you well know from reading the textbook, and attending the lectures. Experimental manipulations are not the only thing that can cause change in our measure. Chance alone can cause change. Our measures are usually variable themselves, so they come along with some change in them due to sampling error.

At a minimum, when we conduct an experiment, we want to know if the change we observed is bigger than the change that can be produced by chance. Theoretically, random chance could produce most any change we might measure in our experiment. So, there will always be uncertainty about whether our manipulation caused the change, or chance caused the change. But, we can reduce and evaluate that uncertainty. When we do this, we make inferences about what caused change in our experiments. This process is called statistical inference. We use inferential statistics as tools to help us make these inferences.

In this lab we introduce you to foundational concepts in statistical inference. This is also commonly termed hypothesis testing. But, for various reasons using that language to describe the process is tied to particular philosophies about doing statistical inference. We use some of that language here, so that you know what it means.

5.2 Excel

5.2.1 Goals

In this lab, we will use Excel to:

  1. Calculate difference scores between pairs of measures
  2. Conduct a sign test on this data
  3. Create an excel sheet that will make calculating future sign tests much easier

5.2.2 Load the data

The data we will examine today is from a fictitious experiment based on the infamous Coke vs. Pepsi Taste Test Challenge. 20 blindfolded participants were presented with two cups of soda (Coke and Pepsi). Presentation sequence was randomly determined, and participants were required to rate the taste of the soda on a scale from 1-5.

First, let’s open the relevant data file; Here is the link; it is called PepsiCoke.sav. The “.sav” file extension means that this data is formatted for a different software (SPSS), so the first thing we will have to do is copy the data to a blank excel workbook

  1. Open Excel
  2. Select “Blank Workbook”
  3. Open “PepsiCoke.sav” in SPSS
  4. Highlight the two columns of data by clicking the left most column label with you mouse, and draging right until both columns are selected.
  5. Press ctrl+c to copy this data
  6. Switch to excel, and click cell A2
  7. Press ctrl+v to copy this data to excel
  8. Label each column in row 1.

5.2.3 Calculate difference scores between pairs of measures

Now for each participant let’s calculate, the difference between the rating they gave for Coke vs. Pepsi. We will use SPSS to create a new variable (in a new column) containing only difference scores.

  1. In cell C1 enter the label “differences”
  2. In cell C2 enter:

=A1-B1

  1. Copy this formula downward until all difference scores are calculated

5.2.4 State the hypotheses

The Alternative hypothesis

H_a_ = People prefer Coke over Pepsi or people prefer Pepsi over Coke

The Null Hypothesis:

H_0_ = People prefer Coke and Pepsi equally.

5.2.5 Determine the observed value

We will now convert these difference scores into a binomial variable, which is required for a sign test (“Pluses and minuses” or “Ps and Qs” or “Heads and tails”)

  1. In cell E1 write the label “Observed value”
  2. In cell E2 enter:

=COUNTIF(C:C,“<0”)

  1. Press enter

This formula will count the number of difference scores that are less than zero, or in this case, the number of people who preferred Coke more than Pepsi (16) (also known as the observed value). In the future, you can change the < to a > if you would rather count the number of positive differences.

5.2.6 Calculate the 2-tailed significance

  1. in cell D1 enter the label “N”
  2. in cell D2 enter the formula:

=count(C:C)

  1. In cell F1 enter the label “Probability”
  2. In cell F2 enter “.5” (Because the chance of someone randomly preferring either Coke or Pepsi is .5 (50/50))
  3. In cell G1 enter the label “sig” (short for significance)
  4. in cell G2 enter the formula:

=2*MIN(BINOM.DIST(E2,D2,F2,TRUE),1-BINOM.DIST(E2,D2,F2,TRUE))

When you press enter on this final cell, you will have your two-tailed significance.

Since our 2-tailed significance (0.0026) is less than our standard alpha (.05), we reject the null hypothesis.

5.2.7 Reporting and Interpreting your result

To properly report a sign test, we use the form:

s(N)= observed value sig<>alpha

or in this case:

s(20)=16 sig<.05

To interpret this result, we might say:

In our sample of 20 subjects who all provided blind taste test ratings of a Pepsi and a Coke, 16 preferred Coke. This result was found to be reliably different from chance using a sign test analysis, s(20)=16 sig<.05. This result supports the idea that people do prefer Coke to Pepsi, and that our result was not due to sampling error.

5.2.8 Calculating a sign test without difference scores

What if you aren’t given a set of paired scores? What if your professor asks you to perform a sign test on a single outcome statement? For example: Test the hypothesis that a coin is weighted if, out of 15 flips of the coin, it lands on tails 13 times.

Notice that in this problem, there are no pairs of scores. We’re essentially only given the number of signs: out of 15 flips, the coin landed on tails 13 times (you can think of this as 13 pluses) and on heads obviously twice (you can think of this as 2 minuses).

In this case you can simply enter the number of “pluses or minuses” into the “observed value” cell of the excel sheet we create, and update the value of “N”. Excel will then automatically update its calculated significance value

5.2.9 On Your Own

  1. In this lab, you will be conducting a sign test to determine whether a coin is weighted. Individually or in small groups, take a coin out of your pocket. If working in groups, use only one coin for the whole group. Now, flip the coin 25 times. Write down how many heads and tails you observed.

  2. Enter this into your excel spreadsheet and conduct a two-tailed sign test using an alpha level of .05. What is your result?

  3. Have students in the class (or groups) announce their results as well. Did anyone have a trick coin? Do you think some of the coins used were actually weighted? Why or why not?