Basic Probability

Do not print the lecture from this location.

Click here for a printable version of this lecture.

This is the text of the in-class lecture which accompanied the Authorware visual graphics on this topic. You may print this text out and use it as a textbook. Or you may read it online. In either case it is coordinated with the online Authorware graphics.

1. To find a topic which interests you: Look at the map of menus above. Choose a menu that interests you. Notice that the menu buttons have topics printed on them. Click on any button (topic) on the menu; you will jump directly to the text that corresponds to the topic printed on the button.

2. To coordinate this web page with Authorware presentations: The corresponding Authorware program should already be open. Go to the menu of your choice in the Authorware program and click any button which interests you. Then on the topic locator map above click on the same button on the same menu; you will jump to the text that corresponds to the Authorware presentation.

Today's lecture is about very basic probability. Probability can get very complicated. However for the purposes of this class, I will present probability theory in as simple a manner as possible.

Let's review the idea of the probability number. You all know the probability number already. For instance, you've been expressing things like the probability of this or that is one half or one fourth or .7, but now let's discuss this idea more formally. In this culture probability is a standard concept and almost everyone has some understanding of it, just by being in the culture.

The probability number, p, has a range between zero and one. The lowest possibility probability is zero and the highest possible probability is one. If you're ever working on a probability problem and you get a probability value outside that range, then you'll know right away that you've made a mistake. There are no probabilities outside of the range from zero to one.

The probability of the impossible event is zero. So if an event can't happen, that is, if it is impossible, then its probability is zero.

Frequently, the impossible event is symbolized by phi, the little Greek letter ( i ).

In contrast to the impossible event is the sure event,often symbolized by S. S also stands for sample space which we'll talk about later.

The probability of the sure event is one. So if an event must happen, than its probability is one.

The probability number has been used in many ways. Often people don't make conceptual distinctions between them. To make these distinctions clearer we are going to look at four different interpretations of the probability number.

These are only interpretations of what the probability number means. In every day life, people interpret probability in many different ways, depending on what sort of practical problem they have in mind.

These interpretations have nothing to do with the mathematics of probability theory itself.

The four distinctions we will make are the long run relative frequency, subjective estimates, equiprobable sample spaces, and areas under curves.

One way to interpret what this probability number means is what people call long run relative frequency. The phrase "long run" is an important part of this interpretation.

Sometimes relative frequency is called proportion. You've probably used the word proportion in your life, but relative frequency is generally less familiar.

So let's take an example which is common in our culture. Imagine there is a baseball player and he gets 118 hits in 342 "at bats". We're a ways into the season and he has batted 342 times. He's gotten a hit 118 times. Another way of saying this is the frequency of at bats is 342 and the frequency of successful at bats, that is the frequency of hits, is 118.

Relative frequency is defined by taking the total hits and dividing them by the total at bats. Using this process you discover that relative frequency is .345. Or you could say that the proportion of total at bats that resulted in a hit is .345. Or simply, the proportion of hits is .345

People who are involved with playing baseball tend to multiply that relative frequency or proportion by a thousand and say the batting average is 345.

Regardless of whether there ever were something called probability theory or not, people would be calculating proportions or relative frequencies. People would be getting batting averages and shooting percentages and all of those kinds of things that we calculate.

But one thing that people do is interpret long run relative frequency as probability. Now the baseball player on our slide is female. Her relative frequency of hits is .345.

It would be a common interpretation of the probability number to say that her probability of getting a hit is .345.

So lets say that 342 at bats is long enough to be acceptable as "long run." Since her relative frequency is .345 we culturally and easily interchange relative frequency and proportion with the idea of probability. It is unlikely that anyone would even notice if someone said "She has a 345 batting average." And somebody else said, "Yeah, her chances of getting a hit are .345."

Most of us go back and forth between probability and relative frequency quite fluidly, but what I'm pointing out is the CONCEPTUAL distinction between the two. Long run Relative Frequency is empirical, it is based on our observations of the world. Probability is theoretical; it is a part of measurement theory in mathematics.

There's data on this batter. She's hit the ball 118 times out of 342 at bats resulting in a a relative frequency of .345. And it's only an interpretation to say that the probability of her getting a hit is .345.

Notice that I have mentioned long run relative frequency is interpreted as probability. The reason for this is because short run relative frequency is so changeable and volatile. For example, imagine that it is opening day in the baseball season. Somebody comes to the plate and they get a hit. Now they've been at bat once and they have one hit. Therefore, the relative frequency of hits to at bats is one.

People with any experience in sports know that this does not mean that this player's getting a hit is a sure event. It does not mean that the player's probability getting a hit is 1.

VOCABULARY. When we define things that have probabilities we're generally going to call the things that have probabilities "events."

In the baseball example the event that we were discussing was a hit. P(Hit) = .345.

So the word "event" generally is used to mean something specific that we're talking about and that has a probability. When we're speaking abstractly about probability then we don't talk about hits or other specific events. Rather, we say really vague things like the probability of event A or the probability of event B or the probability of event C. As we discuss more abstract and lawful principles in probability, we won't be talking about specific events such as hits, but speaking about general events such a A, B, C and D.

So in the baseball example we might ask what is the probability of A, where A is defined as a hit. In this sort of discussion, we would say the probability of A is .345 or P(A) = .345

In the relative frequency interpretation, the probability of event A, or P(A), is what we expect the long run relative frequency of A to be. It also allows us to estimate probabilities from data showing long run probabilities.

A lot of times you'll hear people say things like, "What do you think the chances are that Bill's going to come to our BBQ?" and someone will say, "Well, the chances are .5", or the probability is .5 that Bill will come to our BBQ.

So probability is also used to express a subjective feeling that we have about events occurring. This subjective feeling may be, and often is, in contradiction to data (such as empirical long run frequencies). Someone can have a batting average of 125, which means they're getting a hit about one out of eight times, and the manager will put them in as a pinch hitter, because for some reason the manager's subjective probability is that this person's going to get a hit this time.

Subjective probability may be vastly higher or lower than the actual empirical relative frequency. Sometimes for some internal reason we will think, "Oh, the chances of that are nil." Or, "The chances of that are almost certain".

Subjective estimates are a very different use of the probability number than is relative frequency.

For most people the term equiprobable sample space is an unfamiliar term. For that reason, I am going to use a specific example that most people have had experience with. This experience is rolling a single, six-sided die, with the numbers one through six indicated on the six sides.

You roll the die, and it flies through the air and then it bounces and clicks on some hard surface and it continues to roll and click until it eventually stops.

Next we decide how to measure that process. In our culture we typically count the number of dots facing up when the die comes to rest.

But it should be noted that that is only one way to measure the roll of a die. As we will discuss later, this event could be measured by counting the number of clicks, by measuring the amount of time it takes the die to come to rest, and so on. But for our usual games and for simple probability theory those ways of measuring the event don't serve well.

SAMPLE SPACE: The sample space, often symbolized by a capital S, is all the possible outcomes that can result from a process. The sample space includes everything that can happen. In this specific case of the roll of a die, the sample space is the set: {one, two, three, four, five, and six}. For our purposes, the Sample Space can be defined as the list of all possible outcomes of a process.

You'll notice that S is the same as the sure event. When you roll a die it a sure thing that you'll get a one, two, three, four, five or a six. The sample space includes all possible outcomes, therefore the probability of the sample space is equal to one.

EQUIPROBABLE. On a fair die, all the sides of the cube are equally likely to land facing up. Therefore, all of the outcomes in the sample space are equally likely, or equiprobable. This means it is an equiprobable sample space.

In an equiprobable sample space every single outcome that can happen has the same chance of happening as every other outcome. In this particular case, the probability of a "one" is equal to the probability of a "two" is equal to the probability of a "three" is equal to the probability of a "four" and so forth. The probability of each of these outcomes is one sixth (1/6).

PROBABILITY DISTRIBUTIONS. A probability distribution is a visual way of representing or looking at probabilities. The probability distribution for the roll of a die is shown on the right side of the graphic above. Along the horizontal axis are all the possible outcomes from one to six. We've noted already each outcome has a one sixth probability. On the graph, the height of the little bar above each of the numbers from one through six represents the probability of that number occurring. Of course, they all have the same height because they all have probability, one sixth.

When we represent all of the possible outcomes, each with its probability, in a picture like the one above, we call it a probability distribution. Notice that all the probabilities (the heights of the bars) must add up to 1.

Another way to interpret the probability number is the amount of area under a curve. The particular curve that's shown on this graphic is the normal probability distribution, and we'll have a lot to say about that in another lecture. For now, notice that it is a bell shaped curve. Notice that on the top probability distribution that none of the area is shaded and so the probability of the shaded area is zero because there's no area that's shaded. In the middle probability distribution, all of the area is shaded red, and so the probability of the shaded area is equal to one. On the bottom distribution, we have an example where you can see that half of the area is shaded red and so the probability of falling into that area is one half or .5. One way of representing the probability number is as an amount of area underneath any well-defined curve.

We'll learn more about this later. For now just become familiar with the terminology.

Independence is an important idea in statistics. When two events are independent, they are not related to each other in any way. Another way of saying this is that they share no information about each other. Knowing something about one of them doesn't tell you anything at all about what's going on with the other event. There is no shared information.

For example, I've not investigated this but I assume the following are independent events. Event A is the test score of a randomly selected student in statistics in Salt Lake City, Utah. Event B is the temperature on a randomly selected day in Sidney, Australia. Those are, at least presumably, independent events.

If you took a mid-term statistics test, and you were wondering what your score was, and somebody was on the web and looking up temperatures, and said, "Oh, the temperature in Sydney was 87 on May 7th last year." You would probably say "Well, is that supposed to help me with knowing what my statistics score is?"

The point here is that these two events share no information about each other, knowing the temperature in Sydney, Australia does not tell you anything about your statistics score. Conversely, if someone wanted to know the temperature in Sydney, Australia and you told them the score you got on the statistics exam, you wouldn't be telling them anything about the temperature in Sydney, Australia. When two events share no information about each other, we consider them independent.

The idea of independence is very important in science because we're often trying to figure out if something is causing something else. If one thing causes another then those two things are sharing a lot of information with each other. If first happens the other must happen. If two things don't share any information about each other, they're independent, and they can't be in a causal relationship. So statistically, the idea here is that independent events are not related or they share no information about each other.

Let's go on to an important relationship in statistics called the independence product rule. The independence product rule states that when two events are independent, the probability that they both happen is simply the product of their probabilities.

P(B). Let's consider an example using the long run relative frequency interpretation of probability. Suppose that the probability the Sydney temperature is above 95 degrees is .08. The probability was calculated by looking at the Sydney temperatures over the last hundred years. From this we determine that, on average, 30 days out of a year of 365 days are over 90. The relative frequency of days over 90 in Sydney is 30 over 365, or .08. So the probability of the temperature being over 90 is .08.

P(A). Okay now, suppose that in the last several years, 900 of 1000 students passed their statistics exam. The relative frequency of passing is .9. Using the relative frequency interpretation of probability, the probability that a student passes the statistics exam is .90.

The probability that Sydney is over 95 degrees is .08, the probability of a pass is .9.

JOINT OCCURRENCE OF A and B. When two events BOTH occur, we often refer to that as the "joint occurrence of A and B." Often we are interested in the probability of them both occurring. We write this in symbols as P( A and B) or simply P(AB).

Now imagine that we have no idea what day of the year it is, or who the student is, or anything like that. What is the probability that both Event A and Event B will happen?

We know the probabilities of each event separately. The probability of over 95 is .08, or in symbols, P(B) = .08. And the probability of a pass is .9, or P(A) = .9. But what's the probability that BOTH a certain student passes AND a randomly selected day in Sydney is over 95? That is, what is P(AB)?

By the independent product rule, if two events are independent, then the probability of them both happening is simply the product of their probabilities. So the probability they both happen is .08 times .9, or, as you can see from the graphic above, .072. P(AB) = .072.

In general terms, if event A is independent of event B, then the probability of both A and B is simply equal to the probability of A times the probability of B.

Conversely, if you can demonstrate that the joint probability of A and B is equal to the product of their individual probabilities, then the events A and B must be independent.

This second equation underlies the statistical approach to discovering independence (or a lack of relationship) between variables or events.

Using empirical long run relative frequencies, if the data show that the probability of AB is equal to the probability of A times the probability of B, then A and B must be independent.

WHEN EVENTS ARE RELATED. This product rule is NOT true if the events are not independent. That is, if A and B somehow depend on each other, then P(AB) is NOT EQUAL to P(A)P(B).

Some of the most interesting parts of probability theory address P(AB) for non-independent events. This might be covered under a topic like Conditional Probability. For our purposes in this basic class, we will not go into these topics.

Let's apply the idea of independence and the independence product rule to a few common examples. Let's talk about flipping coins. At first we will flip one coin. By convention, the world over, people do this with their metal money and in our culture we call one side of the coin heads and the other side of the coin tails.

SAMPLE SPACE. When we flip a coin only two outcomes can happen. We can get a head, that is, the coin lands with the head facing up. Or we can get a tail, that is, the coin lands with the tail facing up.

The sample space consists of a head and a tail. That is, S = {Head, Tail}, or even more simply, S = {H,T}. That's the complete list of possibilities.

There is only one way to get a head out of two possibilities, so (with a fair coin) the probability of a head is one-half or .5. And the probability of a tail is the same thing, one half, or .5. I am sure you know all this but we are making these things formal and explicit. It's best to learn a new jargon using content you already know.

With a fair coin the sample space is equiprobable. S = {H, T} and P(H) = P(T) = .5.

Bernoulli trials refer to any very simple process that can result in only two possible events. In other words the process can have only two outcomes. If we flip a coin, there's only two things that can happen, a head or a tail.

Another example of a Bernoulli trial is gender in random sample of human beings from some large population. There are only two possible outcomes, male and female. A Bernoulli trial is a process that can result in just two outcomes.

Traditionally in Bernoulli trials, one of the two possible outcomes is called a success, and the other is called a failure. In this context (learning about the binomial distribution and Bernoulli trials) these terms don't carry any evaluative significance. Normally in our culture, success and failure have a lot of evaluative meaning. However in this context they are only general labels to indicate the two outcomes.

In a coin flipping process, we can call the occurrence of a head a success and the occurrence of a tail a failure. Another way of discussing the probability of a head is to call it the probability of a success which is .5. The probability of a tail, also called the probability of a failure, is also equal to .5.

Continuing the jargon, generally the probability of success is denoted by a small p. The probability of a failure is denoted by a small q. Since there are only two outcomes, and the total probability in any probability system must always be one, then p + q must be equal to one.

Since the statement p + q = 1 must always be true, then if someone gives you p, you always know q. q is equal to one minus p. This knowledge will be useful later when we develop more complex ideas.

The graphic shows two coins, the first coin is gold and the second coin is silver so we can tell the difference. Now we can ask probability questions about the outcomes of two flips instead of just one.

TWO EQUIVALENT CASES. We can flip one coin twice or we can flip two coins as shown on the graphic. The probability discussion is equivalent for both cases. We can think about the independence or non-independence of the two flips.

INDEPENDENCE. Within the tradition of western civilization, we make the strong argument that the outcome of the first flip, either a head or a tail, has no effect on the outcome of the second flip. In other words, there's no information in the first flip about the second flip. The first flip is independent of the second flip.

GAMBLER'S FALLACY. Sometimes people's intuition disagrees with the assertion that two flips of a coin are independent. Their intuition is called the gambler's fallacy. The gambler's fallacy is the belief that if you flip a head, there's a better chance of the tail on the next flip. People are even more likely to assert this if they flip three heads in a row; then they tend to think "the tails have got to come up now."

Two flips of a coin (or one flip each of two coins) are independent processes. With a fair coin flipped twice, the probability of a head is one half and the probability of a tail is one half. The history of the coin has nothing to do with it.

With flip of two coins, for each the probability of a head is one half and the probability of a tail is one half. The behavior of other coins in the universe has nothing to do with it. Flipping a coin multiple times, or multiple coins once each, are independent events.

Now let's find the probability of getting two heads when we flip two coins (or of getting two heads when we flip one coin twice).

The P(A) and the P(B). So imagine that we flip two coins, what's the probability of getting a head on the first coin? The probability of a head is .5. So what's the probability of getting a head on the second coin? It is the same thing; the probability of a head is .5.

P(AB). What's the probability of getting a head on the first flip AND a head on the second flip? That is, what's the probability of head AND head?

Using independence product rule, you should start being able to calculate that yourself. The two flips are independent events so the probability of them both happening is just a product of their probabilities.

Either figure it out in your head or write down in your notes what the probability of two heads is.

Okay, the probability of two heads is simply the probability of a head times the probability of a head, which is .5 times .5, which is .25, or one fourth.

Just to repeat everything -- if we got two heads on the flip of two coins, a head on the first flip and a head on the second, then the probability of two heads in two flips is .25, because the events are independent.

Suppose we flip three coins, which is the same as if we flip one coin three times. It makes no difference as far as the probabilities how we physically set up the experiment. The probability of a head on the first is .5, on the second its .5, and on the third its .5.

The probability of a head is always .5 irrespective of the past behavior a single coin (or irrespective of the behavior of other coins, if we have three coins).

We know that the events are all independent of each other, so what's the probability we get a head on the first flip, a head on the second and a head on the third. In other words, what's P(HHH)?

The coin flips are independent, and so the probability of three heads is simply the probability of a head times the probability of a head times the probability of a head. And that's equal to .125 or one eighth.

In summary, the flips are independent events, the probability of three heads is .125.

Carrying this out to its logical conclusion, you can see that the probability of four heads is just .5 times itself 4 times, which is .0625. That is one sixteenth (1/16). The probability of five heads is .5 times itself 5 times, and that's .03125 or 1/32.

We will use this knowledge to make important arguments underlying the theory of inferential statistics much later in the course.

Now we are a going to use as an example drawing cards from a standard deck. Some cultures play different card games and others don't play cards at all, so let's define what I mean by a standard deck.

In a standard deck of cards there 52 cards in the deck. There are four suits: hearts, diamonds, clubs, and spades. There are 13 cards in each suit. Two of the suits are black; two of the suits are red. So we are discussing the parameters of a basic, standard deck. Each of the 52 cards are all possible outcomes of a random draw from a shuffled deck. So the 52 cards constitute the sample space.

If we shuffle the deck well and draw a card at random, every card has the same chance of being drawn. So this is another example of an equiprobable sample space.

Imagine we randomly draw one card and we draw the king of hearts. What's the probability of drawing the the king of hearts?

P(K of H) = (1/52). It is one card out of 52 possibilities, so the probability of the king of hearts is one over 52 or .01923. That is a little less than two in one hundred.

Now imagine that we put the first card back in the deck, reshuffle the deck, and randomly draw another card.

It is going to be the same thing, one over 52. The probability then of any single card is one over 52 because we have an equiprobable sample space.

Let's ask a little more complicated question. What's the probability that the card is an ace?

In this deck we have four aces. So Event A occurs whenever we draw one of the aces. Event A means that the draw resulted in an ace of clubs or an ace of diamonds or an ace of hearts or an ace of spades. If any of those four cards is drawn, we say that Event A has occurred.

There are four aces out of the 52 cards, so P(A) is 4 divided by 52, or one thirteenth (1/13).

Let's define Event B as a black card. What's the probability of a black card? What's P(B)?

If this is new or long ago forgotten material, it would be a good time to find P(B) for yourself before you go on, because then you'll be actively processing and learning more.

Now we are going to consider two draws from the deck and look at joint probabilities.

What do we mean by two draws with replacement? Here's the operational definition or the procedure. You shuffle the cards and randomly draw one card from the deck, make a record of the first card, and then replace the card back into the deck. Replacing the card is very important for the process here. For the second draw, you need to reshuffle the cards, and randomly draw one card from the deck and make a record of the second card drawn.

If you don't replace the first card in the deck, the two events are no longer independent. For example if I draw the ace of spades on the first draw and I don't replace it, what's the chance that I'm going to draw an ace of spades on the second one? It is zero. If you don't replace the cards that you use, the those cards have information about probabilities in the second draw. To be explicit, if I know that someone has drawn the ace of spades on the first draw and didn't replace it, I can make a perfect prediction that the ace of spades won't be drawn on the second draw since it is no longer in the deck. Moreover, the probability of any card is changed because without replacement there are only 51 cards in the deck for the second draw.

In other words, I have to replace the first card in order for the second draw to be independent of the first draw. If there is replacement, the outcome of the first draw is independent of the outcome of the second draw. What you get on the first draw has no impact whatsoever on what you get on the second draw.

As an illustration we randomly draw one card, and we get the king of hearts. The probability of that is one over 52, as we've seen before. P(A) = 1/52.

After making a record of it we replace the king of hearts and reshuffle the deck. Then we select a second card and get the ace of spades. The probability of that is one over 52. P(B) = 1/52.

What is the Probability of a (king of hearts on the first draw AND an ace of spades on the second draw)?

We have said they're independent events so we know we just have to multiply their probabilities so that's one over 52 times one over 52

Let's redefine Events A and B. Let A be an ace on the first draw and let B be a black card on the second draw. We've already found P(A) = 1/13 and P(B) = 1/2. What is P(AB)?

The events, A and B, are independent because the outcome of the first draw is independent of the outcome of the second draw. We replace whatever we got on the first draw and reshuffle.

P(AB). The probability of an ace on the first draw and a black card on the second draw is simply the product of their probabilities.

This is equal to one thirteenth times one half, which is one over 26. If you divide 26 into 1, to get the more metric form of probability, you'll get close to .084.

If two events A and B are independent then you can get their joint probability, the probability that they both happen, by simply multiplying their individual probabilities.

This is one of the main conceptual points for you to take away from the probability.

1) We learned about the long run relative frequency interpretation of the probability number. This way of assigning the probability number is based on empirical information like the number of hits divided by the number of at bats, or the number of days over 90 degrees per year over the last 100 years.

2) We also talked about how people interpret probability subjectively. We have probabilities in our mind that are vaguely related to data and past experiences but they're not very tightly related to data. These are subjective interpretations of the probability number.

3) We learned about equiprobable events in a sample space. Examples of these are the roll of a die, the flip of a coin, and drawing cards from a deck. We used these examples to determine probabilities of events in an equiprobable sample space.

4) We also introduced the idea of probability as areas under curves. We worked very little with this interpretation of probability, but it will be the most important one of the four interpretations later in the class.

P(AB). The other idea that's important in this lecture is the independence product rule.

If A and B are independent,
then probability of AB is probability of A times probability of B.

If you can show that probability of AB is equal to
the probability of A times the probability of B,
then the two events are independent.

Most important, we reviewed basic probability, some symbols and notations, and made sure that you can follow simple discussions of probability when they come up in future lectures.