What is the Meaning of Probability?

What is the meaning of probabilistic statements; how do we interpret the statement that the probability of a flipped coin coming up heads is $1/2$? There are two interpretations. The most commonly used interpretation is that the probability of an event is the expected frequency that the event occurs. In other words, if you were to do an experiment many times, the average number of times which you get a particular result is the probability of that result (more strictly, this ratio tends to the probability of the result in the limit of a large number of experiments). Probability used in this way is called objective probability, as it is based on the view that probability can be measured objectively via experiments. This is also called the frequency interpretation. The objective interpretation is appropriate if there is a definite probabilistic process which generates the outcomes. If you do a number of experiments, the number of times an outcome occurs divided by the number of experiments is an estimate of this true probability of the process. The estimate approaches the true probability as the number of experiments goes to infinity.

For example of coin flipping, you may believe that the process is fundamentally random with fixed probability. To estimate the probability of a coin coming up heads, flip it 100 times, say, and take ratio of the number of heads to the total number of flips. For any particular trial, this would not come out exactly $1/2$. Sometimes it would come out $52/100$, sometimes $49/100$, and so forth. However, as the number of flips got bigger and bigger, this ratio come closer and closer to $1/2$, if that is the true probability.

Objective probabilities can be computed as frequencies. A common situation is one in which you would like to compute the probability of a particular outcome $x_1$. Suppose there are $n$ possible outcomes, all equally likely, and $n_1$ possible ways of getting the outcome $x_1$. Then

$$P(x_1) = \frac{n_1}{n}.$$

Example:

What is the probability of rolling a pair of dice so that the dots on the upward faces total to 7?

There are 36 different rolls of the dice. There are 5 ways to total to 7. Thus, assuming that each outcome is equally likely, the probability is 5/36.

There is a second interpretation of probability, called subjective probability. It will not be used much in this course, but you may see it in later courses on artificial intelligence. This interprets probability as degree of belief. In this interpretation, when I say that the probability that a flipped coin will come up heads it $1/2$, I am not describing the results of any experiments (necessarily). I am saying that I believe that the coin is equally likely to come up heads as tails. This interpretation makes sense of situations in which there may be no way of getting an estimate of a ``true'' probability by repeated trials. As examples, a doctor may say that the probability of a treatment curing you is 80% or a meteorologist may say that the probability of rain tomorrow is 60%. It is not possible to generate many sick copies of you and measure the fraction cured, nor are we able to generate many copies of today and measure the frequency of rainy tomorrows. These experts are telling you something about the strength of their belief in a given outcome.

You may wonder why belief should behave like frequencies. Why should the same rules of calculation be used for belief as for objective probability? There is an argument which says that it is sensible for beliefs to behave this way. Take the case of the doctor. The doctor may require a very accurate measure of the likelihood of the outcome, because decisions will be made on the basis of it. For example, the amount of time and money will the doctor be willing to spend on treating you may depend on this likelihood. Thus, the doctor's beliefs may be measurable from this behaviour. If the doctor's beliefs satisfy the laws of probability, the doctor may spend money and time sensibly; if the beliefs violate basic probability rules, the doctor might not use time and money sensibly. (This notion is usually stated in terms of betting behaviour. The doctor's belief's can be determined by setting betting odds with the doctor; if these beliefs violate rules of probability, you can design a set of bets which will cause the doctor to lose money no matter what the outcomes. This is a so-called Dutch Book. See the text ``Artificial Intelligence'', by Ian Pratt, for a good exposition of this view.) Thus, probability theory gives a formalism for quantifying beliefs in a sensible way. More than that, it gives a way of combining beliefs with objective evidence.

Viewed this way, Bayes rule can be taken as a way of updating beliefs due to new evidence. If $b$ is a belief in some classification or prediction, etc and $x$ is the evidence, Bayes rule is expressed as

$$P(b \vert x) = P(x \vert b)P(b)/P(x).$$

The prior, $P(b)$ is your prior belief, that is, what you think the likelihood of each possibility is prior to receiving the evidence. This formula tells how you should change your belief after receiving the evidence $x$. What might seem to be nice about this is that you can give a computer or an automated system ``beliefs'', and those beliefs can change based on new evidence. Whether humans actually update belief in this way is another matter. Whether humans should update belief in this way is another matter again.