Combining Probabilities

If there are two variables, $X$ and $Y$, we might want combined probabilities involving both $X$ and $Y$. There are three ways of combining random variables.

Joint Probability:

First, we can ask for the probability that $X$ takes the value $x$ and $Y$ takes the value $y$. This is denoted $P(x,y)$ (also written $P(x \mbox{ and } y)$ or $P(x\cap y)$) and is called the joint probability of $X$ and $Y$. This is simply the probability that both things occur: $X$ is $x$ and $Y$ is $y$.

Conditional Probability:

Another quantity is the conditional probability that $X$ takes the value $x$ given that $Y$ takes the value $y$. This is denoted $P(x\vert y)$. This is the probability that $X$ takes the value $x$ assuming that $Y$ definitely takes the value $y$ (no matter how unlikely it is that $Y$ takes the value $y$).

The relationship between the two is

$$\fbox {$P(x,y) = P(x \vert y)P(y)$}$$

the probability of $x$ and $y$ is the probability of $x$ given $y$ times the probability of $y$.

Bayes Rule:

From the formula relating joint probability with conditional probability, Bayes Rule follows, since

$$P(x \vert y)P(y) = P(x, y) = P(y \vert x)P(x),$$

so,

$$\fbox {$P(y \vert x) = P(x \vert y)P(y)/P(x).$}$$

Note that if $X$ and $Y$ are independent, $P(x, y) = P(x)P(y)$, the joint probability is the product of the independent probabilities.

Example:

An example may illustrate the difference between the joint probability and the conditional probability. Suppose we pick an arbitrary British adult. We wonder about two things: is this person rich and did they attend public school? Thus, we have two variables:

• $X =$ the person is rich,
• $Y =$ the person went to public school.

The variables $X$ and $Y$ can each take two values, TRUE and FALSE. Presumably, the probability that the person is rich is small, as is the probability that they attended public school, because there are 10s of millions of British adults, few of them are rich, and few of them attended public school. The joint probability, the probability that the person is both rich and an alumnus of public school is smaller still, since that is the fraction of British adults who are both rich and public school grads.

Now, consider the conditional probability, $P(X=true\vert Y=true)$. That is the probability that a British adult is rich given the fact that they went to public school, i.e. the fraction of public school alumni who are rich. I would assume that this is closer to 1, adults who went to public school are likely to be rich (I assume this is true).

Probability of $X$ or $Y$:

There is a third way of combining $X$ and $Y$ that you should be aware of. That is the probability that $X$ takes the value $x$ or $Y$ takes the value $y$. For this the expression is

$$\fbox {P(x \mbox{ or } y) = P(x) + P(y) - P(x,y).}$$

Thus, if $x$ and $y$ are mutually exclusive, the probabilities simply add.