Discrete Probabilities and Probability Distributions

The simplest form of probability functions are discrete probabilities. In this case, we have some variable $X$ which can take different values or outcomes, $x_1,x_2,\dots$.

Examples:

if $X$ represents a statement such as ``it will rain tomorrow'', it can take two values TRUE and FALSE; if $X$ represents the result of a coin flip, there would be the two possible outcomes, $x_1$ meaning heads and $x_2$ meaning tails; if $X$ is the number drawn in the next lottery draw, there would be 13,983,816 possibilities.

We denote the probability that $X$ takes a particular value $x_i$ as $P(x_i)$. The probability must be between $0$ and $1$. If $P(x_i)=1$, $X$ takes the value $x_i$ with certainty; if $P(x_i)=0$, $X$ definitely does not take the value $x_i$; the closer is $P(x_i)$ to $1$, the greater the likelihood that $X$ takes this value. Assuming that $X$ can take only one of the values at a time (the values are mutually exclusive), the probability that $X$ takes $x_1$ or $x_2$ is $P(x_1)+P(x_2)$. Since $X$ must take some value, the sum of the the probabilities of all possibly values is $1$. Thus, the probability is normalised,

$$\sum_{\mbox{all $x_i$}} P(x_i) = 1$$

which means that $X$ always takes one of its allowed values.

We are often interested, however, in cases in which a random variable takes a continuous range of values. In this case, the above must be modified. We write a probability distribution or density $P(x)$ as a function of the continuous variable $x$.

Example:

A farmer might be interested in the probability that a given amount of rainfall will occur in a particular growing season. The rainfall, $X$, could be modelled by a continuous positive real number $x$.

We cannot interpret $P(x)$ as the probability that $X$ takes the value $x$, because that probability is almost always going to be $0$. You cannot have an infinite number of possibilities with nonzero probability, because it could not be normalised. For continuous distributions, $P(x)$ is defined so that the probability that $X$ is between $x$ and $x+dx$ is $P(x) dx$, where $dx$ is arbitrarily small. The normalisation condition converts from a sum to an integral, and is,

$$\int_{\mbox{all $x$}} dx P(x) = 1.$$