Let’s firstly think what is the probability of two events happening. Lets say
- Coin toss is heads, \(P(Heads)\)
- It will rain, \(P(Rain)\)
Now these two events are independent of each other, so the probability of both occurring is:
$$P(Heads \cap Rain) = P(Heads)P(Rain)$$
But!! The probability of two events happening is not always this simple! Lets take another two events:
- Next person you meet is male, \(P(Male)\)
- Next person you meet is wearing a dress, \(P(Dress)\)
The probability of this happening is not independent, since we know (or at least have a prior belief) that there are not too many males who wear dresses. That is the essense of Baysian probability; that you have a prior belief (or evidence) on how an underlying should behave.
Baye’s theorem is typically expressed like this
$$ P(Male \cap Dress) = P(Male)\times P(Dress | Male) $$
or by symmetry
$$ P(Male \cap Dress) = P(Dress)\times P(Male | Dress) $$
Side Note
$$P(A | B) \neq P(B | A)$$
Lets say (using a now defunct wikipedia example, thanks to Think Bayes)
- We have two bowls of cookies
- Bowl 1 :
- 30 Vanilla
- 10 Chocolate
- Bowl 2 :
- 20 Vanilla
- 20 Chocolate
- Bowl 1 :
If we choose one of the bowls at random, and then one cookie at random, then we can then see
$$ P(Vanilla | Bowl 1) = 3/4 $$
But if we move it around \( P(Bowl 1 | Vanilla) \), not only is it not obvious how to compute this, but it would not be \(3/4\).
$$ P(Bowl 1 | Vanilla) = \frac{P(Bowl 1) P(Vanilla | Bowl 1)}{P(Vanilla)} = 0.5 \times 0.75/0.625 = 0.6 $$
next time: bayesian inference