Quantifying Uncertainty

Study Summary: Quantifying Uncertainty (STA112: Probability I)

This module introduces the fundamental concepts and principles of probability, building a rigorous mathematical system to quantify uncertainty. It progresses from basic definitions to key theorems, culminating in Bayes' Theorem.

Module Outline:

• Introduction: The Language of Probability
• Sample Spaces and Events
• The Axioms of Probability
• Conditional Probability
• Independence
• Bayes' Theorem
• Summary and Key Takeaways

From Counting to Probability

Building on Module 1's counting techniques, Module 2 focuses on assigning a numerical measure—probability—to events. The goal is to establish a logical, consistent, and rigorous mathematical system for quantifying uncertainty.

Learning Objectives:

• Define sample spaces, events, and the axioms of probability.
• Apply addition rules, conditional probability, and independence.
• Master Bayes' Theorem for "reversing" probabilities.

Sample Spaces and Events

Definition (Experiment)

Any process or action that produces an observation or outcome.

Definition (Sample Space S)

The set of all possible, mutually exclusive, and exhaustive outcomes of an experiment.

• Discrete Example: Tossing a coin. S = {H, T}
• Continuous Example: Measuring the lifetime of a car battery. S = [0, ∞)

Definition (Event)

Any subset E of the sample space S. An event E occurs if the actual outcome of the experiment lies in E.

• Example: Rolling a Die
  • S = {1, 2, 3, 4, 5, 6}
  • E = {Even numbers} = {2, 4, 6}
  • F = {Numbers less than 3} = {1, 2}

Events can be combined using set operations to model complex ideas.

Set Operations for Events

• Definition (Union: E U F): The event that E or F (or both) occurs.
• Definition (Intersection: E ∩ F or EF): The event that both E and F occur.
• Definition (Complement: E&supc;): The event that E does not occur. Everything in S that is not in E.
• Definition (Mutually Exclusive (Disjoint)): E and F cannot occur together. Their intersection is empty: EF = Ø.

The Axioms of Probability

These are the fundamental rules for assigning a probability P(E) to every event E in a sample space S.

The Three Axioms of Probability

1. Non-negativity: 0 ≤ P(E) ≤ 1. (Probabilities must be between 0 and 1, inclusive).
2. Probability of the Sample Space: P(S) = 1. (The probability of something happening is 1).
3. Additivity for Disjoint Events: If E1, E2, ... are mutually exclusive, then:

   P(∪∞n=1 En) = Σ∞n=1 P(En)

   Every other rule in probability is a consequence of these three simple axioms.

Immediate Consequences of the Axioms

• Theorem (Complement Rule): For any event E: P(E&supc;) = 1 – P(E)
  • Derivation: E and E&supc; are mutually exclusive, and S = E U E&supc;.
    • Using Axiom 3: P(S) = P(E U E&supc;) = P(E) + P(E&supc;).
    • Using Axiom 2: 1 = P(E) + P(E&supc;).
    • Therefore, P(E&supc;) = 1 – P(E).
  • Example: If P(rain tomorrow) = 0.3, then P(NO rain) = 1 – 0.3 = 0.7.

The General Addition Rule

• Theorem (Addition Rule for Any Two Events): For any events E and F (not necessarily disjoint):

  P(E U F) = P(E) + P(F) – P(EF)

  This is because the intersection area P(EF) is counted twice when P(E) and P(F) are added, so it must be subtracted once.

• Solved Example: The Addition Rule

  Problem: A card is drawn from a standard 52-card deck. What is the probability it is a Heart or a Face Card?

  Solution:

  • Let E = Heart. P(E) = 13/52.
  • Let F = Face Card (J, Q, K). P(F) = 12/52.
  • The intersection EF = Heart AND Face Card (J, Q, K of Hearts). P(EF) = 3/52.
  • Applying the Rule: P(Heart U Face) = P(E) + P(F) – P(EF) = 13/52 + 12/52 – 3/52 = 22/52 = 11/26.

Conditional Probability

Motivating Question

If you roll two dice and I tell you the first die is a 6, what's the probability the sum is 10?

• Without information: P(sum = 10) = 3/36 = 1/12 (outcomes: (4,6), (5,5), (6,4)).
• With the information: The new sample space is only outcomes where the first die is 6: {(6,1),..., (6,6)}.
• In this reduced world, only (6,4) gives a sum of 10. So, the probability is 1/6.

Definition (Conditional Probability)

The probability of E given that F has occurred is:

P(E|F) = P(EF) / P(F), for P(F) > 0.

The Multiplication Rule

Rearranging the definition of conditional probability provides a way to find joint probabilities:

P(EF) = P(F) · P(E|F)

This extends naturally to more events:

P(E1E2E3) = P(E1)P(E2|E1)P(E3|E1E2)

• Solved Example: Drawing Balls Without Replacement

  Problem: An urn has 3 red (R) and 2 blue (B) balls. We draw 2 balls. What is P(both red) = P(R1R2)?

  Solution:

  • P(R1) = 3/5.
  • Given the first was red, P(R2|R1) = 2/4 (since one red ball is removed).
  • P(R1R2) = (3/5) × (2/4) = 6/20 = 0.3.

Independence

Definition (Statistical Independence)

Events E and F are independent if the occurrence of one does not affect the probability of the other. Equivalently:

P(EF) = P(E)P(F)

• If P(F) > 0, this is also equivalent to P(E|F) = P(E).

Critical Warning: Disjoint (EF = Ø) and Independent are totally different!

• If E and F are disjoint, they are highly dependent. If one happens, the probability of the other becomes ZERO.
• Independence means knowing one happened tells you nothing about the other.

Solved Example: Independence in Practice

Problem: Roll two fair dice. Let E = {sum is 7}, F = {first die is 4}. Are E and F independent?

Solution:

• P(EF) = P({(4,3)}) = 1/36 (The first die is 4 AND the sum is 7, so the second die must be 3).
• P(E) = 6/36 = 1/6 (Outcomes for sum 7: (1,6), (2,5), (3,4), (4,3), (5,2), (6,1)).
• P(F) = 6/36 = 1/6 (Outcomes for first die is 4: (4,1), (4,2), (4,3), (4,4), (4,5), (4,6)).
• Since P(E) × P(F) = (1/6) × (1/6) = 1/36 = P(EF), the events ARE independent.
• Intuition: Knowing the first die is a 4 doesn't make a sum of 7 any more or less likely; you still need exactly a 3 on the second die, which has a 1/6 chance.

Bayes' Theorem

The Law of Total Probability

If F1, F2, ..., Fn are mutually exclusive and exhaustive (their union is S), then for any event E:

P(E) = Σni=1 P(E|Fi)P(Fi)

This means the total probability of E is the weighted average of its conditional probabilities, weighted by the probabilities of the "causes" Fi.

Bayes' Theorem: Reversing the Condition

Using the Law of Total Probability, we can "reverse" a conditional probability:

P(Fj|E) = P(EFj) / P(E) = [P(E|Fj)P(Fj)] / [Σni=1 P(E|Fi)P(Fi)]

Terminology:

• P(Fj): Prior probability (belief before evidence).
• P(E|Fj): Likelihood of seeing evidence E given Fj.
• P(Fj|E): Posterior probability (updated belief after seeing evidence).
• Prior   —E— >   Posterior

Solved Example: The Disease Testing Problem

Problem: Suppose 1% of a population has a disease. A test is 95% accurate for those who have the disease, and gives a false positive for 2% of healthy people. If a person tests positive, what is the probability they actually have the disease?

Step 1: Define Events & Knowns

• D = has disease. P(D) = 0.01.
• D&supc; = healthy. P(D&supc;) = 0.99.
• T+ = tests positive.
• P(T+|D) = 0.95 (True positive rate).
• P(T+|D&supc;) = 0.02 (False positive rate).

Step 2: Apply Bayes' Theorem

We want to find P(D|T+) (probability of having the disease given a positive test).

P(D|T+) = [P(T+|D)P(D)] / [P(T+|D)P(D) + P(T+|D&supc;)P(D&supc;)]

P(D|T+) = (0.95 × 0.01) / [(0.95 × 0.01) + (0.02 × 0.99)]

P(D|T+) = 0.0095 / (0.0095 + 0.0198)

P(D|T+) = 0.0095 / 0.0293 ≈ 0.324

The Shocking Result: Despite the positive test, the probability of actually having the disease is only 32.4%! This highlights how Bayes' Theorem updates our prior beliefs with new evidence.

Module 2: Summary & Key Takeaways

The Foundation

Probability is a mathematical system built on three axioms. Every rule—complements, addition, conditioning—follows logically from them.

Key Concepts to Remember

1. Conditional Probability updates our beliefs with evidence.
2. Independence means evidence provides no update.
3. Bayes' Theorem is the engine for learning from data, taking us from a Prior P(F) to a Posterior P(F|E).

Looking Ahead to Module 3

"Now that we understand events and their probabilities, we will introduce random variables—numerical summaries of outcomes—and study their distributions."