Sample Space and Events

The sample space (denoted by S) is the set of all possible outcomes of a random experiment. An event is any subset of the sample space. For example, when tossing a fair coin twice, the sample space is:

{HH, HT, TH, TT}

An event could be getting at least one head: {HH, HT, TH}.

Probability Axioms

Probability is a measure P that maps events to numbers in [0,1], satisfying:

1. P(S) = 1 (The probability of the entire sample space is 1)
2. P(E) ≥ 0 for any event E
3. If E1, E2, … are mutually exclusive events, then P(∪Ei) = Σ P(Ei)

Addition and Multiplication Rules

The probability of union of events:

For two events E and F:

P(E ∪ F) = P(E) + P(F) – P(E ∩ F)

Multiplication Rule: For two events E and F:

P(E ∩ F) = P(E) × P(F|E)

This leads naturally to conditional probability and independence.

Conditional Probability

Conditional probability answers: “Given that event E has occurred, what is the probability that event F will occur?” It is defined as:

P(F|E) = P(E ∩ F) / P(E), provided P(E) > 0

Example: If a card is drawn from a standard deck and is known to be a heart, the probability it is an ace is:

P(ace | heart) = P(ace ∩ heart) / P(heart) = (1/52) / (13/52) = 1/13

Independence of Events

Two events E and F are independent if:

P(E ∩ F) = P(E) × P(F)

This implies P(F|E) = P(F), meaning occurrence of E does not affect the probability of F.

Misunderstanding independence is a common pitfall; independence differs from mutual exclusivity.

Real-World Example: Medical Diagnosis

Consider a disease with prevalence 1%, a test with 99% sensitivity (true positive rate) and 95% specificity (true negative rate). What is the probability a patient has the disease given a positive test?

• P(Disease) = 0.01
• P(Positive | Disease) = 0.99
• P(Positive | No Disease) = 0.05

Bayes’ theorem gives:

P(Disease | Positive) = [0.99 × 0.01] / [0.99 × 0.01 + 0.05 × 0.99] ≈ 0.167

This illustrates how even a seemingly accurate test can yield a surprisingly low probability of disease presence due to low prevalence.

Applications

• Spam detection in emails using Bayesian classifiers
• Fault diagnosis in engineering systems
• Updating probabilities in machine learning models

Discrete and Continuous Random Variables

A random variable (RV) assigns numerical values to outcomes of a random experiment.

• Discrete RV: Takes countable values (e.g., number of heads in coin tosses)
• Continuous RV: Takes values in intervals (e.g., time, height)

Probability distributions describe the likelihood of outcomes.

Common Distributions

• Binomial Distribution: Number of successes in n independent Bernoulli trials.
• Poisson Distribution: Counts of rare events occurring in a fixed interval.
• Normal Distribution: Bell-shaped curve describing many natural phenomena.

Understanding distribution properties aids in modeling and inference.

Permutations and Combinations

Counting techniques are essential when calculating probabilities involving selection and arrangement.

• Permutation: Ordering matters. Number of permutations of n distinct objects: n!
• Combination: Ordering does not matter. Number of combinations of k from n: C(n,k) = n! / (k!(n-k)!)

Example: Probability of drawing 2 aces from a deck of 52 cards:

P = C(4,2) / C(52,2) = 6/1326 ≈ 0.0045

Advanced Combinatorial Probability

Problems may involve multinomial coefficients, inclusion-exclusion principle, and generating functions for complex event counting.

Expected Value (Mean)

The average or mean value of a random variable, weighted by probabilities.

E[X] = Σ x_i P(X = x_i) for discrete, or E[X] = ∫ x f(x) dx for continuous.

Example: Expected value of rolling a fair die is:

E[X] = (1+2+3+4+5+6)/6 = 3.5

Variance and Standard Deviation

Variance measures the spread of the distribution:

Var(X) = E[(X – E[X])²] = E[X²] – (E[X])²

Standard deviation is the square root of variance.

Law of Large Numbers (LLN)

States that as the number of independent trials increases, the sample average converges to the expected value.

This explains why frequencies stabilize with larger samples.

Central Limit Theorem (CLT)

When independent random variables with finite mean and variance are summed, their normalized sum tends toward a normal distribution, regardless of original distribution shape.

This is a cornerstone of inferential statistics.

Markov Chains and Stochastic Processes

Models where future states depend only on the current state, not the past — useful in queueing theory, finance, and genetics.

Moment Generating Functions and Characteristic Functions

Tools for studying distributions and proving limit theorems.

Bayesian Inference and Decision Theory

Frameworks for updating beliefs and making optimal decisions under uncertainty.

Random Walks and Martingales

Concepts with applications in finance, gambling theory, and algorithm analysis.