STAT 230 - Probability
MWF 2:30PM - 3:20PM
DC 1351
Cecilia Cotton

15% quizzes (3/5) | 25% midterm | 15% midterm | 45% final

1 | Introduction to Probability

1.1 | Definitions of Probability

This course is a probability course, not a statistics course (STAT 231)

Probability = A discipline of mathematics concerned with describing and modeling uncertain events
Statistics = The study of the collection, analysis, and interpretation of data as it relates to underlying probability models

Definitions of Probability (each has flaws)

Classical Probability:
1. Flaw: not all events are equally likely

P = \frac{number of ways the event can occur}{the total number of possible outcomes}

Relative Frequency: Repeat the event a lot of times
1. Flaw: we can never repeat an experiment indefinitely
Subjective Probability: Vibes
1. Flaw: how do you resolve disagreements?

1.2 | Mathematical Probability Models

Probability Model = Sample Space + Probability Distribution + Event

Definition: Sample Space A sample space $S$ is the set of all possible outcomes of a random experiment where only one outcome can occur in a single trial

Definition: Probability Distribution A probability distribution on $S$ assigns probabilities (between 0 and 1) to outcomes

Definition: Event An event $A$ is a subset of the sample space, and the probability of an event is the sum of the probabilities for each outcomes in $A$ .

Sample Space

Is Discrete if it’s finite and non-Discrete if infinite
Ex: Rolling a die. Let $a_{i}$ be the number of dots facing upward

S = {a_{i} \dots a_{6}} = {1, 2, 3, 4, 5, 6}

Ex: Runtime of a computer program.

S = {a_{i} : a_{i} < 0, a_{i} \in R}

Ex: Flipping a coin (there can be multiple sample spaces)
- $S_{1} = {HH, H T, T H, TT}$
- $a_{i} = # tails, S_{2} = {a_{0}, a_{1}, a_{2}} = {0, 1, 2}$
- Are the outcomes the same? $S_{3} = {a_{yes}, a_{n o}}$
- $S_{4} = {a_{1}, a_{yes}, a_{n o}}$ - not a sample space (overlapping outcomes)
- $S_{5} = {a_{0}}$ - not a sample space (not all possibilities)

Event

Ex: Rolling a die:
- $S = {1, 2, 3, 4, 5, 6}$
- Simple event: Rolling a 1, $A_{1} = {a_{i}} = 1$
- Compound event: Rolling an even, $A = {a_{2}, a_{4}, a_{6}} = {2, 4, 6}$
Ex: Runtime of a program
- Event that the program takes >30 seconds

A = {t : t > 30}

Definition: Probability Let $S$ denote the set of all events on a sample space $S$ . > A probability defined on $S$ is a real valued function that satisfies the following axioms

P : S \to [0, 1]

Scale: Everything is normalized to a size of 1

Something Happens: $P (S) = 1$

Infinite Additivity: $P (A_{1}) + P (A_{2}) = P (A_{1} \cup A_{2}) = P (A)$

Facts about the probability function

$P (\emptyset) = 0$
1. Proof: Define an infinite collection of events (all $\emptyset$ , all disjoint, union is $\emptyset$ ) and use the Additivity axion
Finite Additivity:

P (\cup_{i = 1}^{n} A_{i}) = n = 1 \sum n P (A_{i})

1. Proof: Extend to an infinite collection of disjoint events by adding infinite $\emptyset$ and the union stays the same

3. $P (\overset{ˉ}{A}) = 1 - P (A)$ 1. Proof: Use the axiom that $P (S) = 1$

1.3 | Counting in Uniform Probability Models

Counting Rules

Definition: Equally Likely Model Let $∣ A ∣$ denote the number of outcomes in an event $∣ A ∣$ . Then, for an equally likely sample space:

P (A) = i : a_{i} \in A \sum P (A_{i}) = \frac{∣ A ∣}{∣ S ∣}

Note: This can only happen in a discrete sample set

Counting Rules

Addition Rule

∣ A \cup B ∣ = ∣ A ∣ + ∣ B ∣

Multiplication Rule

∣ A ∣ = n_{1} n_{2} \dots n_{k} = i = 1 \prod k n_{i}

Example: Suppose that 3 of the numbers from 1-9 are chosen without replacement to form a 3 digit number. What is the probability that:

The number is bigger than 500?

P (A) = \frac{5}{9}

The number is even? (you could also count EEE, EOE, OEE, OOE)

P (B) = \frac{4}{9}

The number is larger than 700 and even? Consider $P (> 700 \cup even)$ 7xx : $1 \times 4 \times 7$ 8xx: $1 \times 3 \times 7$ 9xx: $1 \times 4 \times 7$ $S_{a ll} = {d_{1} d_{2} d_{3}, d_{1} \neq = d_{2}, d_{2} \neq = d_{3}, d_{1} \neq = d_{3}}$ $∣ S_{a ll} ∣ = 9 \times 8 \times 7$

P (C) = \frac{11 \times 7}{7 \times 8 \times 9} = \frac{11}{72}

Permutations and Combinations

Definition: Permutation Given $n$ distinct objects, a permutation of size $k$ is an ordered subset of $k$ of the individuals. The number of permutations of size $k$ taken from $n$ objects is

n^{(k)} = \frac{n !}{( n - k )!}

Definition: Combination Given $n$ distinct objects, a combination of size $k$ is an unordered subset of $k$ of the individuals. The number of permutations of size $k$ taken from $n$ objects is

(k n) = \frac{n ^{(k)}}{k !} = \frac{n !}{( n - k )! k !}

Suppose you have 20 distinct books, 7 of which are written by Mark Twain.

How many ways can you arrange 12 books on a shelf if the order they are on the shelf matters?

∣ A ∣ = 2 0^{(12)} = \frac{20 !}{12 !}

How many ways can you arrange 12 books on a shelf if exactly 3 of them must be Mark Twain books?

∣ B ∣ = (3 7) (9 13) 12!

A monkey picks books at random from the 20 books and puts them on the shelf until it contains 12 books. What is the probability that at least 3 of the books on the shelf are written by Mark Twain?

Denote $C_{i}$ to be {number of Twain books}. Choose from 7 Twain and 13 non-Twain books:

∣ C_{0} ∣ = (0 7) (12 13) = 13

∣ C_{1} ∣ = (1 7) (11 13) = \frac{7 \times 13 \times 12}{2} = 546

∣ C_{2} ∣ = (2 7) (10 13) = \frac{7 \times 6 \times 13 \times 12 \times 11}{2 \times 6} = 6006

∣ S ∣ = (12 20) = 127970

P (C) = 1 - \frac{13 + 546 + 6006}{127970} = \frac{119405}{125970}

Consider drawing 3 numbers with replacements from 10 digits. What is the probability that there’s a repeated number?

P (A) = 1 - P (\overset{ˉ}{(} A)) = 1 - \frac{10 \times 9 \times 8}{1 0 ^{3}}

∣ A ∣ + 10 + (2 10) 2 \times 3 = 10 + 10 \times 9 \times 3 ⟹ \frac{10 + 10 ( 9 ) ( 3 )}{1 0 ^{3}}

Melissa participates in a lottery in which she selects 7 numbers between 1 and 50, and then a computer randomly picks 7 numbers between 1 and 50. She wins if her selected numbers match 5 or more of the randomly selected numbers, in any order. What is the probability that Melissa wins?

S = {selections of computer numbers}

∣ S ∣ = (7 50)

A = {Melissa wins} - consider 5, 6, 7 matches

P (A) = \frac{( 5 7 ) ( 2 43 ) + ( 6 7 ) ( 1 43 ) + ( 7 7 ) ( 0 43 )}{( 7 50 )}

Make sure you decide whether or not order matters for everything!

Suppose a room contains n people. What is the probability at least two people in the room share a birthday? Assumption: Suppose that each of the n people is equally likely to have any of the 365 days of the year as their birthday, so that all possible combinations of birthdays are equally likely.

Put a uniform model over $S_{n}$ :

S_{n} = {(d_{1}, \dots, d_{n}), d_{i} + {1, \dots, 365}}

$A_{n}$ = {at least 2 match}

P (A_{n}) = 1 - P (\overset{ˉ}{A_{n}}) = 1 - \frac{36 5 ^{(n)}}{36 5 ^{n}}

Multinomial Coefficient Theorem

There are 4 passengers on an elevator that services 5 floors. Assume that each of the 4 passengers are equally likely to get off the elevator on any of the 5 floors. What is the probability that:

The passengers all get off on different floors? (# ways to select 5 floors)

P (A) = \frac{5 ( 4 ) ( 3 ) ( 2 )}{5 ^{4}}

2 passengers get off on floor two, and two get off on floor three? (# ways to select 2 passengers)

P (B) = \frac{( 2 4 )}{5 ^{4}}

2 passengers get off on one floor, and two passengers get off on another, different floor? (# ways to select 2 passengers) x (# ways to select 2 floors)

P (C) = \frac{( 2 4 ) \times ( 2 5 )}{5 ^{4}}

Exactly 1 passenger gets off on floor 1 (# ways to select one passenger) x (# ways to select remaining floors)

P (D) = \frac{( 1 4 ) 4 ^{3}}{5 ^{4}}

Exactly one passenger gets off on floor 1 and exactly one gets off on floor 2 (# ways to select 2 passengers) x (# ways to assign them) x (# ways to select remaining floors)

P (E) = \frac{( 2 4 ) \times 2 \times 3 ^{2}}{5 ^{4}}

Definition: Multinomial Coefficient Theorem Consider $n$ objects with consist of $k$ indistinguishable types. Suppose that there are $n_{i}$ objects of type $i$ . Then, the number of distinguishable arrangements of the $n$ objects is

\frac{n !}{n _{1} ! n _{2} ! \dots n _{k} !}

Consider rearranging the letters at random in the name “ZENYATTA” to form a single ‘word’.

How many ways can this be done? (# total permutations) / (# replacements)

∣ A ∣ = \frac{8 !}{2 ! 2 !}

What is the probability that all of the letters appear in alphabetic order? (# alphabetical permutations) / (# total permutations)

P (B) = \frac{1}{8 !}

What is the probability that the word begins and ends with “T”? (# first letter is T) x (# last letter is T)

P (C) = \frac{2}{8} \times \frac{1}{7}

3 members of the C-S department, 2 members of the math department, and 3 members of the stats department sit down at random in a row of 8 seats.

What is the probability that each department’s members are sitting in consecutive seats? (# permutations of faculties) x (# permutations in each faculty)

P (A) = \frac{3 ! \times 3 ! 2 ! 3 !}{8 !}

What is the probability that members of the same department are sitting on both ends of the row? P(CS on both ends) + P(Math both ends) + P(Stats both ends)

P (B) = \frac{3 ( 2 ) + 2 ( 1 ) + 3 ( 2 )}{8 \times 7}

Or, excluding order:

P (B) = \frac{3 ( 2 ) 6 ! + 2 ( 1 ) 6 ! + 3 ( 2 ) 6 !}{8 !}

5 floor and 5 people on an elevator, what is the probability that 2 people get off on one floor and 3 get off on the other?

$∣ A ∣$ = (# of ways to choose 2 floors) (# of ways to choose 3 people) (2 ways to permute people)

P (A) = \frac{( 2 5 ) ( 3 5 ) 2}{5 ^{5}}

2 | Probability Rules and Conditional Probability

2.1 | Use of Sets

Venn Diagrams

De Morgan’s Laws (they generalize to summations)

\overline{(A \cup B)} = \overline{A} \cap \overline{B}

\overline{(A \cap B)} = \overline{A} \cup \overline{B}

2.2 | Addition Rules for Unions of Events

Inclusion-Exclusion Principle

Inclusion-Exclusion Rules: For arbitrary events $A$ and $B$ :

P (A \cup B) = P (A) + P (B) - P (A \cap B)

Proof:

A \cup B = A \cup (B \cap \overline{A})

P (A \cup B) = P (A) + P (B \cap \overline{A})

B = B \cap S = B \cap (A \cup \overline{A}) = (H \cap A) \cup (B \cup \overline{A})

P (B) = P (B \cap A) + P (B \cap \overline{A}) ⟹ P (B \cap \overline{A}) + P (B)

Inclusion-Exclusion Principle (Generalized)

PP (\cup_{i = 1}^{n} A_{)} = i \sum P (A_{i}) = i < j \sum P (A_{i} A_{j}) + i < j < k \sum P (A_{i} A_{j} A_{k}) - i < j < k < l \sum P (A_{I} A_{j} A_{k} A_{l}) + \dots

Ex: At least one die shows 6

S = {(d_{1}, d_{2}), d_{i} \in {1, \dots, 6}}

|S| = 36 $A_{1}$ = P(first die shows 6) $A_{2}$ = P(second die shows 6)

P (A) = P (A_{1}) + P (A_{2}) - P (A_{1} A_{2}) = \frac{1}{6} + \frac{1}{6} - \frac{1}{36} = \frac{11}{36}

Ex: 80% Tim Hortons, 63% Canadian Tire, 51% both

Canadian Tire and Tim Hortons T = {TH}, C = {CT}

P (T) = 0.8, P (C) = 0.63, P (T \cap C) = 0.51

P (T \cup C) = P (T) + P (C) - P (T \cap C) = 0.8 + 0.63 - 0.51 = 0.92

Canadian Tire but not Tim Hortons We want $P (C \cap \overline{T})$

P (C) = P (C \cap (T \cup \overline{T})) = P (C \cap T) + P (C \cap \overline{T})

P (C \cap \overline{T}) = P (C) - P (C \cap T) = 0.63 - 0.51 = 0.12

Ex: Rearrange the letters in “FOOD”

$P (A)$ , A = the event that the two O’s appear together S = {Distinguishable Arrangements assuming Os are indistinguishable}

∣ S ∣ = \frac{4 !}{2 !} = 12

∣ A ∣ = 3!

P (A) = \frac{3 !}{12} = \frac{1}{2}

$P (B)$ , B = the event that the word starts with F

∣ B ∣ = \frac{3 !}{2 !} = 3

P (B) = \frac{1}{4}

$P (\overline{A} \cap \overline{B})$

P (\overline{A} \cap \overline{B}) = 1 - P \overline{(\overline{A} \cup \overline{B})} = 1 - P (A \cup B) = 1 - [P (A) + P (B) - P (A \cap B)] = 1 - [\frac{1}{2} + \frac{1}{4} - \frac{2}{12}] = \frac{5}{12}

2.3 | Dependent and Independent Events

Independence

Definition: Independence Two events $A$ and $B$ are said to be independent of

P (A \cap B) = P (A) P (B)

A sequences of events $A_{1}, A_{2}, \dots$ are said to be independent if

P (A_{i 1} \cap A_{i 2} \cap \dots \cap A_{ik}) = P (A_{i 1}) P (A_{i 2}) \dots P (A_{ik})

for all possible subsets of size k, $i_{1}, i_{2}, \dots i_{K}$ . Events that are not independent are called dependent.

Ex: Consider rolling two fair die, and let A = {sum is 10}, B = {first die is 6}, C = {sum is 7}

A and B are dependent

P (A) = \frac{3}{36}, P (B) = \frac{1}{6}, P (A \cap B) = \frac{1}{36} \neq = P (A) P (B)

B and C are independent

P (C) = \frac{1}{6}, P (B \cap C) = \frac{1}{36} = P (B) P (C)

A and C are dependent

P (A \cap C) = \emptyset \neq = P (A) P (B)

Proposition: If $A$ and $B$ are independent and mutually exclusive (disjoint), then either $P (A) = 0$ or $P (B) = 0$

Proof: Suppose $A, B$ are independent and disjoint, then $P (A) P (B) = P (A \cap B) = P (\emptyset) = 0$

Remark: Pairwise independence of events doesn’t mean that the events are independent

Experiment: Roll two fair die A = {first die = 1}, B = {second die = 1}, C = {sum = 7} Each pair is independent, but

P (A \cap B \cap C) = 0 \neq = P (A) P (B) P (C) = (\frac{1}{6})^{3}

Proposition: If $A, B$ are independent, then $\overline{A}, \overline{B}$ ; $A, \overline{B}$ ; $\overline{A}, B$ are all pairwise independent.

Proof: Suppose $A, B$ are independent.

P (\overline{A} \cap \overline{B}) = 1 - P (A \cup B) = 1 - [P (A) + P (B) - P (A \cap B)] = 1 - [P (A) + P (B) - P (A) P (B))] = (1 - P (A)) (1 - P (B)) = P (\overline{(} A) P (\overline{B})))

2.4 | Conditional Probability and Product Rules

Conditional Probability

Definition: Conditional Probability

P (A ∣ B) \frac{P ( A \cap B )}{P ( B )}

A = {sum is 10}, B = {first die is 6}, C = {sum is 7}

$P (A ∣ B)$

= \frac{P ( A \cap B )}{P ( B )} = \frac{\frac{1}{36}}{\frac{1}{6}} = \frac{1}{6}

$P (B ∣ A)$

= \frac{P ( A \cap B )}{P ( B )} = \frac{\frac{1}{36}}{\frac{3}{36}} = \frac{1}{3}

$P (A ∣ C)$

= \frac{P ( A \cap C )}{P ( C )} = \frac{0}{P ( C )} = 0

$P (C ∣ B)$

= \frac{P ( C \cap B )}{P ( B )} = \frac{\frac{1}{36}}{\frac{1}{6}} = \frac{1}{6}

Definition: Independence and Conditionals Two events $A, B$ are independent if

P (A ∣ B) = P (A)

Properties of conditional probability:

$0 \leq P (A ∣ B) \leq 1$
$P (\overline{A} ∣ B) = 1 - P (A ∣ B)$
If $A_{1}, A_{2}$ are disjoint, $P (A_{1} \cap A_{2} ∣ B) = P (A_{1} ∣ B) + P (A_{2} ∣ B)$
$P (S ∣ B) = 1 = P (B ∣ B)$

Ex: Consider rearranging the letters in the word RACECAR

What is the probability that the random word ends in “R” given that it starts with “ACE”? A = {word starts ACE} R = {word ends R} S = {distinguishable arrangements}

∣ S ∣ = \frac{7 !}{2 ! 2 ! 2 !}

P (R ∣ A) = \frac{P ( R ∣ A )}{P ( A )} = \frac{3 !}{\frac{4 !}{2 !}} / \frac{\frac{7 !}{2 ! 2 ! 2 !}}{\frac{7 !}{2 ! 2 ! 2 !}} = \frac{1}{2}

Is the event that the word starts with “ACE” independent of the event that it ends with an “R”?

P (R) = \frac{\frac{6 !}{2 ! 2 !}}{\frac{7 !}{2 ! 2 ! 2 !}} = \frac{2}{7} < P (R ∣ A)

No, they’re dependent

Product Rules

Product Rule:

P (A \cap B) = P (A ∣ B) P (B) = P (B ∣ A) P (A)

Ex: Suppose a bag contains 12 red balls and 7 blue balls. Suppose that a ball is drawn at random, then, without replacement, a second ball is drawn at random.

What is the probability that both balls are red? R1 = {first marble is red} R2 = {second marble is red}

P (R_{1} \cap R_{2}) = R (R_{2} ∣ R_{1}) P (R_{1}) = \frac{11}{18} \times \frac{12}{19}

Alternate solution: S = {all selections of first 2 marbles}

∣ S ∣ = (2 19)

∣ R_{1} \cap R_{2} ∣ = (2 12)

What is the probability that the second ball is red?

P (R_{@} = P (R_{2} \cap S)) = P (R_{2} \cap (R_{1} \cup \overline{(} R_{1}))) = P (R_{2} \cap R_{1}) + P (R_{2} \cap \overline{R_{1}})) = P (R_{2} ∣ R_{1}) P (R_{1}) + P (R_{2} ∣ \overline{R_{1}}) P (\overline{R_{1}}) = \frac{11}{18} \frac{12}{19} + \frac{12}{18} \frac{7}{19} = \frac{12}{19} \frac{11 + 7}{18} = \frac{12}{19}

Law of Total Probability

Definition: Partition A sequence of sets $B_{1}, B_{2}, \dots B_{k}$ are said to partition the sample space $S$ if they partition the sample space $S$ I’m not writing the formal definition

Theorem: Law of total probability: Suppose that $B_{1}, \dots B_{k}$ partition $S$ . Then for any event $A$ ,

P (A) = P (A ∣ B_{1}) P (B_{1}) + P (A ∣ B_{2}) P (B_{2}) + \dots + P (A ∣ B_{K}) P (B_{k})

Usually we use $B, \overline{B}$ because that partitions any $S$ :

P (A) = P (A ∣ B) P (B) + P (A ∣ \overline{B}) P (\overline{B})

Example: A rare disease occurs in 0.3% of a population. A test has been developed for the disease. If the test is performed on an individual who has the disease, the probability of a positive test result (i.e. the test says that they have the disease) is 0.96. If the test is performed on an individual who does not have the disease, the probability of a positive test result is 0.05. Suppose the test is performed on a random individual in the population.

What is the probability that the test result will be positive? D = {Disease} T+ = {Test is positive} $P (D) = 0.003$ $P (T^{+} ∣ D) = 0.96$ $P (T + ∣ \overline{D}) = 0.05$

P (T^{+}) = P (T^{+} \cap D) + P (T^{+} \overline{D}) = P T^{+} ∣ D) P (D) + P (T^{+} ∣ \overline{D}) P (\overline{D}) = (0.96) (0.003) + (0.05) (0.997) = 0.05

Suppose the test comes back positive. What is the probability that the individual actually has the disease?

P (D ∣ T^{+}) = \frac{P ( D \cap T ^{+} )}{P ( T ^{+} )} = \frac{P ( T ^{+} ∣ D ) P ( D )}{P ( T ^{+} )} = \frac{0.96 ( 0.003 )}{( 0.96 ) ( 0.003 ) + ( 0.05 ) ( 0.997 )} = 0.05

If the test comes back negative, what is the probability that the individual actually has the disease?

P (D ∣ \overline{T^{+}}) = \frac{P ( D \cap T ^{+} )}{P ( T ^{+} )} = \frac{P ( T ∣ D ) P ( D )}{1 - P ( T ^{+} )} = \frac{( 1 - P ( D ^{+} ∣ D )) P ( D )}{1 - P ( T ^{+} )} = \frac{( 0.04 ) ( 0.003 )}{1 - 0.05273} = 1.2 \times 1 0^{- 4}

Bayes Theorem

Bae’s Theorem

P (B ∣ A) = \frac{P ( A ∣ B ) P ( B )}{P ( A ∣ B ) P ( B ) + P ( A ∣ B ) P ( B )}

Gold Coin question

P (GG ∣ G) = \frac{P ( G ∣ GG ) P ( GG )}{P ( G ∣ GG ) P ( GG ) + P ( G ∣ GS )} = \frac{2}{3}

Two factories, Factory I and Factory II, produce phones for a phone brand “JimSang”. Factory I produces 60% of all JimSang phones, and Factory II produces the other 40 %. Of phones produced in Factory I, 10% are defective, and 20% of phones produced by Factory II are defective.

What is the probability that a randomly selected JimSang phone is defective?

D = {Defective phone} $F_{i}$ = {Phone made in Factory i}

P (F_{i}) = 0.6, P (F_{2}) = 0.4

P (D ∣ F_{1}) = 0.10, P (D ∣ F_{2}) = 0.20

P (D) = P (D \cap F_{1}) + P (D \cap F_{2}) = 0.6 (0.10) + 0.4 (0.20) = 0.14

A customer has purchased a phone, and it is not defective. What is the probability that it was produced in Factory II?

P (F_{2} ∣ \overline{D}) \neq = 1 - P (F_{2} ∣ D)

= \frac{P ( D ∣ F _{2} ) P ( F _{2} )}{P ( D )} = \frac{( 1 - P ( D ∣ F _{2} )) P ( F _{2} )}{1 - P ( D )} = (\frac{( 1 - 0.2 ) 0.4}{1 - 0.14}) = 0.37209

3 | Univariate Discrete Probability Distributions

3.1 | Discrete Random Variables

Geometric Series Formula

i = 0 \sum \infty r^{i} \frac{1}{1 - r}

Partial Geometric Series

i = 0 \sum k r^{i} = \frac{1 - r ^{k + 1}}{1 - r}

Binomial Theorem

(a + b)^{n} = i = 0 \sum n (i n) a^{i} b^{n - i}

Taylor Series for $e^{x}$

e^{x} = i = 0 \sum \infty \frac{x ^{i}}{i !}

Example

i = 1 \sum \infty (\frac{1}{2})^{i} = \frac{1}{1 - \frac{1}{2}} - 1 = 2 - 1 = 2

Definition: Random Variable A random variable is a function that maps the sample space $S$ into the set of real numbers $R$ . In other words:

X : S \to R

The values that a random variable takes is called the range, $X (S)$ We say that a random variable is discrete if its range is finite, and continuous otherwise.

Definition: Probability Mass Function The probability function of a discrete random variable $X$ is the function

f_{x} (x) = P (X = x)

Ex: Suppose that $X$ is the sum of the outcomes of a two dice roll. Calculate the probability function of X.

X ((d_{1}, d_{2})) = d_{1} + d_{2}

X (S) = {2, 3, \dots, 12}

f_{X} (2) = P (x = 2) = P ({1, 1}) = \frac{1}{36}

f_{X} (3) = P (X = 3) = P ({(1, 2), (2, 1)}) = \frac{2}{36}

f_{X} (4) = \frac{3}{36}, f_{X} (5) = \frac{4}{36}, \dots

Properties of the probability function

It’s between 0 and 1

0 \leq f_{X} (x) \leq 1

The sum of all probabilities is 1

x \in X (S) \sum f_{X} (x) = 1

3.2 | Functions of Random Variables

Cumulative Distribution Function (CDF)

F_{X} (X) = P (X \leq x)

When $X$ is discrete:

P (X \leq x) = y : y \leq x \sum f_{X} (y)

Properties of the CDF

$0 \leq F_{X} (x) \leq 1$
$F_{X} (x) \leq F_{X} (y)$ for $x < y$
$lim_{x \to - \infty} F_{X} (x) = 0$ and $lim_{x \to \infty} F_{X} (x) = 1$

Ex: Compute and graph $F_{X} (x)$ for a fair die roll

f_{x} (x) = \frac{1}{6}, x \in {1, \dots, 6}

F_{X} (x) = P (X \leq x)

CDF is CADLAG (continuous from the right, limited from the left)

Ex: $P (A) = 0.9, P (B) = 0.7, P (C) = 0.4$

Compute the probability function on $X$

X (S) = {0, 1, 2, 3}

f_{X} (0) = P (\overline{A} + \overline{B} + \overline{C}) = (0.1) (0.3) (0.6) = 0.018

f_{X} (1) = P (A \cap \overline{B} \cap \overline{C}) + P (\overline{A} \cap B \cap \overline{C}) + P (\overline{A} \cap \overline{B} \cap C)

= (0.9) (0.3) (0.6) + (0.1) (0.7) (0.6) + (0.1) (0.3) (0.4) = 0.216

f_{X} (2) = 0.514, f_{X} (3) = 0.252

Compute the CDF
Compute P(at least 1 correct)

P (X \geq 1) = 1 - F_{X} (0) = 1 - 0.018 = 0.982

3.3 | Expectation of a Random Variable

Expected Values

Definition: Expectation Suppose $X$ is a discrete random variable with probability function $f_{X} (x)$ . Then $E (X)$ is called the expected value of $X$ , and is defined by

E (X) = x \in X (S) \sum x f_{x} (x)

The expected value of $X$ is sometimes referred to as the mean / full moment of $X$

Ex: Compute $E (X)$ of a dice roll

E (X) = x \in X (S) \sum \times f_{X} (x) = x = 1 \sum 6 = \frac{21}{6} = 3.5

Ex: Lotto 649

R = Return on $1 b e t :$ R(S)={-1, 499999}$

f_{R} (4, 999, 999) = \frac{1}{( 6 49 )} = 1 - f_{R} (- 1)

E [R] = (4, 999, 999) (\frac{1}{( 6 49 )}) + (- 1) (1 - \frac{1}{( 6 49 )}) \approx - 0.642

Properties of Expectation

Property: Bounding If a random variable $X$ is bounded by $a \leq X (w) \leq b$ , then

a \leq E (X) \leq b

If $g (x) = x^{2}$ and $X$ is the result of a fair six sided die roll, then compute $E [g (x)]$

f_{X} (x) = \frac{1}{6}, x \in {1, \dots, 6}

E (x^{2}) = x = 1 \sum 6 x^{2} \frac{1}{6} = \frac{91}{6} \approx 15.1667 ⟹ E (x) = 3.5

Property: Linearity of Expectation If $g (x)$ is a linear function $g (x) = a x + b$ , then for a random variable $X$

E [a X + b] = a E [x] + b

MISTAKE: It is not true in general that

g (E [x]) = E [g (x)]

Variance

Deviation

E [(X - μ)] = E [x] - μ = 0

Absolute deviation

E ∣ X - μ ∣

Squared deviation:

E (x - μ)^{2}

Definition: Variance The variance of a random variable $X$ is denoted $Va r (X)$ , and is defined by

Va r (X) = E [(X - E [X]^{2})]

Shortcut to compute variance: $μ = E (x)$

Va r (x) = E [(x - μ)^{2}] = E [x^{2} - 2 xμ + μ^{2}] = E [x^{2}] + e (- 2 xμ) + E [μ^{2}]

= E [x^{2}] - 2 μ E (x) + μ^{2} = E (x^{2}) - μ^{2}

Definition: Moments The $k^{t h}$ moment of a random variable $X$ is defined by

E (X^{k})

Central Second Moment

Va r (X) = E [(X - E (X))^{2}]

Suppose $X$ is a random variable such that $E (X^{2}) < \infty$ . Then $E (X^{2}) \leq E (X))^{2}$

Definition: Standard Deviation

S D (X) = Va r (X)

Theorem: Lyapounov’s Inequality If $E (X) = μ$ ,

E [∣ X - μ ∣] \leq S D (X)

Ex: Compute $Va r (X), S D (X)$ for a die roll

Va r (X) = E (X^{2}) - (E (X)^{2}) = \frac{91}{6} - (3.5)^{2} = 2.91667

S D (X) = Va r (X) = 1.70

Theorem: Variance of a linear combination:

Va r (a X + b) = a^{2} Va r (X)

Variance Properties

For all random variables $X$ :

Va r (X) \geq 0

Larger values of $Va r (X)$ indicate that the probability function / distribution of $X$ is more “spread out” around the mean
$Va r (X) = 0$ if and only if $P (X = E (X)) = 1$ .

0 = Va r (X) = E [(X - E (X))^{2}] = X \in X (S) \sum (X - E (X))^{2} f_{X} (x) ⟹ f_{X} (x) = 0

3.4 | Moment Generating Functions

Casino Day

Roulette:

R = return on a Red bet of $1,$ R(s)={1,-1}$

E [R] = \frac{1}{38} + (- 1) \frac{20}{38} = - \frac{2}{38}

D = return on first dozen bet, $D (S) = {2, - 1}$

E [D] = (2) \frac{12}{38} + (- 1) \frac{26}{38} = - \frac{2}{38}

N = return on “00” bet: $N (S) = {35, - 1}$

E [N] = 35 \frac{1}{38} + (- 1) \frac{37}{38} = - \frac{2}{38}

All bets are EV $- 0.053$ , but variance is different

Va r (R) = E [R^{2}] - (E [R]^{2}) = 1 - (\frac{2}{38})^{2} = 0.997

Va r (D) = E [D^{2}] - (E (D))^{2} = 2^{2} \frac{12}{38} + (- 1)^{2} \frac{26}{38} - (\frac{3}{38})^{2} = 1.945

Craps: P(A before B) = $P (A, C A, CC A, \dots)$

= i = 0 \sum \infty P (C \dots C A) = i = 0 \sum \infty r^{i} p = \frac{p}{1 - r} = \frac{p}{p + q}

P(winning in craps): F = first roll $F (S) = {2, \dots, 12)$

P (w in) = j = 2 \sum 12 P (w in ∣ F = j) P (F = j)

= P (w in ∣ F = 7, 11) P (7, 11)

+ 2 [P (w in ∣ F = 5) P (F = 4) + P (w in ∣ F = 5) P (F = 5) + P (w in ∣ F = 6) P (F = 6)]

= \frac{8}{36} + 2 [\frac{\frac{3}{36}}{\frac{3}{36} + \frac{4}{36}} \frac{3}{36} + \frac{\frac{4}{36}}{\frac{4}{36} + \frac{5}{36}} \frac{4}{36} + \frac{\frac{5}{36}}{\frac{5}{36} + \frac{6}{36}} \frac{5}{36}] = 0.4929

E V = 1 (0.4929) - 1 (0.5071) = 0.01414

Odds bet

O = E [R] = P \frac{\frac{3}{36}}{\frac{3}{36} + \frac{6}{36}} + (- 1) [1 - \frac{\frac{3}{36}}{\frac{3}{36} + \frac{6}{36}}] ⟹ p = 2

If point is 5 or 9: 3 to 2 (1.5 to 1) If point is 6 or 8: 6 to 5 (1.2 to 1)

Transformations of random variables

What is the probability that a 2 is rolled before an odd number? N = { neither 2 nor odd } $P (N) = \frac{1}{3}$

P (2, N 2, NN 2, \dots) = j = 0 \sum \infty P (N \dots N 2) = j = 0 \sum \infty (\frac{1}{3})^{j} \frac{1}{6} = \frac{1}{6} [\frac{1}{1 - \frac{1}{3}}] = \frac{1}{6} \frac{3}{2} = \frac{1}{4}

Remark: If $X$ is a discrete random variable with PMF $f_{X} (x)$ and $Y = g (X)$ for some $g : R \to R$ , then $Y$ is a discrete random variable. Sometimes we wish to compute $f_{Y} (y)$ based on $f_{X} (x)$ .

Determine $Y (S)$ , the range of $Y$
For each $y \in Y (S)$ , compute $f_{y} (y)$ by determining for which $x \in X (S) g (x) = y$ , then computing

f_{Y} (y) = x : g (x) = y \sum f_{X} (x)

Let $X$ denote the number of kings in a random 3 card hand.

Calculate the PMF of $X$ :

f_{X} (0) = \frac{( 3 48 )}{( 3 52 )} \approx 0.782

f_{X} (1) = \frac{( 1 4 ) ( 2 48 )}{( 3 52 )} \approx 0.204

f_{X} (2) = \frac{( 2 4 ) ( 1 48 )}{( 3 52 )} \approx 0.013

f_{X} (3) = \frac{( 3 4 ) ( 0 48 )}{( 3 52 )} =\approx 0.001

Let $Y = (X - 1)^{2}$ . Compute the PMF of $Y$ $f_{Y} (0) = P (X = 1) = f_{X} (1)$ $f_{Y} (1) = P (X = 0, 2) = f_{X} (0) + f_{X} (2) = 0.795$ $f_{y} (4) = P (X = 3) = 0.001$
Let $Z = 2 X + 1$ . Compute the PMF of $Z$ $f_{Z} (z) = F_{X} (\frac{z - 1}{2})$

Expectation and variance give a simple summary of distribution Skewness

E [\frac{X - E ( X )}{Va r ( X )}]^{3}

Kurtosis

\frac{E ( X - E ( X ) ) ^{4}}{( E ( X - E ( X ) ) ^{2} ) ^{2}}

There exist distributions without expectation: Suppose X is a random variable with probability mass function

f_{X} (x) = \frac{6}{π ^{2}} \frac{1}{x ^{2}}, x = 1, 2, \dots

Then $E (X) = \infty$ and $Va r (X)$ is not defined

Moment generating functions

Definition: Moment Generating Function (MGF) The MGF of a random variable $X$ is given by

M_{X} (t) = E (e^{tX}), t \in R

If $X$ is discrete with PMF $f_{X} (x)$ , then

M_{X} (t) = x \in X (2) \sum e^{t x} f_{X} (x), t \in R

An MGF is well defined if $M_{X} (t) < \infty$ for $t \in [- ϵ, ϵ], ϵ > 0$

Properties of the MGF

Fubin’s Theorem

M_{X} (t) = j = 0 \sum \infty \frac{t ^{j} E ( x ^{j} )}{j !}

So long as $M_{X} (t)$ is defined in a neighborhood of the origin

\frac{d}{d t ^{k}} M_{X} (0) = E (X^{k})

Suppose $X$ is the outcome of a fair die roll. Use the MGF to compute its mean and variance.

f_{X} (x) = \frac{1}{6}, x \in {1, \dots, 6}

M_{X} (t) = E [E^{t x}] = x = 1 \sum 6 e^{t x} \frac{1}{6}

\frac{d}{d t} M_{X} (t) = x = 1 \sum 6 \frac{x e ^{t x}}{6} ⟹ \frac{d}{d t} M_{X} (0) = x = 1 \sum 6 = 3.5

\frac{d ^{2}}{d t ^{2}} M_{X} (t) = x = 1 \sum 6 \frac{x ^{2} e ^{t x}}{6} ⟹ \frac{d ^{2}}{d t ^{2}} M_{X} (0) = x = 1 \sum 6 \frac{x ^{2}}{6} = \frac{91}{6}

Suppose $X$ is the coin flips required to flip a heads. Compute the MGF of $X$ :

X (s) = {1, 2, 3, \dots}, x \in X (S)

f_{X} (x) = P (X = x) = P (T \dots T H) = (\frac{1}{2})^{x}

M_{X} (t) = E [e^{t x}] = x = 1 \sum \infty e^{t x} (\frac{1}{2})^{x} = x = 1 \sum \infty (\frac{e ^{t}}{2})^{x}

well defined if $\frac{e ^{t}}{2} < 1 ⟹ t < ln (2)$ well defined on $(- \infty, ln (2))$ , thus it’s a good MGF because it contains the origin

= \frac{1}{1 - \frac{e ^{t}}{2}} - 1 = (1 - \frac{e ^{t}}{2})^{- 1} - 1

E (x) = \frac{d}{d t} M_{X} (0)

\frac{d}{d t} M_{X} (t) = \frac{d}{d t} [(1 - \frac{e ^{t}}{2})^{- 1} - 1] = - 1 (1 - \frac{e ^{t}}{2})^{- 2} (- \frac{e ^{t}}{2})

Alternatively, you could make it look like a derivative:

E (X) = \frac{1}{2} x = 1 \sum \infty x (\frac{1}{2})^{x - 1}

Theorem: Uniqueness Theorem for MGFs Two functions with the same well-defined MGFs have the same probability distributions (because the MGF is just a transform of the probability distribution)

Suppose $X$ has the following MGF. Compute the distribution.

M (t) = \frac{1}{4} + \frac{11}{20} e^{- 7 t} + \frac{1}{5} e^{11 t}

X (S) = {0, - 7, 11}

f_{X} (0) = \frac{1}{4}, f_{X} (- 7) = \frac{11}{20}, f_{X} (11) = \frac{1}{5}

M_{X} (t) = x \in {0, - 7, 11} \sum e^{t x} f_{X} (x) = \frac{1}{4} e^{0} + \frac{11}{20} e^{- 7 t} + \frac{1}{5} e^{11 t}

Theorem: MGFs are closed under linear combinations

M_{a X + b} (t) = e^{t b} M_{X} (a t)

Ex: $E (X^{2}) = 6, Va r (X) = 2, E (X) = ?$

Z = E (X) = E (x^{2}) - [Va r (X)]^{2} = 6 - 2^{2} = 2

3.5 | Special Discrete Probability Distributions

Discrete Uniform Distribution

Definition: Equality

X \sim Y

Definition: Discrete Uniform Distribution Range is $[a, b]$

X \sim D U (a, b)

Supposed $X \sim D U (a, b)$ . Compute $f_{X} (x), F_{X} (x)$

X (S) = {a, \dots, b}, ∣ X (s) ∣ = b - a + 1

f_{X} (x) = \frac{1}{b - a + 1}, x \in {a, a + 1, \dots, b}

F_{X} (x) = 0 = \frac{[ x ] - a + 1}{b - a + 1} = 1 (x < a) (a \leq x \leq b) (x \geq b)

Mean and Variance of the Uniform Distribution

E (X) = \frac{b + a}{2}

Va r (X) = \frac{( b - a + 1 ) ^{2} - 1}{2}

Calculating the mean:

E (X) = x = a \sum 6 \frac{x}{a} = \frac{1}{n} j = 1 \sum n (a + j - 1) = \frac{1}{n} [an + \frac{n ( n + 1 )}{2} - n] = a + \frac{n + 1}{2} - 1 = \frac{b + a}{2}

MGF

M_{X} (t) = \frac{e ^{a t} - e ^{(b + 1) t}}{n ( 1 - e ^{t} )}

Use the partial sum of geometric series:

j = a \sum b (e^{t})^{j} \frac{1}{n}

Hypergeometric Distribution

Definition: Hypergeometric Distribution Yugioh reference :) Imagine drawing $n$ cards without replacement from an $N$ card deck with $r$ successes. The number of successes you draw is:

X \sim h y p (N, r, n)

The name comes from the hypergeometric series / hypergeometric identity

Calculate the PMF of $X$ :

X (S) = max {0, n - (N - r)}, \dots, min {n, r}

f_{X} (x) = P (X = x) = \frac{( x r ) ( N - x N - r )}{( n N )}

Probabilities of drawing a 5 card hand:

P (J \leq 1) = P (J = 0) + P (J = 1) = \frac{( 0 4 ) ( 8 48 )}{( 8 52 )} + \frac{( 1 4 ) ( 4 48 )}{( 5 52 )} = 0.958

P (J \geq 3) = P (J = 3) + P (J = 4) = \frac{( 3 4 ) ( 2 48 )}{( 5 52 )} + \frac{( 4 4 ) ( 1 48 )}{( 5 52 )} = 0.00175

Binomial Distribution

Definition: Binomial Distribution A Bernoulli trial is a single success/failure experiment where success has probability $p$ A Binomial Distribution is the distribution of successes in $n$ Bernoulli trials

X \sim B in o mina l (n, p) \sim B in (n, p)

Compute the PMF of a binomial distribution

P (X = x) = (x n) p^{x} (1 - p)^{n - x}

Mean and Variance

E (X) = n p

Negative Binomial Distribution

Definition: Negative Binomial Preform Bernoulli trials with a probability of success $p$ until exactly $k$ successes are observed. Then if $X$ denotes the number of failures before $k$ successes

X \sim NB (k, p)

If $y$ is the number of trials required, $y = x + k$

X (S) = {0, 1, \dots}

Y (S) = {k, k + 1, \dots}

Compute the probability function of a NB random variable: $X \sim NB (k, p), X (S) = {0, 1, \dots}, x \in X (S)$

F_{X} (x) = P (X = x) = (x x + k - 1) (1 - p)^{x} p^{k}

(number of sequences with $x$ failures and $k - 1$ successes)

Geometric Distribution

Definition: Geometric Distribution Perform Bernoulli trials with probability of success $p$ until one success is observed. $X \sim NB (1, p)$ is written as $X \sim G eo (p)$

F_{X} (x) = (1 - p)^{x} p

MGF of Geometric Distribution:

M_{X} (t) = \frac{p}{1 - ( 1 - p ) e ^{t}}, t < l o g (\frac{1}{1 - p})

$X \sim G eo (p), f_{X} (x) = p (1 - p)^{x}, x \in {0, 1, \dots}$

M_{X} (t) = E [e^{t x}] = x = 0 \sum \infty e^{t x} p (1 - p)^{x} = P x = 0 \sum \infty [e^{t} (1 - p)]^{x} < \infty

This is well-defined at the origin, and thus the uniqueness theorem applies

= p \frac{1}{1 - e ^{t} ( 1 - p )} = \frac{p}{1 - e ^{t} ( 1 - p )}

Mean and Variance

E (X) = \frac{1 - p}{p} = \frac{1}{p} - 1, Va r (x) = \frac{1 - p}{p ^{2}}

M_{X} (t) = \frac{p}{1 - ( 1 - p ) e ^{t}} ⟹ \frac{d}{d t} M_{x} (t) = - p (1 - (1 - p) e^{t})^{- 2} [- (1 - p) e^{t}] = \frac{( 1 - p ) e ^{t}}{( 1 - ( 1 - p ) e ^{t} ) ^{2}}

⟹ \frac{d}{d t} M_{X} (0) = E (X) = \frac{1 - p}{1 - ( 1 - p )} = \frac{1 - p}{p} = \frac{1}{p} - 1

What is the probability that a coin takes more than 5 flips to show heads? $X$ = # of tails before observing 1 heads $\sim G eo (\frac{1}{2})$

P (Fl i p s > 5) = P (X \geq 5) = P (X > 4)

= x = 5 \sum \infty (\frac{1}{2}) (\frac{1}{2})^{x} = (\frac{1}{2}) (\frac{1}{2})^{5} x = 0 \sum \infty (\frac{1}{2})^{x} = (\frac{1}{2}) (\frac{1}{2})^{5} \frac{1}{1 - \frac{1}{2}} = \frac{1}{2}^{5}

Trivial Pursuit: What’s the probability the number of rolls exceeds the mean?

P (X > 6) = P (X - 1 > 5) = x = 6 \sum \infty (\frac{1}{6}) (\frac{5}{6})^{x} = (\frac{1}{6}) (\frac{5}{6})^{X}

= (\frac{1}{6}) (\frac{5}{6})^{6} x = 0 \sum \infty (\frac{5}{6})^{x} = (\frac{1}{6}) (\frac{5}{6})^{6} \frac{1}{1 - \frac{5}{6}} = \frac{5}{6}^{=} 0.334

Question: Suppose for a $p \in (0, 1), X \sim NB (5, p)$ and $Y \sim G eo (p)$ :

$E (X) > E (Y)$

E (X) = \frac{5 ( 1 - p )}{p} > \frac{1 - p}{p} = E (Y)

For all $t > 0, P (X > t) > P (Y > t)$
$Va r (X) > Va r (Y)$

P (X = 0) = p^{5} = P (Y = 0)

$P (X = 0) \leq P (Y = 0)$

{X > t} \geq {Y > t}

Remark: Memory-less property of Geometric Distribution If $X \sim G eo (p)$ , then

P (X \geq t + s ∣ S \geq t) = P (X \geq s)

Poisson

Definition: Poisson Distribution (paramater is the mean) Binomial distribution as $n \to \infty$ and $p \to 0$

f_{X} (x) = e^{- λ} \frac{λ ^{x}}{x !}, x = 0, 1, 2, 3, \dots

x = 0 \sum \infty f_{X} (x) = x = 0 \sum \infty e^{- λ} \frac{λ ^{x}}{x !} = e^{- λ} e^{λ} = 1

Remark: If $X \sim P o i (λ), E (X) = λ$

Example: Poisson approximation of Binomial Suppose $λ \in (0, \infty)$ . Consider a sequence of Binomial experiments each consisting of $n$ trials in which $p = p_{n} = \frac{λ}{n}$ . Then for any fixed $x \in {0, 1, \dots}$

n \to \infty lim (x n) p_{n}^{x} (1 - p_{n})^{n - x} = e^{- λ} \frac{λ ^{x}}{x !}

Let $λ = n p$ . If $n$ is large and $p$ is close to zero,

(x n) p^{x} (1 - p)^{n - x} \sim \frac{e ^{- λ} λ ^{x}}{x !}

Ex: A bit error occurs for a given data transmission method independently in one out of every 1000 bits transferred. Suppose a 64 bit message is sent using the transmission system. What is the probability that there are exactly 2 bit errors? Approximate this using a Poisson approximation.

X = # of errors $\sim B in (64, \frac{1}{1000})$

P (X = 2) = (2 64) (\frac{1}{1000})^{2} (\frac{999}{1000})^{62} = 0.001894

X \approx P o i (\frac{64}{1000}) ⟹ P (X = 2) \approx exp (\frac{- 64}{1000} \frac{64}{1000}^{2} \frac{1}{2 !}) = 0.001921

If $X \sim P o i (μ), Va r (X) = μ$

Shiny versions of Pokemon are possible to encounter and catch starting in Generation 2 (Pokemon Gold/Silver). Normal encounters with Pokemon while running in grass occur according to a Poisson process with rate 1 per minute on average. 1 in every 8192 encounters will be a Shiny Pokemon, on average.

If you run around in grass for 15 hours, what is the probability you will encounter at least one Shiny pokemon? $X_{15}$ = # of shiny pokemon encountered in 15 hours

\sim P o i (\frac{15 \times 60}{8192})

P (X \geq 1) = 1 - P (X = 0) = 1 - e^{\frac{15 \times 60}{8192}} \frac{λ ^{x}}{x !}

How long would you have to run around in grass so that you have a better than 50 percent chance of encountering at least one Shiny pokemon? $X_{t}$ = # of shiny pokemon after t hours

\sim P o i (\frac{t \times 60}{8192})

P (X_{t} \geq 1) = 1 - e^{\frac{- t \times 60}{8192}} = 0.5

t = - \frac{8192}{60} ln 0.5 = 94.63

4 | Multivariate Discrete Probability Distributions

4.1 | Basic Terminology and Techniques

Multiple (discrete) random variables

Joint PMF

f (x, y) = P (X = x, Y = y)

If $A \subset R^{2}$

P ((X, Y) \in A) = (x, y) \in A \sum f (x, y)

Marginal = add up on the margins to get the PMFs of $x, y$

f_{X} (x) = P (X = x) = y \in Y (S) \sum f (x, y)

Def: Independence of Random Variables Discrete random variables $X, Y$ with joint PMF $f (x, y)$ and marginal PMFs $f_{X} (x), f_{Y} (y)$ are said to be independent variables if

f (x, y) = f_{X} (x) f_{Y} (y)

P (X = x, Y = y) = P (X = x) P (Y = y)

Let $f (x, y) = f_{X} (x) f_{Y} (y)$ , then this is supported on

X (S) \times Y (S) = {(x, y) : x \in X (s), y \in Y (S)}

Properties

0 \leq f (x, y) \leq 1

some other thing but i was on twitter

Ex: $X, Y$ has the following joint PMF.

f (x, y) = \frac{1}{6} (\frac{1}{2})^{x} (\frac{2}{3})^{y}

Compute the marginal PMFs $f_{X} (x), f_{Y} (y)$

f_{X} (x) = y + Y (s) \sum f (x, y) = y = 0 \sum \infty (\frac{1}{6}) (\frac{1}{2})^{x} (\frac{2}{3})^{y} = \frac{1}{6} (\frac{1}{2})^{x} y = 0 \sum \infty (\frac{2}{3})^{y} = \frac{1}{2} (\frac{1}{2})^{x} \sim G eo (\frac{1}{2})

f_{Y} (y) = x + X (s) \sum f (x, y) = x = 0 \sum \infty (\frac{1}{6}) (\frac{1}{2})^{x} (\frac{2}{3})^{y} = \frac{2}{6} (\frac{2}{3})^{y} = \frac{1}{3} (\frac{2}{3})^{y} \sim G eo (\frac{1}{3})

Compute $P (X < Y)$

(x, y), x < y \sum f (x, y) = x = 0 \sum \infty y = x + 1 \sum \infty (\frac{1}{6}) (\frac{1}{2})^{x} (\frac{2}{3})^{y} = x = 0 \sum \infty (\frac{1}{6}) (\frac{1}{2})^{x} (y = x + 1 \sum \infty (\frac{2}{3})^{y}) = x = 0 \sum \infty (\frac{1}{6}) (\frac{1}{2})^{x} (\frac{2}{3})^{x + 1} 3 = x = 0 \sum \infty \frac{1}{2} (\frac{2}{3}) (\frac{1}{3})^{x} = (\frac{1}{2}) (\frac{2}{3}) (\frac{3}{2}) = \frac{1}{2}

If $X, Y$ are independent, their marginal PMFs satisfy

f (0, 0) = f_{X} (0) f_{Y} (0)

Conditional Probability

f_{Y} (y ∣ x) = P (Y = y ∣ X = x) = \frac{f ( x , y )}{f _{X} ( x )}

4.2 | Multinomial Distribution

Definition: Multinomial Distribution

Individual trials have $k$ possible outcomes with the i-th being denotes $p_{i}$

Trials are repeated $n$ times, so the i-th one occurs $X_{i}$ times Then, $X_{1}, \dots X_{k}$ have a Multinomial Distribution with parameters $n$ , $p_{1} m \dots p_{k}$

Multinomial Distribution

(X_{1}, \dots X_{K}) \sim M u lt i (n, p_{1}, \dots, p_{k})

f (x_{i}, \dots, x_{k}) = \frac{n !}{x _{1} ! \dots x _{k}} p_{1}^{x_{1}} \dots p_{1}^{x_{k}}, x_{1} + \dots + x_{k} = n

X_{i} \sim B in (n, p_{i})

X_{i} ∣ X_{j} = m \sim B in (n - m, \frac{p _{i}}{1 - p _{j}})

Roulette

(B, R, G) \sim M u lt in o mia l (10, \frac{18}{38}, \frac{18}{38}, \frac{2}{38})

P (B = 4, R = 4, G = 2)

P = (\frac{18}{38})^{4} (\frac{18}{38})^{4} (\frac{2}{38})^{2} \frac{10 !}{4 ! 4 ! 2 !}

General PMF

f (x_{1}, \dots, x_{k}) = \frac{n !}{x _{1} ! x _{2} ! \dots x _{k} !} p_{1}^{x_{1}} \dots p_{1}^{x_{k}}

Properties

If $(X_{1}, \dots, X_{k}) \sim M u lt i (n, p_{1}, \dots, p_{k})$ , then $X_{i} \sim B in (n, p_{i})$
$X_{i} = n - \sum_{j \neq = i} X_{j}$ so that $X_{i}$ are dependent
Suppose $(X_{1}, \dots, X_{m}) \sim M u lt i (n, p_{1}, \dots, p_{k})$ . Then

X_{i} ∣ X_{j} = m \sim B in o m (n - m, \frac{p _{i}}{1 - p _{j}})

(Intuition: the “margins” of a multinomial are a binomial)

Ex: Roulette

R ∣ B = 5 \sim B in (10 - 5, \frac{\frac{18}{38}}{1 - \frac{18}{38}}) \sim B in (5, \frac{18}{20})

P (R = 5 ∣ B = 5)

= (5 5) = (\frac{18}{20})^{5} (\frac{2}{20})^{0} = (\frac{18}{20})^{5} = 0.59

Ex: 2 hearts, 2 spades, 1 diamond with replacement (w/o would be multi-hypergeometric)

(C, D, H, S) \sim M u lt i (5, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4})

P (H = 2, S = 2, D = 1) = (\frac{1}{4})^{2} (\frac{1}{4})^{2} (\frac{1}{4}) (\frac{5 !}{2 ! 2 ! 1 !})

Ex: Given 3 hearts in 5 cards, what is the possibility of no spades?

S ∣ H = 3 \sim B in (2, \frac{\frac{1}{4}}{1 - \frac{1}{4}}) \sim B in (2, \frac{1}{3})

P (S = 0 ∣ H = 3) = (\frac{2}{3})^{2}

Ex: What if we add two distributions?

H + S \sim B in o m (5, \frac{1}{2})

4.3 | Expectations, Covariance, Correlation

Distributions of multivariate transformations: Suppose that

h : R^{2} \to R

Then, for jointly distributed random variables $X, Y, U = h (X, Y)$ is a random variables. If $X, Y$ have joint p.f. $f (x, y)$ , then the probability function of $U$ is given by

f_{U} (t) = P (U = t) = (x, y) : H (x, y) = t \sum f (x, y)

Show that $X + Y \sim P o i (μ + λ)$

f_{X} (x) = \frac{e ^{- μ} μ ^{x}}{x !}

f_{X} (y) = \frac{e ^{- λ} λ ^{y}}{y !}

f (x, y) = \frac{e ^{- μ} μ ^{x}}{x !} \frac{e ^{- λ} λ ^{y}}{y !}

T = X + Y, T (S) = {0, 1, 2, \dots}

For $t \in T (S)$

P (T = t) = P (X + Y = t) = P (X + Y = t \cap S) = P (X + Y = t \cap \cup_{x \in X (S)} {X = x}) = x \in X (S) \sum P (X + Y = t \cap {X = x}) = x = 0 \sum \infty f (x, t - x) = x = 0 \sum t e^{- (μ + λ)} \frac{\frac{μ ^{x}}{x !} λ ^{t - x}}{( t - x )!} = \frac{e ^{- μ + λ}}{t !} x = 0 \sum t \frac{t !}{x ! ( t - x )!} μ^{x} λ^{t - x} = \frac{e ^{- μ + λ} ( μ + λ ) ^{t}}{t !}

Properties

Convolution formula

P (X + Y = t) = y \in Y (S) \sum f (t - y, y) = x \in X (S) \sum f (x, t - x)

Binomial is closed under addition

B in (n, p) + B in (m, p) \sim B in (n + m, p)

Geometric + Geometric = Negative Binomial

G eo (p) + G eo (p) \sim NB (2, p)

Sums of independent:

Poissons are Poisson
Geometrics with same $p$ are NB
Binomials with same $p$ are Binomial

Definition: Multivariate Expectation

E (g, (X, Y)) = (x, y) \sum g (x, y) f (x, y)

E (g (X_{1}, \dots, X_{n}))) (x_{1}, \dots, x_{n}) \sum g (x_{1}, \dots, x_{n}) (x_{1}, \dots, x_{n})

Properties of EV of Jointly Distributed Random Variables

Closed under linear combinations

E [a \cdot g_{1} (X, Y) + b \cdot g_{2} (X, Y)] = a \cdot E (g_{1} (X, Y)) + b \cdot E (g_{2} (X, Y))

Decomposing addition

E (X + Y) = X \sum x f_{X} (x) + y \sum y f_{Y} (y) = E (X) + E (Y)

Covariance

C o v (X, Y) = E [(X - E [X]) (Y - E [Y])] = E [X Y] - E [X] E [Y]

If $X, Y$ independent

E [g (X) h (Y)] = E [g (X)] E [h (Y)]

C o v (X, Y) = 0

Correlation

corr (X, Y) = p = \frac{C o v ( X , Y )}{S D ( X ) S D ( Y )}

Bilinear form property of $C o v$

C o v (a X, bY) = ab C o v (X, Y)

C o v (X + Y, Z + W), = C o v (X, Z) + C o v (X, W) + C o v (Y, Z) + C o v (Y, W)

If $X, Y$ independent, then

M_{X + Y} (t) = M_{X} (t) M_{Y} (t)

4.4 | Linear Combinations of Random Variables

Expectation of random variable combinations

E (i = 1 \sum n a_{i} X_{i}) = i = 1 \sum n a_{i} E (x_{i})

Variance of random variable combinations

Va r (i = 1 \sum n a_{i} X_{i}) = i = 1 \sum n a_{i}^{2} Va r (X_{i}) + 1 \leq i \neq = j \leq n \sum a_{i} a_{j} C o v (X_{i}, X_{j}) = i = 1 \sum n a_{i}^{2} Va r (X_{I}) + 2 1 \leq i \leq j \leq n \sum a_{i} a_{j} C o v (X_{i}, X_{j})

Indicator (Bernoulli) Random Variables: Let $A \subset S$ be an event. We say that $1_{A}$ is the indicator random variable of event $A$

1_{A} = {10 w \in A w \in \overline{A}

Properties

E (1_{A}) = P (A)

Va r (1_{A}) = P (A) (1 - P (A))

C o v (1_{A}, 1_{B}) = P (A \cap B) - P (A) P (B)

4.5 | Markov’s Inequality, Chebyshev’s Inequality, Law of Large Numbers

Weak Law of Large Numbers: Suppose $X_{1}, \dots X_{n}$ are independent RVs with mean $μ$ and variance $σ^{2}$ , and

\overline{X} = \frac{1}{n} i = 1 \sum n X_{i}

is the sample mean. Then for all $ϵ > 0$ ,

n \to \infty lim P (∣ \overline{X} - μ ∣ > ϵ) = 0

Markov’s Inequality If $X$ is an RV with $E [∣ X ∣] < \infty$ , then for any $ϵ > 0$ ,

P (∣ X ∣ > ϵ) \leq \frac{E [ ∣ X ∣ ]}{ϵ}

Chebyshev’s Inequality If $X$ is a RV with $E (X) = μ$ , $Va r (X) = σ^{2}$ , then for any $ϵ > 0$

P (∣ X ∣ > ϵ) \leq \frac{E [ X ^{2} ]}{ϵ ^{2}}

P (∣ X - μ ∣ > ϵ) \leq \frac{σ ^{2}}{ϵ ^{2}}

For 10 coin tosses, what’s the probability that the sample mean $\overline{X}$ deviates by more than $0.01$ from $p = 0.5, n = 10$

\overline{X} = \frac{1}{10} i = 1 \sum 10 X_{i} \sim B in (10, 0.5)

P (∣ \overline{X} - 0.5∣ > 0.01) = 1 - P (\overline{X}_{1} + \dots + \overline{X}_{10} = 5) = 1 - P (X = 5), X \sim B in (10, 0.5) = 1 - (5 10) 1 0^{5} = 1 - \frac{63}{256} \approx 0.75

4.6 | Conditional Probability Distributions

Conditional distribution:

f_{X ∣ Y} (x ∣ y) = P (X = x ∣ Y = y) = \frac{f ( x , y )}{f _{Y} ( y )}

Conditional mean:

E [X ∣ Y = y] = x \in X (S) \sum x f_{X ∣ Y} (x ∣ y)

E [g (X, Y) ∣ Y = y] = x \in X (S) \sum g (x, y) f_{X ∣ Y} (x ∣ y)

5 | Univariate Continuous Probability Distributions

5.1 | Continuous Random Variables

How to model continuous RVs?

Define the analog of the probability function (hard)
Model relative frequency from the histogram (calc)

5.2 | Functions of Random Variables

Definition: Continuous RV $X$ has probability density function (pdf) $f (x)$ if

f (x) \geq 0

\int_{- \infty}^{\infty} f (x) d x = 1

P (a \leq X \leq b) = \int_{a}^{b} f (x) d x

Support of a pdf is

s u pp (f) = {x \in R f (x) \neq = 0}

Definition: CDF of a random variable X is

F_{X} (x) = P (X \leq x)

If $X$ is continuous with pdf $f (x)$ , then

F_{X} (x) = \int_{- \infty}^{x} f (y) d y

Moreover, by FTC part I:

\frac{d}{d x} F_{X} (x) = f (x)

FTC part II:

P (a \leq X \leq b) = F_{X} (b) - F_{X} (a) = \int_{- \infty}^{b} f (y) d y - \int_{- \infty}^{a} f (y) d y = \int_{a}^{b} f (y) d y

Properties of the CDF of a continuous RV:

$F (x)$ is continuous

$lim_{x \to \infty} F (x) = 1, lim_{x \to - \infty} F (x) = 0$

$F (x)$ is differentiable except at a countable number of exceptional points

5.3 | Expectation of a Random Variable

If $X$ is a continuous RV with pdf $f (x)$ and $g : R \to R$ , then

E (g (X)) = \int_{\infty}^{\infty} g (x) f (x) d x

It follows that

E (X) = \int_{- \infty}^{\infty} x f (x) d x

Va r (X) = E ([X - E (X)^{2}]) = \int_{- \infty}^{\infty} (x - E (X))^{2} f (x) d x

Cool things

Expectation still linear

E [a g (x) + b] = a E [g (X)] + b

Variance shortcut still works

Va r (X) = E (X^{2}) - (E (X))^{2}

Definition: Moment Generating Function

M_{X} (t) = E [e^{tX}] = \int_{- \infty}^{\infty} e^{t x} f (x) d x

5.4 | Special Continuous Probability Distributions

Theorem: Let $X$ be a continuous random variable with pdf $f_{X} (x)$ . Suppose that $g$ is a strictly monotonic differentiable function on the range of $X$ . Then, the pdf of $Y = g (x)$ in the region where $f_{Y} (y) > 0$ is given by the following, where $g^{- 1}$ denotes the inverse of $g$

f_{Y} (y) = f_{X} (g^{- 1} (y)) ∣ \frac{d g ^{- 1} ( y )}{d y} ∣

Let $X$ be a continuous RV with pdf $f_{X} (x)$ . Then, $Y = a X + b$ has pdf

f_{Y} (y) \frac{1}{∣ a ∣} f_{X} (\frac{y - b}{a})

Definition: We say that $X$ has a (continuous) uniform distribution on $(a, b)$ , or $X \sim U (a, b)$ , if $X$ has pdf

f (x) = {\frac{1}{b - a} 0 x \in (a, b) otherwise

Properties

E (X) = \frac{a + b}{2}

Va r (X) = \frac{( b - a ) ^{2}}{12}

M_{X} (t) = {\frac{e ^{t b} - e ^{t a}}{t ( b - a )} 1 t \neq = 0 t = 0

Definition: We say that $X$ has an exponential distribution $(X \sim e x p (θ))$ if the density of $X$ is

f (x) = {\frac{1}{θ} e^{- x / θ} 0 x > 0 x \leq 0

Definition: Quantile / Percentile The 100 x q-th percentile (or $100 \times q %$ quantile) of the distribution of $X$ is the value $c_{q}$ such that

F_{X} (c_{q}) = q

The quantile is the “inverse” if the CDF, and $F_{X}^{- 1}$ is called the quantile function. The median of a distribution is its 50-th percentile.

Definition: Normal Distribution $X \sim N (μ, σ^{2})$ with mean $μ$ and variance $σ^{2}$ if the density is

f (x) = \frac{1}{2 π σ ^{2}} e^{\frac{- ( x - μ ) ^{2}}{2 σ ^{2}}}

Properties

Symmetric about the mean:

P (X \leq μ + t) = P (X \geq μ - t)

Unimodal with a peak at $μ$
(Standardization)

Z \sim N (0, 1) and X = σ Z + μ ⟹ X \sim N (μ, σ^{2}) ⟹ Z = \frac{X - μ}{σ} \sim N (0, 1)

mean $μ$ and variance $σ^{2}$
MGF looks like the density (that’s why it’s special)

M_{X} (t) = e^{μ t + \frac{σ ^{2} t ^{2}}{2}}

Definition: Standard normal random variable $X \sim N (0, 1)$ if

ϕ (x) = \frac{1}{2 π} e^{\frac{- x ^{2}}{2}}

and the CDF is

ϕ (x) = \int_{- \infty}^{x} \frac{1}{2 π} e^{\frac{- y ^{2}}{2}} d y

PROBLEM: Normal distribution has no elementary antiderivative ( $e^{- x^{2}}$ )! Evaluate them numerically with the table

Definition: The 68-95-99.7 Rule: If $X \sim N (μ, σ^{2})$ , then the probability that $X$ is within ${1, 2, 3}$ standard deviations of the mean is:

P (μ - σ \leq X \leq μ + σ) \sim .68

P (μ - 2 σ \leq X \leq μ + 2 σ) \sim .95

P (μ - 3 σ \leq X \leq μ + 3 σ) \sim .997

Theorem: Joint Density Function If $X_{i} \sim N (μ_{i}, σ_{i}^{2})$

i = 1 \sum n a_{i} X_{i} \sim N (i = 1 \sum n a_{i} μ_{i} i = 1 \sum n a_{i}^{2} σ_{i}^{2})

Sample Mean Suppose that $X_{1}, \dots, X_{n}$ are independent and that $X_{i} \sim N (μ, σ^{2})$

\overline{X} = \frac{1}{n} i = 1 \sum n X_{i} \sim N (μ, \frac{σ ^{2}}{n})

5.5 | Central Limit Theorem

Theorem (Central Limit Theorem) Suppose $X_{1}, X_{n}$ are independent RVs, each with a common CDF $F$ . Suppose further that $E (X_{i}) = μ, Va r (X_{i}) = σ^{2} < \infty$ . Then

P (\frac{X - μ}{\frac{σ}{n}} \leq x) \to ϕ (x)

as $n \to \infty$ . In other words, if $n$ is large,

\overline{X} \approx (μ, \frac{σ ^{2}}{n}), i = 1 \sum n X_{i} \approx N (n μ, nσ)

Binomial to Normal If $X_{n} \sim B in (n, p)$ , then for large $n$

\frac{X _{n} - n p}{n p ( 1 - p )} \approx N (0, 1)

Poisson to Normal If $X_{λ} \sim P o i (λ)$ , then for large $λ$

\frac{X _{λ} - λ}{λ} \approx N (0, 1)

Suppose that you are interested in buying some red potatoes from a local grocery store. The potatoes are priced at 1.60 per kilogram. The average weight of a red potato at the store is 200 grams with a standard deviation of 25 grams. The distribution of the weights of potatoes is unknown.

Suppose you take a random sample of 49 potatoes. What is the approximate probability that the average weight of the 49 potatoes sampled is between 195 and 205 grams (inclusive)?

Use CLT to approximate sample mean with normal distribution

\overline{X}_{49} \approx N (200, \frac{2 5 ^{2}}{49})

P (195 \leq \overline{X}_{49} \leq 205) = P (\frac{195 - 200}{\frac{2 5 ^{2}}{49}} \leq Z \leq \frac{205 - 200}{\frac{2 5 ^{2}}{49}})

= P (- 1.4 \leq Z \leq 1.4) = 2 ϕ (1.4) - 1 \approx 0.838

Now, suppose you take a random sample of 36 potatoes. What is the approximate probability that the random sample of 36 potatoes will cost more than 12.

T_{36} = i = 1 \sum 36 x_{i} \approx N (36 \times 200, 36 \times 2 5^{2})

C_{36} = \frac{1.60}{1000} \times T_{36} \approx N (\frac{1.60}{1000} \times 36 \times 200, (\frac{1.60}{1000})^{2} 36 \times 2 5^{2})

P (C_{36} > 12) = P (Z > 2) = 0.0228

Now, suppose that you only have 10 in cash, and you want to buy as many potatoes as you can with this 10. Determine the largest number of potatoes that you can select (randomly, of course), in order to be at least 85% certain that their overall cost will be at most 10.

C_{n} \approx N (\frac{1.60}{1000} \times n \times 200, (\frac{1.60}{1000})^{2} \times n \times 2 5^{2})

P (C_{n} \leq 10) \geq 0.85

P (Z \leq \frac{10 - m _{n}}{v _{n}}) \geq 0.85

\frac{10 - m _{n}}{v _{n}} = x_{0..85} = 1.04

n = 30

Julian's Vault

Explorer

STAT 230

1 | Introduction to Probability

1.1 | Definitions of Probability

1.2 | Mathematical Probability Models

1.3 | Counting in Uniform Probability Models

Counting Rules

Permutations and Combinations

Multinomial Coefficient Theorem

2 | Probability Rules and Conditional Probability

2.1 | Use of Sets

Venn Diagrams

2.2 | Addition Rules for Unions of Events

Inclusion-Exclusion Principle

2.3 | Dependent and Independent Events

Independence

2.4 | Conditional Probability and Product Rules

Conditional Probability

Product Rules

Law of Total Probability

Bayes Theorem

3 | Univariate Discrete Probability Distributions

3.1 | Discrete Random Variables

3.2 | Functions of Random Variables

3.3 | Expectation of a Random Variable

Expected Values

Variance

3.4 | Moment Generating Functions

Casino Day

Transformations of random variables

Moment generating functions

3.5 | Special Discrete Probability Distributions

Discrete Uniform Distribution

Hypergeometric Distribution

Binomial Distribution

Negative Binomial Distribution

Geometric Distribution

Poisson

4 | Multivariate Discrete Probability Distributions

4.1 | Basic Terminology and Techniques

4.2 | Multinomial Distribution

4.3 | Expectations, Covariance, Correlation

4.4 | Linear Combinations of Random Variables

4.5 | Markov’s Inequality, Chebyshev’s Inequality, Law of Large Numbers

4.6 | Conditional Probability Distributions

5 | Univariate Continuous Probability Distributions

5.1 | Continuous Random Variables

5.2 | Functions of Random Variables

5.3 | Expectation of a Random Variable

5.4 | Special Continuous Probability Distributions

5.5 | Central Limit Theorem

Table of Contents