Lecture Notes for Simulation

31 January 2005 - Basic Probability

What is Probability?

• Probability is a method for characterizing uncertainty.

  • A probabilistic characterization can be manipulated to reduce (but not eliminate) uncertainty.

• Probability is a concise description of massive amounts of data.

  • The description is lossy.

  • The probabilistic description can't be inverted to get the original data back

• Statistics is analysis of massive ammounts of data.

Why Probability?

• It's important to be able to deal with uncertainty.

• Probability permeates simulation.

  • Probabibility helps define the model used.

    • Sometimes it may be the model.

  • Probability can define and generate the simulation input.

  • Probability (and statistics) help analyze and characterize the simulation output.

Experiments and Outcomes

• An experiment is a box with a button on top.

• Push the button and get an outcome.

  • Pushing the button is called a trial.

• For example,

  • The coin landing tails up is an outcome of a coin-flip experiment.

  • Horse 1 winning is an outcome of a six-horse race experiment.

The Sample Space

• An experiment's sample space is the set of all possbile outcomes.

  • For the coin-toss experiment, SS = { H, T }.

  • For the horse-race experiment, SS = { 1, 2, 3, 4, 5, 6 }.

Experimental Uncertainty

• Experimental outcomes are (usually) uncertain.

  • Otherwise, why bother to experiment?

• Can we say anything non-trivial about experimental outcomes?

  • We must speak of outcome tendencies or likelyhoods.

  • "Non-trivial" in this case menas "without certainty".

• Run lots of experiments to gain experience with outcomes.

  • For 1000 coin-toss trials, tails came up 480 times, heads 520 times.

  • For 1000 horse-race trials,

Outcome Probabilities

• Suppose outcome o_i occurs n_i times in an N-trial experiment.

• The relative frequency of o_i is RF(o_i) = n_i/N

  • Relative frequency is a measure of tendency.

• Some relative frequency properties.

  • 0 <= RF(o) <= 1.0 for any outcome in any experiment.

  • sum(o in SS, RF(o)) = 1.

• The probability of outcome o_i is RF(o_i) as N goes to infinity.

  • By virtue of its definition, probability has properties similar to relative frequency's properties.

Events

• An event is a subset of an experiment's sample space.

  • Events are the first chance at abstracting experiment outcomes.

• An event E occurs if experiment outcome o is in E.

• Let an experiment be three coin flips in succession.

  • SS = { HHH, HHT, ..., TTT }

  • Event E	=	{ o in SS | The first and last flips are different. }
    	=	{ HHT, HTT, THH, TTH }

Event Probabilities

• Suppose event E_i occurs n_i times in an N-trial experiment.

• The relative frequency of E_i is RF(E_i) = n_i/N

• The probability of event E_i is RF(E_i) as N goes to infinity.

Events as Sets

• Given the two events A and B:

  • union(A, B) = { o in SS | o in A or o in B }

  • intersection(A, B) = { o in SS | o in A and o in B }

    • A and B are mutually exclusive if intersection(A, B) = { }

  • A, the compliment of event A, is { o in SS | o not in A }.

Probability Axioms

• The probability function P() assigns to each event A in SS a number in [0, 1] such that

  1. 0 <= P(A) <= 1 for any A.

  2. P(SS) = 1.

  3. If A₁, A₂, ... is an infinite sequence of mutually exclusive events, then
     P(union(i = 1 to infinity, A_i)) = sum(i = 1 to n, P(A_i))

Event Probabilities

• The union, intersection, and compliment of events are events.

  • What's their probability?

• P(union(A, B)) =

• P(intersection(A, B)) =

• P(A) = 1 - P(A)

More Examples

• An example experiment: a six-horse race.

  • The sample space are permutations of (1, 2, 3, 4, 5, 6).

  • The event "horse 2 wins the race" is the subset A = { all permutations starting with 2. }

  • Recall A = { horse 2 wins the race. }

  • All race outcomes equally likely, i.e., P({w}) = 1/6!, then
    P(A) = sum(o in A, P({ o })) = 5!/6! = 1/6

  • If B = { horse 3 comes in second place } then P(B) = 1/6.

    • P(intersection(A, B)) = 4!/6! = 1/6*5 = 1/30

    • P(union(A, B)) = P(A) + P(B) - P(intersection(A, B)) = 1/3 - 1/30 = 3/10.

Example

• Roll a six sided die once.

• Event A = { 4 comes up. }

• Event B = { 3, 4, 5, or 6 comes up. }

• Then P(A | B) = P(AB)/P(B) = P(A)/P(B) = (1/6)/(4/6) = 1/4.

• Also P(B | A) = P(AB)/P(A) = P(A)/P(A) = 1.

• What is P(A | an even number comes up)?

Independent Events

• Events A and B are independent if
  P(A | B) = P(A) and P(B | A) = P(B)

• The independence of A and B means knowing that one event occurred does not influence the probability of the other event occurring.

• From the definition of conditional probability, events A and B are independent if and only if P(AB) = P(A)P(B).

• Example: flip a fair coin three times.

  • Each outcome o in SS is a triple in { H, T }³

  • For each o in SS, P(o) = 1/8.

  • Are events A and B independent if

    • A = { H on the first toss }, B = { T on the third toss }

    • A = { H on the first toss }, B = { at least one T }

Probability Properties

• Some probability properties:

  • P(A) = 1 - P(A)

  • For all events A and B,
    P(union(A, B)) = P(A) + P(B) - P(intersection(A, B))

Event Probability Examples

• The experiment flips a fair coin three times.
  SS = { HHH&comma; HHT&comma; ...&comma; TTT }

• P({o}) = 1/8 for each o in SS.

• If A = { The second toss is H, the third toss is T } then

  • A = { HHT&comma; THT }.

  • P(A) = sum(o in A, P({o})) = 1/8 + 1/8 = 1/4.

  • P(A) = 1 - P(A) = 3/4.

• If A = { The first toss is H } and B = { There are at least two Ts }, then

  • P(intersection(A, B)) = P({ HTT }) = 1/8.

  • P(union(A, B))	=	P(A) + P(B) - P(intersection(A, B))
    	=	1/2 + 1/2 - 1/8
    	=	7/8

Random Variables

• A random variable X is a function from a sample space to the real numbers.

• Can translate probability P() to random variable X via the relation
  P(X in (a, b]) = P(o in SS : X(o) in (a, b])

• Example: flip a coin three times.
  SS = { HHH, HHT, ..., TTT }

  • Define the random variable X(o) to be the number of heads in an outcome.

  • If o = HTH, then X(o) = 2 and

    P(X = 2)	=	P({ o in SS | X(o) = 2})
    	=	P({ HHT, HTH, THH })
    	=	3/8

Cumulative Density Function

• A random variable can be defined by the cumulative distribution function (CDF) F:
  F(x) = P(X <= x)

• CDF properties:

  • 0 <= F(x) <= 1

  • F(x) is a non-decreasing function.

  • F(x) is 0 as x goes to minus infinity.

  • F(x) is 1 as x goes to infinity.

• Notation: X ~ F means random variable X has CDF F.

• Definition: The support of random variable X is set of all possible values X can take on.

This page last modified on 9 February 2005.