Bayesian Inference

1. Probability as Extended Logic

From Logic to Plausible Reasoning

Traditional deductive logic is based on syllogisms that provide certain conclusions:

But real-world reasoning often requires handling uncertainty. Consider this scenario:

Suppose some dark night a policeman walks down a street. He hears a burglar alarm, looks across the street, and sees a jewelry store with a broken window. Then a gentleman wearing a mask comes crawling out of the window, carrying a bag full of jewelry. The policeman decides immediately that the gentleman is a thief. How does he decide this? — E.T. Jaynes, Probability Theory

Let's analyze this:

• A: "the gentleman is a thief"
• B: "the gentleman is wearing a mask and exited a broken window holding a bag of jewelry"

The policeman appears to use a weak syllogism:

Developing a Theory of Plausible Reasoning

To formalize reasoning under uncertainty, our theory must satisfy:

1. Degrees of plausibility are represented by real numbers
2. Qualitative correspondence with common sense
3. Consistency:
   • All valid reasoning routes give the same result
   • Equivalent states of knowledge have equivalent plausibilities

The Rules of Probability

From Cox's axioms, we derive:

Notation and Continuous Variables

For continuous variables, we use probability densities:

• If X can take any value between 0 and 10, then $P(X = x) = 0$ always
• Instead, define: $P(x < X < x + \delta) = \delta f(x)$
• $f(x)$ is a probability density
• Normalization: $\int_0^{10} f(x)dx = 1$
• Non-negative: $f(x) \geq 0$
• Can have $f(x) > 1$ at specific points!

Bayesian vs. Frequentist Interpretations

Example illustrating the difference:

A single proposition can have different probabilities depending on available information

Justifications for the Bayesian Approach

Axiomatic approach

Derivation from axioms for manipulating reasonable expectations (Cox's theorem)

Dutch book approach

If probability rules aren't followed for gambling odds, one party can always win

Decision theory approach

Every admissible statistical procedure is either Bayesian or a limit of Bayesian procedures

2. Bayesian Inference

What is Inference?

Example: The Urn Problem

Setup

• An urn contains 11 balls: $N_r$ red and $11 - N_r$ blue
• We draw a ball, record its color, and replace it
• Repeat 3 times, obtaining: R, B, R

Question: What is $P(N_r | d_1=R, d_2=B, d_3=R)$?

Solution

First, calculate the likelihood:

Apply the product rule to get:

Where:

• $P(R,B,R | N_r)$ is the likelihood
• $P(N_r)$ is the prior probability
• $P(R,B,R)$ is a normalization constant

Posterior distribution for number of red balls given observations R,B,R

Bayes' Theorem

The urn example illustrates the general form of Bayes' theorem:

3. Prior Probabilities

What is a Prior?

Key points about priors:

• In principle, two rational people with the same information should specify the same prior
• In practice, this often doesn't happen due to different background knowledge
• Priors are necessary — you cannot do inference without assumptions
• Frequentist methods also use priors implicitly

Priors for Discrete Variables

• Finite support: Often use uniform distribution (principle of indifference)
• Count data: Poisson distribution may be appropriate
• Maximum entropy: Choose the distribution with maximum entropy subject to constraints

Poisson distributions for different rate parameters

Priors for Continuous Variables

For bounded continuous variables $a < x < b$:

• Uniform: $f(x) = 1/(b-a)$
• Beta: $f(x) \propto (x-a)^{\alpha-1}(b-x)^{\beta-1}$

For positive rate parameters $\lambda > 0$:

• Uniform: $f(\lambda) = c$ (improper!)
• Jeffreys: $f(\lambda) = 1/\lambda$ (uniform in log space)
• Log-normal: Proper alternative to Jeffreys prior

Improper Priors

A uniform prior on an unbounded domain cannot be normalized

Remember:

• One almost never knows absolutely nothing
• Upper and lower bounds can almost always be placed
• Log-normal can replace the improper $1/x$ prior

Log-normal distributions as proper alternatives to improper priors

Which Prior is Best?

Only the person doing the analysis can answer this! Priors encapsulate expert knowledge (or its absence). This is your opportunity to contribute your expertise to the analysis.

4. Summarizing Uncertainty

Credible Intervals

Bayesian credible intervals summarize uncertainty in parameter estimates:

95% credible interval [0.41, 0.91] contains 95% of the posterior probability

5. Inference in Practice

The Computational Challenge

Bayes' theorem has a troublesome denominator:

The evidence $P(D|M)$ requires integration:

Monte Carlo Methods

Named after the Monte Carlo casino — using randomness to solve deterministic problems

Rejection Sampling

One of the simplest Monte Carlo methods:

Green samples are accepted, red samples are rejected

The Curse of Dimensionality

As dimensions increase, the fraction of the envelope occupied by the target diminishes rapidly. For high-dimensional problems, rejection sampling becomes impractical.

Markov Chain Monte Carlo (MCMC)

The Metropolis-Hastings algorithm generates samples by creating a random walk:

MCMC explores the target distribution through a directed random walk

Key advantages:

• Walk explores mostly high-probability areas
• Does not require normalized target distribution
• Works in high dimensions

MCMC Output

Left: MCMC trace showing parameter values over iterations. Right: Resulting density estimate

Convergence and Mixing

MCMC challenges:

• Adjacent samples are correlated
• Initial state is arbitrary (burn-in needed)
• Must assess convergence and mixing

Burn-in period before chain converges to target distribution

Assessing Convergence

Compare multiple chains from different starting points:

Multiple chains should converge to the same distribution

Assessing Mixing

Use the autocorrelation function:

Autocorrelation shows how quickly samples become independent

Other Approaches

Monte Carlo Methods

• Gibbs sampling
• Hamiltonian Monte Carlo
• Particle filtering

Deterministic Methods

• Variational Bayes
• Laplace approximation

Summary

This lecture introduced Bayesian inference as the extension of logic to handle uncertainty:

1. Probability as logic: Cox's theorem shows probability theory uniquely extends deductive logic
2. Bayes' theorem: Simply the product rule rearranged, provides a coherent framework for updating beliefs
3. Priors: Necessary for any inference, encode background knowledge
4. Computational methods: MCMC makes Bayesian inference practical for complex problems
5. Advantages:
   • Direct probability statements about parameters
   • Coherent handling of uncertainty
   • Natural incorporation of prior knowledge