$p(\theta \vert X)$ :: Posterior Distribution

The posterior distribution is a density over the possible values of the parameter $\theta$ that take both the prior and the observed data into account. This distribution represents what we’ve learned from the data through our understanding of likely model parameters.

$p(X \vert \theta)$ :: Likelihood

This is the probability distribution of the data under the model with the given parameters $\theta$. This value/function is typically interpreted in one of two ways:

• As a function of the data, $X$: when $\theta$ is fixed, this is a parametrized conditional density over the possible values of the data. For any particular set of observations we can compute the probability of having observed those data under the (fixed) model parameters.
• As a function of the parameter, $\theta$: when the data $X$ are fixed, this is typically termed the likelihood function. This is a function of the parameter, $\theta$, allowing it to vary as a random variable. In this case the function does not represent a probability density (as we know changing the conditional value is not considered by the axioms of probability), but it is often useful for observing the likelihood of the observed data under different model parameters.

$p(\theta)$ :: Prior Distribution

The prior distribution is a distribution over the model parameters before any observations are made. This distribution encodes initial subjective beliefs about the parameters, and is one of the defining factors of Bayesian inference.

When the prior is approximately accurate, it can help a model remain useful even when there is a small amount of data available. In such a case the prior will be able to influence the posterior in a meaningful way. However, as more and more data become available, the prior becomes less and less meaningful. The data begin to reveal reality more clearly, and the prior has less influence on shaping the posterior.

Marginal Likelihood $p(X)$

This is the marginal likelihood, or the prior predictive distribution, of the data $X$. This naming convention is due to the fact that it’s value is derived from marginalizing over the parameter $\theta$:

$$p(X) = \int{p(X|\theta)p(\theta)d\theta}$$

In other words, it is the likelihood function averaged over the parameters with respect to their prior. Intuitively, for some fixed data this value represents the probability of the data having been generated by the model at hand. Similarly, from the prior predictive distribution perspective it represents what we would expect X to look like before actually having made any observations.

One interesting question that comes to mind here is how does the model impact this value? If the value does not focus on any particular parameters $\theta$, is this value the same under all models? The answer here is no. Often times (and has been above) the model context is implicit in the probabilistic statements being made. Making this dependence explicit, the marginal likelihood for a model M is

$$p(X | \mathcal{M}) = \int{p(X|\mathcal{M},\theta)p(\theta|\mathcal{M})d\theta}$$

The conditioning on the model identity here is important, as it describes the context from which our priors and likelihood come. Under different models, these distributions will be different, and the presence of this model identity in the posterior explicitly shows why this is the case.

Remarks

It’s important to keep the goal of Bayesian inference in mind when reasoning about all these components and their definitions. At the end of the day, Bayesian inference is a method of inference, inferring things about the way a particular process works and can be explained. As a result, the goal is to be able to describe a framework that allows one to use observations from the world to inform a model of how the world works (here the “world” can be any process). We typically phrase this learning scenario in terms of some model and its parameters. Loosely speaking, the model represents some structural relation between internal parameters and the real world.

[One might consider the “model” to incorporate global structure about a process and define its overall capacity, while tuning its parameters allows for local control. This global and local trade-off (i.e. how much control the high level model is willing to give to its parameters) can change across models, and is indicative of its capacity. Giving more influence to the parameters corresponds to higher model capacity and flexibility, as less structural assumptions are made by the high level model itself. However, the trade-off here is that the model can be quite difficult to train, as the parameter space is large (see learning theory for what the size of the parameter space says about generalization guarantees). For a model with less flexibility, stronger assumptions are made initially, and finding parameters that work well enough is a much easier task. So which model to choose then, a flexible one or one that’s easy to train? This is one of the core problems in machine learning, and more formally known as the bias variance trade-off. When you increase the capacity of a model you reduce the inherent structural bias present within the model, but you simultaneously increase the variance (or stochasticity of sorts) of the model. When you decrease the capacity of the model, you take on a less noisy view of the world where stronger assumptions are made, thus reducing the variance of the model. However, by making stronger assumptions you are inherently restricting yourself to fewer ways to think about the world, increasing the chance you are far off (i.e. the bias) from reality.]

When our prior is uniform, we are saying that the likelihood describes the posterior, or at least is directly proportional. In this case there is no prior information to use; any value of the parameters is equally likely.

$$\text{posterior} \propto \text{likelihood} \times \text{prior}$$

Model

It might be helpful to define a model. A model is a parametric family of probability distributions, any one of which could explain our data D.