Probability mass function

The probability mass function (pmf) is a function that gives the probability of a discrete random variable taking on a particular value. The pmf is the primary way in which discrete probability distributions are characterized or defined. For a discrete random variable X

$$f_X : \mathbb{R} \mapsto [0,1], \;\;\; f_X(x) = P(x) = P(X=x) = P(\{\omega \in \Omega | X(\omega)=x\})$$

In line with the axioms of probability, it holds that

$$P(x) \ge 0 \;\; \text{and} \;\;\sum^{\infty}_{i=1}{P(x_i)} = 1$$

Cumulative distribution function

pass

Expected value properties

• Linearity of expectation

Expectation of a function of a random variable

For a random variable X and some real valued function g, we have that the expected value of g(X)

$$\mathbb{E}[g(X)] = \sum_{i=1}^{\infty}{g(x_i)p(x_i)}$$

We first transform the value of X using g, and then make use of the probability mass function of X to weight this new value. This shows, for example, that 𝔼[X²] is not necessarily equal to 𝔼[X]².

Variance value properties

• Covariance
• Density and Likelihood

Probability Density Function

The probability density function (PDF) of a random variable is a function that maps points in the random variable’s codomain to values that can be interpreted as the relative likelihood of observing that point. Note that the absolute probability of a continuous random variable taking on any particular value is zero; there are uncountably many possible outcomes, and all outcomes cannot be assigned any positive finite value (note that for a PMF with countably infinite possible outcomes, we can assign positive finite values to all outcomes so long as their infinite sum is one). But then how can the total probability sum up to one if the probability of all outcomes is zero? This is the typical contradicting dichotomy one encounters when attempting to think about PDFs.

It turns out we need to shift our focus to a different question. We can’t really ask about the probability of any particular value occurring, since that probability is zero (and isn’t very useful). We can instead think tractably by asking about the probability that the random variable takes on a value within a range of values, or the probability that the random variable is close to a particular value (dealing in sets of uncountable many objects). That is, we can define a function f such that

$$P(X\in B) = \int_B{f_X(x)dx} \\ P(a < X < b) = \int_a^b{f_X(x)dx}$$

Here f is the probability density function of X. f is defined such that the integral between any two points gives the probability that X takes on a value from within that interval. This aligns with the notion that any particular value has no chance of occurring: as the interval shrinks around a single value, the integral on that interval goes to zero. That is,

$$P(X=a) = P(a < X < a) = \int_a^a{f_X(x)dx} = 0$$

We can also formulate the scenario where X takes on a value in an infinitesimally small range

$$P(X=a) \approx P(a < X < a+dx) = \int_a^{a+dx}{f_X(x)dx} = f_X(a)dx$$

which is like taking the area of a single one of the infinitesimally thin slices used to compute an integral.

f also must integrate to one when considering the entire sample space:

$$P(-\infty < X < \infty) = \int_{-\infty}^\infty{f_X(x)dx} = 1$$

In summary, the PDF is a function that, when integrated over an interval, gives the probability of a continuous random variable (having a density according to the PDF) of taking on a value in that interval. One can liken the notion of a PDF to something like a velocity curve: we can integrate the velocity curve between two points in time to find the total distance traveled between those two times. However, if we focus in on any particular point in time, we might have some notion of instantaneous velocity (i.e. the value of the velocity function at that point in time) but the integral is zero, as there’s no time for that velocity to act and thus no movement occurs. This is just like the issue with having a probability of zero for any particular outcome while the PDF returns a value (i.e. a relative likelihood), as well as only having absolute probabilities show up when considering an interval of outcomes. More on this confusing stuff here.

Conditional Probability Density Function

$$f_{X|Y}(x|y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} = \frac{P(X=x,Y=y)}{P(Y=y)}$$

$$f_Y(y) = \int{f_{X,Y}(x,y)dx}$$

Note here that y must be fixed before taking the integral. It’s not that the function itself is equal to t

Convolutions