“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.” (Mitchell, 1997)
The goal of most learning tasks is to enable a model to infer a functional relationship between inputs and outputs relating to the task. Whether it be classifying images or learning to play chess, there is a model learning about a process through observations, typically in the form of (input, output) pairs. Most importantly, we want our model to perform well on data it’s never seen before, i.e. true data in the real world; classifying random images across the internet or playing good chess moves against various play styles.
To find such a model, we want to balance how well it performs on the data we have available (training data) and the complexity of the model. Models with greater complexity are typically more powerful and can fit more functionally complex data, but are much more sensitive to variance in the data and can perform terribly if overfitting has occurred. Simpler models often have the advantage of being more explainable (Occam’s razor) and are less likely to overfit on noise in the data, better modeling the underlying patterns.
Let’s take a look at how these principles can be generalized and used across a variety of model formulations via empirical and structural risk minimization.
Suppose we have a dataset of training examples
and our goal is to learn some functional relationship
where is our hypothesis function, mapping from the input space X to the output space . Here refers to the whole model function, whether it be classification or regression. represents a perfect “dream” function that always gives the correct output for any input.
Risk is a term heavily correlated with loss or cost. We define the risk associated with a hypothesis as the expected loss:
where is the joint probability distribution over the input and output space from which we assume our training examples are sampled i.i.d, and is the loss function. The goal of a learning algorithm is to find the hypothesis from the hypothesis space which minimizes risk . That is,
Note that risk here is often called true risk or test error.
Empirical Risk Minimization
We don’t know the joint probability over the data , so we cannot compute the risk of hypotheses directly. We can instead use empirical risk:
which is another term for training error. This is the best approximation of the true risk associated with with respect to our finite samples from .
Structural Risk Minimization
Strictly minimizing the training error without checks on model complexity often lead to extreme overfitting and thus poor performance on test data (i.e. large true risk). Structural risk minimization incorporates regularization into the formulation to help penalize higher model complexity, reducing the chance we overfit on training data. Structural risk is defined as follows:
where is a regularizer that penalizes large model complexity, is the vector of model parameters, and is a hyperparameter used to control the trade-off between training loss and regularization. Minimizing this structural risk is theoretically better than simply minimizing the empirical risk, leading to a lower true risk. Structural risk minimization is the generalized problem that most classical machine learning models attempt to solve, differing primarily by their choice of loss function and regularizer.