Bias and Variance – the struggle of daily life
Bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model. Variance refers to the amount by which the prediction of model would change if we estimated it using a different training data set. The challenge lies in finding a method for which both the variance and the bias are low, which is desirable for a good model. Increasing the model complexity decreases bias, however, when increased beyond a certain amount it starts increasing variance.
Similarly, replacing complex models having high variance and low bias with simplified models decreases variance, however, when decreased beyond a certain amount it starts increasing bias. It’s necessary to have relatively simple yet complex models to have a balanced bias-variance trade-off relationship. It’s amazing that we can relate almost every concept of machine learning to our daily lives. For example, think of bias as setting expectations in life. Basing your expectations on weak assumptions (over-simplified models) never ends up in getting the desired outcome, leading to high bias.
Similarly, think of variance as overthinking. Thinking about all endless situations and possibilities way in advance (overfitting on training data) to solidify our assumptions still doesn’t always end up in getting the desired outcome, leading to high variance. The key to making rationale decisions which end up getting you the desired outcome in most situations requires setting expectations on the basis of well thought of relatively solid assumptions, similar to how a good model should have low bias and low variance. Of course, there exists a trade-off and with experience (training) one is able to make better decisions (predictions) in real life. Well, the idea of desired outcomes is subjective, where we consider happiness, sorrow and other altered states of mind as our evaluation metrics.
By: Rishi Khetan