Comparing ensemble methods
As mentioned, the ensemble method combines multiple base models into one stronger model. The core principal idea is that models rarely make the same mistake in the same cases when they look at the data from different angles, and aggregating their output will minimize random error.
When we talk about the model errors, conceptually, it happens because of two things:
- High bias model because the models are too simple (underfitting)
- High variance model because the model generalized too well (overfitting)
Bias is the systematic (average) error caused by restrictive or mismatched modelling assumptions, while variance is how much the fitted model would change across different data samples.
In many machine learning problems, there is a bias-variance trade-off. Greater model capacity to detect patterns tends to reduce bias while increasing variance. In contrast, regularization reduces variance at the expense of higher...
