How can you prevent overfitting?

Okay, now let’s say you found that your model overfits. But what to do now to prevent your model from overfitting? Fortunately, there are many ways you can try to prevent your model from overfitting. Below I have described a few of the most widely used solutions for overfitting.

1. Reduce the network size

The simplest way to prevent overfitting is to reduce the size of the model: the number of learnable parameters in the model (which is determined by the number of layers and the number of units per layer).

2. Cross-Validation

In cross-validation, the initial training data is used as small train-test splits. Then, these splits are used to tune the model. The most popular form of cross-validation is K-fold cross-validation. K represents the number of folds. Here is a short video from Udacity which explains K-fold cross-validation very well.

3. Add weight regularization

Given two explanations for something, the explanation most likely to be correct is the simplest one — the one that makes fewer assumptions. This idea also applies to the models learned by neural networks: given some training data and a network architecture, multiple sets of weight values could explain the data. Simpler models are less likely to overfit than complex ones. A simple model in this context is a model where the distribution of parameter values has less entropy (or a model with fewer parameters). Thus a common way to mitigate overfitting is to put constraints on the complexity of a network by forcing its weights to take only small values, which makes the distribution of weight values more regular. This is called weight regularization, and it’s done by adding to the loss function of the network a cost associated with having large weights. This cost comes in two flavors: L1 regularization — The cost added is proportional to the absolute value of the weight coefficients . L2 regularization — The cost added is proportional to the square of the value of the weight coefficients . L2 regularization is also called weight decay in the context of neural networks. [1]

4. Removing irrelevant features

Improve the data by removing irrelevant features. A dataset may contain many features that do not contribute much to the prediction. Removing those less important features can improve accuracy and reduce overfitting. You can use the scikit-learn’s feature selection module for this pupose.

5. Adding dropout

Dropout, applied to a layer, consists of randomly dropping out(setting to zero) a number of output features of the layer during training. Let’s say a given layer would normally return a vector [0.2, 0.5, 1.3, 0.8, 1.1] for a given input sample during training. After applying dropout, this vector will have a few zero entries distributed at random: for example, [0, 0.5, 1.3, 0, 1.1].

6. Data Augmentation

The simplest way to reduce overfitting is to increase the size of the training data. Let’s consider we are dealing with images. In this case, there are a few ways of increasing the size of the training data — rotating the image, flipping, scaling, shifting, etc. This technique is known as data augmentation. This usually provides a big leap in improving the accuracy of the model.

链接地址：https://medium.com/@sksoumik/how-to-prevent-overfitting-in-machine-learning-models-803f23bd9b8