In the last entry we saw basic sampling methods (over/under sampling). In this entry we try two another methods to overcome an imbalanced classification problem.
We use the same data as the last time. That is, the male passengers on titanic. We recall that the survival rate is about 20%.
Method 1) SMOTE
library(DMwR) set.seed(11) df.sm <- df.train %>% mutate(survived=as.factor(survived)) df.sm <- SMOTE(survived~., df.sm, perc.over=288,k=2) ### applying SMOTE algorithm
While the original training contains only 81 positive samples, the data set after SMOTE contains 243 positive samples. The positive rate has become around 43%. (I set
perc.over so that the size of the positive samples becomes 3 times.)
> table(df.train$survived) ### original training set n y 314 81 > table(df.sm$survived) ### after SMOTE n y 324 243
Note that the number of negative samples has slightly been increased.
The training result is following.
method acc.train acc.test y.rate recall precision F1.score 1 nobody 0.7949367 0.7938931 0.0000000 0.0000000 0.0000000 0.0000000 2 glmnet 0.7721519 0.7709924 0.2519084 0.5555556 0.4545455 0.5000000 3 svmRadial 0.7797468 0.7404580 0.3129771 0.6296296 0.4146341 0.5000000 4 rf 0.9265823 0.7328244 0.2977099 0.5740741 0.3974359 0.4696970 5 xgbTree 0.8911392 0.7328244 0.3206107 0.6296296 0.4047619 0.4927536
Comparing with other sampling methods which we used in the last entry, the F1 scores are slightly better. We note that the result largely depends on the choice of
Method 2) Ensemble method
One of the disadvantage of under sampling is that we just ignore many negative samples. That is, we do not use the ignored negative samples if we apply the under sampling method. To reduce the number of ignored samples, we may use multiple under-sampled training sets.
The logic is easy to understand. We generate $k$ training sets $X_1, \cdots, X_k$ by under sampling. We train models $\hat f_1, \cdots, \hat f_k$ on the training sets. Given a sample $x$, we take the sum $\hat f_1 (x) + \hat f_k (x)$. The larger the sum is, the more it is likely to be positive. To make an explicit prediction we have to choose a threshold $t$, i.e. if the sum $\geq t$, then the sample is predicted as a positive sample. (The implementation can be found on this repository. We do not show it be cause it is quite long to put on a blog entry.)
Following the logic, we can think that a largeer $k$ is better, but this is wrong. You can find the reason by looking at the result for the case when $k=3$:
The x-axis shows that the sum. Note that the result on the test set lies on the left hand. As you see in the histogram, the positive samples are distributed for all bin. Therefore we have to choose $t = k$.
There is another problem: the behaviors of the trained models are quite different. (For example the recall of glmnet is high, while one of random forest is relatively small.)
method yrate accuracy recall precision F1.score 1 glmnet 0.2748092 0.7786260 0.6296296 0.4722222 0.5396825 2 svmRadial 0.3396947 0.6984733 0.5925926 0.3595506 0.4475524 3 rf 0.2137405 0.7480916 0.4074074 0.3928571 0.4000000 4 xgbTree 0.2175573 0.7900763 0.5185185 0.4912281 0.5045045
For the case where $k=2$ is following:
method yrate accuracy recall precision F1.score 1 glmnet 0.2786260 0.7748092 0.6296296 0.4657534 0.5354331 2 svmRadial 0.3664122 0.7022901 0.6666667 0.3750000 0.4800000 3 rf 0.2938931 0.7213740 0.5370370 0.3766234 0.4427481 4 xgbTree 0.2938931 0.7595420 0.6296296 0.4415584 0.5190840
Note that the case $k=1$ is just an under sampling method we saw in the last entry. The following is just a copy from the previous entry.
method acc.train acc.test y.rate recall precision F1.score 1 nobody 0.7949367 0.7938931 0.0000000 0.0000000 0.0000000 0.0000000 2 glmnet 0.7367089 0.7251908 0.3282443 0.6296296 0.3953488 0.4857143 3 svmRadial 0.7113924 0.6793893 0.3969466 0.6851852 0.3557692 0.4683544 4 rf 0.7468354 0.6717557 0.3740458 0.6111111 0.3367347 0.4342105 5 xgbTree 0.7620253 0.7137405 0.3473282 0.6481481 0.3846154 0.4827586
So we may see the ensemble method is better than the only one under sampling. But to be honest it is not worth trying an ensemble model because it cannot defeat SMOTE (at least this time). (But the performance of elasticnet (glmnet) becomes slightly better.)