I’ve read a great paper by Delgado et al. namely “Do we Need Hundreds of Classifiers to Solve Real World Classication Problems?” in which they compare 179 different classifiers from 17 families on 121 data sets composed by the whole UCI data base and some real-world problems. Classifiers are from R with and without caret pack, C and Matlab (I wish I could see Sklearn as well).

I really recommend you to read the paper in detail but I will share some of the highlights here. The most impressive result is the performance of Random Forests (RF) Implementations. For each dataset, RF is always at the top places. It gets 94.1% of max accuracy and goes by 90% in the 84.3% of the data sets. Also, 3 out of 5 best classifiers are RF for any data set. This is pretty impressive, I guess. The runner-up is SVM with Gaussian kernel implemented in LibSVM and it archives 92.3% max accuracy. The paper points RF, SVM with Gaussian and Polynomial kernels, Extreme Learning Machines with Gaussian kernel, C5.0 and avNNet (a committe of MLPs implemented in R with caret package) as the top list algorithms after their experiments.

One shortcoming of the paper, from my beloved NN perspective, is used Neural Network models are not very up-to-date versions such as drop-out, max-out networks. Therefore, it is hard to evaluate algorithms against these advance NN models. However, for anyone in the darn dark of algorithms, it is a quite good guideline that shows the power of RF and SVM against the others.