AutoML Benchmark (AMLB)
AutoML frameworks compared on classification accuracy and ROC AUC across a shared set of OpenML tasks.
Results of the AutoML Benchmark (AMLB), which ran automated machine-learning frameworks on a shared set of classification tasks and published the evaluations on OpenML as study 226 (www.openml.org/s/226, 2019). Each subject is one framework; each run is one dataset, with the predictive accuracy and ROC AUC recorded there (0-1, higher is better).
Results
predictive_accuracy per subject over 6 measurements, ranked by average.
| Subject | Avg | Median | Min | Max | P95 | P99 | Count |
|---|---|---|---|---|---|---|---|
| h2oautoml | 0.7149 | 0.7149 | 0.7149 | 0.7149 | 0.7149 | 0.7149 | 1 |
| tunedrandomforest | 0.6922 | 0.6922 | 0.6922 | 0.6922 | 0.6922 | 0.6922 | 1 |
| autosklearn | 0.6871 | 0.6871 | 0.6871 | 0.6871 | 0.6871 | 0.6871 | 1 |
| randomforest | 0.6779 | 0.6779 | 0.6779 | 0.6779 | 0.6779 | 0.6779 | 1 |
| tpot | 0.6677 | 0.6677 | 0.6677 | 0.6677 | 0.6677 | 0.6677 | 1 |
| autoweka | 0.663 | 0.663 | 0.663 | 0.663 | 0.663 | 0.663 | 1 |
Metrics
- predictive_accuracy — recorded on each measurement. Fraction of test instances classified correctly, from 0 to 1. Higher is better
- area_under_roc_curve — recorded on each measurement. Area under the ROC curve, from 0 to 1. Higher is better; 0.5 corresponds to random guessing
Subjects (6)
- autosklearn
- autoweka
- h2oautoml
- randomforest
- tpot
- tunedrandomforest
Published by OpenML.
No observations in this window.