Software/implementations comparison

Pick a machine tab. The top of the tab describes the machine and each software environmentOne Pixi environment of this repository: a fixed set of package versions and builds. (versions, BLASBasic Linear Algebra Subprograms: the library that runs the matrix and vector products of NumPy and SciPy. OpenBLAS and MKL are two implementations of it. and OpenMPStandard for running the loops of compiled code on several threads. scikit-learn's Cython code (HistGradientBoosting, KMeans, ...) uses it. libraries). The baselineThe reference that speed-ups are computed against. defaults to stock scikit-learn from PyPIThe Python Package Index, where pip installs packages from. Its scikit-learn wheels bundle their own OpenMP runtime, and NumPy and SciPy bundle OpenBLAS.. To change it, open another software tab and click "Pick as baseline". The conda-forgeCommunity-maintained channel of conda packages. Its scikit-learn can be installed with different BLAS and OpenMP libraries. MKLIntel's Math Kernel Library (oneMKL): a BLAS implementation tuned for Intel CPUs, available as a conda-forge build option. build (sklearn-cf-mkl) can be picked as a baseline too, but isn't plotted otherwise.

The plots are arranged in a grid: one row per estimator category (linear, tree-based, clustering) and one column for fitTraining a model on data. More in the scikit-learn glossary and one for predictMaking predictions with a fitted model. More in the scikit-learn glossary. In each plot:

• the x-axis lists the estimators, with one color per implementation. An implementation keeps its color on every machine and for every baseline;
• each point is one benchmark case (dataset and hyperparametersParameters set before fit that control how an estimator learns, such as max_depth or the regularization strength. More in the scikit-learn glossary);
• the y-axis is the speed-up over the baseline, in log scale. Points above the dashed 1x line are faster than the baseline.

The marker shape flags cases to double check:

• circle (●): metricsScores of the fitted model on held-out data (ROC AUC, R2 score). They check that the compared implementations learn equivalent models. More in the scikit-learn glossary and setup match the baseline;
• square (■): metrics match but the setup differs, for instance a different solverThe optimization algorithm used by fit, for instance lbfgs or newton-cholesky for LogisticRegression. or number of iterations, scikit-learn-intelex using histogram-based splitsSplits chosen among bin edges instead of among every distinct feature value. Faster, and usually as accurate., or cuML Random Forest using quantile-based splitsSplit candidates chosen from a fixed number of quantiles of each feature, instead of among every distinct value. cuML Random Forest does this, with n_bins=256 in these benchmarks (128 by default)., where scikit-learn uses exact onesSplits chosen among every distinct value of the feature.;
• open diamond (◇): the metrics differ;
• grey point (●): scikit-learn-intelex fell backscikit-learn-intelex ran the stock scikit-learn code, because it doesn't support this case's parameters or data. to scikit-learn;
• cross (✕): failed run.

The notes under each plot count these cases.

Hover a point to see its case. Click it to find its row in "Detailed results", the table below each row, which has the exact timings and metrics and can be filtered by column.

On the benchmarked cases:

• sklearnex-cpu is the most consistently fast option. Linear models gain less than tree-based models and KMeans.
• sklearnex tree-based models are fast, but don't behave exactly like scikit-learn:
  • By default, sklearnex random forests and extra trees bin featuresMapping each feature to a small number of integer bins (a few hundred at most) before building trees. (max_bins=256Maximum number of bins per feature for histogram-based splits.), which scikit-learn doesn't implement yet. Binning roughly doubles or triples the fit speed-up. Cases run with exact splitsSplits chosen among every distinct value of the feature. (max_bins=n_samples) are on the left of each implementation in the tree-based plots.
  • Two oneDALoneAPI Data Analytics Library: Intel's C++ library that implements the algorithms of scikit-learn-intelex. bugs make some fits worse: oneDAL#3648 (ExtraTreesRegressor) and oneDAL#3771 (max_leaf_nodesTree parameter capping the number of leaves. With it, trees grow best-first: the most promising node is split first.).
• sklearnex KMeans fits ~3x faster and is never slower than scikit-learn.
• Array APIStandard API shared by array libraries (NumPy, PyTorch, CuPy, dpnp). With it, some scikit-learn estimators run directly on these libraries' arrays, including on GPU. backends are mixed. Only LogisticRegression and Ridge are benchmarked for now:
  • LogisticRegression on GPU is fairly fast.
  • Ridge uses the SVDSingular value decomposition: a stable but costly way to solve Ridge's least squares problem. solver instead of CholeskyMatrix factorization used to solve Ridge's normal equations. Faster than SVD. under Array API, which makes it slower (scikit-learn#35060).
  • PyTorchDeep learning library. scikit-learn can take its tensors as Array API inputs, on CPU or GPU. More on CPU parallelizes every operation, including small vector operations. That's why LogisticRegression is slower than scikit-learn with NumPy.
• cuML LogisticRegression always runs the quasi-Newton solverThe optimization algorithm used by fit, for instance lbfgs or newton-cholesky for LogisticRegression. qn (L-BFGS or OWL-QN). The point label can still say lbfgs, newton-cg, or newton-cholesky: the wrapper maps every one of those names onto qn. On the cuML side, the three synthetic solver cases time the same algorithm. scikit-learn runs the named solver.