Models core-count scalability

Each tab is a machine and each plot an (estimator, dataset) pair, with one line per software environmentOne Pixi environment of this repository: a fixed set of package versions and builds.. The x-axis is in log scale. For tree models, the y-axis is in log scale too, and each environment has a grey dashed perfect-scaling lineThe time the fit would take if n times more cores made it n times faster. starting from its first point, so the gap to it is the lost efficiency. Look for curves that flatten or go up. On machines with hyper-threadingSimultaneous multithreading, called hyper-threading by Intel: one physical core runs two threads at once by sharing its execution units., the 0.5 point of the non-tree models is a single logical CPUWhat the operating system sees as one CPU. With SMT, each physical core shows up as two logical CPUs., without its SMT siblingThe other logical CPU of the same physical core., so the step from 0.5 to 1 shows what SMTSimultaneous multithreading, called hyper-threading by Intel: one physical core runs two threads at once by sharing its execution units. brings on one core. For RandomForest and ExtraTrees, a section below compares runs with and without SMT, one plot per environment.

On the benchmarked cases:

• RandomForest and ExtraTrees scale best, up to the 172 cores of the Xeon server. scikit-learn-intelex makes RandomForest several times faster, but doesn't scale better. On the laptop, the gains get smaller past 4 cores, probably because the other cores are slower E-coresEfficiency cores: smaller and slower cores that recent Intel laptop CPUs have next to their fast performance cores..
• Ridge, LogisticRegression and KMeans with scikit-learn stop scaling after a few cores, and can get slower with more cores on the Xeon server.
• scikit-learn-intelex scales these three models much further, but on the Xeon server, they get slower past 16 to 64 cores.
• With the PyPIThe Python Package Index, where pip installs packages from. Its scikit-learn wheels bundle their own OpenMP runtime, and NumPy and SciPy bundle OpenBLAS. build, LogisticRegression gets slower as soon as it runs on more than one logical CPUWhat the operating system sees as one CPU. With SMT, each physical core shows up as two logical CPUs.. The OpenBLASOpen-source BLAS implementation. The NumPy and SciPy wheels on PyPI each ship their own copy of it. shipped with scipy 1.18 wakes all its threadsUnits of execution that share the memory of their process. Each one can run on its own CPU core. for tiny operations in L-BFGS-BQuasi-Newton optimization algorithm, behind LogisticRegression's default lbfgs solver. L-BFGS-B is its variant with bounds.. This is fixed in scipy 2.0 (see scipy#26193), and should also be fixed in 1.18.2 (scipy#26199).
• On the laptop, scikit-learn-intelex LogisticRegression looks much faster from 8 cores on, but that's a oneDALoneAPI Data Analytics Library: Intel's C++ library that implements the algorithms of scikit-learn-intelex. bug: on 4 threads or more, L-BFGS stops too early and returns a worse model. See oneDAL#3820.