RandomizedSearchCV outer-parallelism scalability

The outer n_jobsParameter setting how many jobs scikit-learn runs in parallel with joblib. -1 means one per CPU. More in the scikit-learn glossary uses joblibThe library scikit-learn uses to run tasks in parallel, in worker processes or threads. More in the scikit-learn glossary's default backend (lokyjoblib's default backend. It runs tasks in separate worker processes that are reused across calls.), which fits candidates in separate worker processesSeparate Python processes running tasks in parallel. Unlike threads, they don't share memory.. Inner parallelism depends on the estimator:

• HistGradientBoosting uses OpenMP threadsStandard for running the loops of compiled code on several threads. scikit-learn's Cython code (HistGradientBoosting, KMeans, ...) uses it., Ridge and LogisticRegression use BLASBasic Linear Algebra Subprograms: the library that runs the matrix and vector products of NumPy and SciPy. OpenBLAS and MKL are two implementations of it. threads. joblib limits both to n_logical_cpus // n_jobs threads in each worker process. With SMTSimultaneous multithreading, called hyper-threading by Intel: one physical core runs two threads at once by sharing its execution units., that is 2 threads per physical coreOne hardware core of the CPU. With SMT, it shows up as two logical CPUs.: on the Xeon server (172 cores, 344 logical CPUsWhat the operating system sees as one CPU. With SMT, each physical core shows up as two logical CPUs.), n_jobs=86 gives 4 threads per worker. The laptop has no SMT.
• RandomForest and ExtraTrees fit their trees with joblib's threading backendjoblib backend that runs tasks in threads of the same process. It only helps when the code releases the GIL, as tree building does. and n_jobs=-1. joblib does not limit this nested call, so each worker process starts one thread per core, which can lead to oversubscriptionMore busy threads than CPU cores. They compete for the cores and the operating system keeps switching between them, which slows everything down.. joblib#1825 would limit nested n_jobs=-1 calls like it does for OpenMP and BLAS.
• scikit-learn-intelex uses oneTBBoneAPI Threading Building Blocks: Intel's C++ library for parallel tasks. oneDAL uses it to run on several threads, with its own thread pool, separate from OpenMP and BLAS. More threads, which joblib does not limit either.

There is one tab per machine and one plot per (estimator, dataset) pair. The x-axis is the outer n_jobs (log scale), the y-axis is the mean time per .fit()Training a model on data. More in the scikit-learn glossary call over the search, with one line per environmentOne Pixi environment of this repository: a fixed set of package versions and builds.. The marker size is the CPU loadMean share of all the machine's CPUs that were busy during the run. 100% means all cores busy. (100% means all cores busy). The line shape tells how the two levels interact:

• going down as 1/n_jobs: a single fit has little inner parallelism, and the search scales with outer workers;
• flat: a single fit was likely already using all cores, so outer workers bring nothing;
• rising: oversubscription, or bad interactions between joblib and the inner threading layer (they don't coordinate).

In general, we expect more outer workers to run in parallel the sequential parts of each fit, so the search gets faster and the CPU load goes up.

These results match what the models core-count scalability dashboard shows. On the benchmarked cases:

• More outer workersSeparate Python processes running tasks in parallel. Unlike threads, they don't share memory. make the search faster with a higher CPU load in most cases, as expected. The exceptions are HistGradientBoosting and the linear models on the Xeon server: they get faster while the CPU load goes down. It's unclear why.
• Ridge and LogisticRegression: outer parallelism helps, but less than one could hope. These workloads are probably memory boundLimited by how fast data moves from memory rather than by computation, so more cores don't help much., so more cores don't help much.
• HistGradientBoosting: outer parallelism works well on small datasets or on a big machine, because HGB's own threads are counter-productive there (see the HistGradientBoosting thread-scalability breakdown).
• RandomForest and ExtraTrees: on susy, the curve is mostly flat, as expected since a single fit already uses all cores. On the smaller datasets with preprocessingSteps run before the model, such as encoding categorical columns and imputing missing values., the search still speeds up with outer workers and the CPU load goes up. OversubscriptionMore busy threads than CPU cores. They compete for the cores and the operating system keeps switching between them, which slows everything down. is mostly harmless on the laptop. On the Xeon server, RandomForestClassifier gets up to ~1.8x slower beyond ~22 outer workers.
• scikit-learn-intelex: oversubscription is harmless on the laptop. On the Xeon server, the search gets up to ~2x slower beyond ~22 outer workers, for the trees and for Ridge.