Hyper-parameter search outer-parallelism scalability

The outer n_jobsParameter setting how many jobs scikit-learn runs in parallel with joblib. -1 means one per CPU. More in the scikit-learn glossary uses joblibThe library scikit-learn uses to run tasks in parallel, in worker processes or threads. More in the scikit-learn glossary's default backend (lokyjoblib's default backend. It runs tasks in separate worker processes that are reused across calls.), which fits candidates in separate worker processesSeparate Python processes running tasks in parallel. Unlike threads, they don't share memory.. Inner parallelism depends on the estimator:

• HistGradientBoosting uses OpenMP threadsStandard for running the loops of compiled code on several threads. scikit-learn's Cython code (HistGradientBoosting, KMeans, ...) uses it., Ridge and LogisticRegression use BLASBasic Linear Algebra Subprograms: the library that runs the matrix and vector products of NumPy and SciPy. OpenBLAS and MKL are two implementations of it. threads. joblib limits both to n_logical_cpus // n_jobs threads in each worker process. With SMTSimultaneous multithreading, called hyper-threading by Intel: one physical core runs two threads at once by sharing its execution units., that is 2 threads per physical coreOne hardware core of the CPU. With SMT, it shows up as two logical CPUs.: on the Xeon server (172 cores, 344 logical CPUsWhat the operating system sees as one CPU. With SMT, each physical core shows up as two logical CPUs.), n_jobs=86 gives 4 threads per worker. The laptop has no SMT.
• RandomForest and ExtraTrees fit their trees with joblib's threading backendjoblib backend that runs tasks in threads of the same process. It only helps when the code releases the GIL, as tree building does. and n_jobs=-1. joblib does not limit this nested call, so each worker process starts one thread per core, which can lead to oversubscriptionMore busy threads than CPU cores. They compete for the cores and the operating system keeps switching between them, which slows everything down.. joblib#1825 would limit nested n_jobs=-1 calls like it does for OpenMP and BLAS.
• scikit-learn-intelex uses oneTBBoneAPI Threading Building Blocks: Intel's C++ library for parallel tasks. oneDAL uses it to run on several threads, with its own thread pool, separate from OpenMP and BLAS. More threads. It reads the OpenMP limit set by joblib and applies it to its own threads, so they don't oversubscribe the machine either.

There is one tab per machine and one plot per (estimator, dataset) pair. The x-axis is the outer n_jobs (log scale), the y-axis is the wall time of the search divided by its number of .fit()Training a model on data. More in the scikit-learn glossary calls, with one line per environmentOne Pixi environment of this repository: a fixed set of package versions and builds.. Since each fit is the same everywhere, the y-axis also compares machines. The marker size is the CPU loadMean share of all the machine's CPUs that were busy during the run. 100% means all cores busy. (100% means all cores busy). The line shape tells how the two levels interact:

• going down as 1/n_jobs: a single fit has little inner parallelism, and the search scales with outer workers;
• flat: a single fit was likely already using all cores, so outer workers bring nothing;
• rising: oversubscription, or bad interactions between joblib and the inner threading layer (they don't coordinate).