Hyper-parameter search with n_jobs=-1

With n_jobs=-1, a hyper-parameter search starts one worker per logical CPUWhat the operating system sees as one CPU. With SMT, each physical core shows up as two logical CPUs. it is allowed to run on. This dashboard checks how such a search scales with the CPUs it may use.

Each point restricts the process to some physical coresOne hardware core of the CPU. With SMT, it shows up as two logical CPUs., with their SMT siblingsThe other logical CPU of the same physical core., through its CPU affinityThe set of logical CPUs a process may run on. joblib only counts those for n_jobs=-1. (like taskset). The searches are those of the hyper-parameter search dashboard: 8 candidates on as many CV splitsTrain/test splits of the data used by cross-validation: each candidate is fitted on the train part and scored on the test part. More in the scikit-learn glossary as physical cores.

On the benchmarked cases, compared with one physical core:

• RandomForest and ExtraTrees scale the furthest: up to ~16x on the 24 cores of the desktop and ~55x to ~60x on the 172 cores of the Xeon, or ~70x to ~110x with scikit-learn-intelex.
• The exception is RandomForest on susy on the Xeon. It goes from ~54x at 86 cores to ~30x at 172 cores, except with scikit-learn-intelex.
• HistGradientBoosting reaches ~70x to ~75x on the Xeon, but peaks at ~39x at 86 cores on the small ames_housing dataset.
• The linear models depend more on the dataset on the Xeon: LogisticRegression reaches ~25x to ~35x, and Ridge goes from ~8x on susy to ~55x on year_prediction_msd. Ridge on ames_housing peaks at ~21x at 43 cores.
• On the laptop, the 16 cores give ~2x to ~7x.
• On the Xeon, the CPU affinity fills the NUMA nodesOn big servers, a group of CPU cores and the memory attached to them. A core reaches the memory of its own group faster than that of another group. in order, so the plots also reflect that placement from 22 to 86 cores. In a separate run on PyPIThe Python Package Index, where pip installs packages from. Its scikit-learn wheels bundle their own OpenMP runtime, and NumPy and SciPy bundle OpenBLAS. scikit-learn with the same 43 cores spread evenly over the NUMA nodes, LogisticRegression on susy and Ridge on year_prediction_msd are ~2.3x to ~2.4x faster and HistGradientBoosting on kddcup09_churn and year_prediction_msd is ~1.6x to ~1.8x faster, probably because memory bandwidth limits them (not measured). RandomForest on the small ames_housing dataset barely changes. NUMA nodes 0 and 1 share a bottleneck: 43 cores on nodes 0 and 2 are ~1.6x to ~1.8x faster than on nodes 0 and 1, and crossing sockets costs nothing.