Limiting or eliminating parallel processing in v1.5.1

[pavlovc2]

Hi, I am running a series of processes using v1.5.1 on a large machine where I want to only use one thread. It seems that vario_estimate uses all available processes, increasing processing time from ~0.005 seconds to 20-30 seconds. How can this multithreaded behavior be turned off? I have tried setting os.environ['RAYON_NUM_THREADS'] = '1' in several places but this does not seem to have the desired effect.

[MuellerSeb]

Hey there,

one option is to use the serial version of GSTools provided by PyPI (or self compiled), or you could try setting OMP_NUM_THREADS to 1 before running python as described here: https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html

We could think about adding num_threads to vario_estimate and forward this to prange used in the cython implementation.

RAYON_NUM_THREADS is only applicable if you use the gstools-core backend which is a rust implementation of the core algorithms.

Hope that helps.

[MuellerSeb]

Another option could be to use threadpoolctl:

import numpy as np
import gstools as gs

from threadpoolctl import threadpool_limits

x = np.random.RandomState(19970221).rand(1000) * 10000.
y = np.random.RandomState(20011012).rand(1000) * 10000.
model = gs.Exponential(dim=2, var=2, len_scale=8)
srf = gs.SRF(model, mean=0, seed=19970221)
field = srf((x, y))

with threadpool_limits(limits=1, user_api='openmp'):
    bin_center, gamma = gs.vario_estimate((x, y), field)

[pavlovc2]

Thank you for these answers. To add to the context here, in addition to vario_estimate, further investigation suggests that gstools.krige.Krige is also contributing to the slowdown. I'm running this within a larger parallel process with multiprocessing, so I would need a solution that allows parallel processing within the overall environment while restricting specifically the behavior of the gstools calls. I will look at setting openmp settings. Parameters to the key functions themselves, or a gstools-specific setting/environment, that control the behavior would definitely be helpful.

[LSchueler]

Hi Nathan,

I think a global setting in GSTools for the number of cores to use is a really good idea. I hope to find some time this year to implement such a setting. I'll keep you updated here.

[LSchueler]

Hi Nathan, you can have a look at PR #336. For now, you can set the number of parallel threads for the Cython code with the global variable gs.config.NUM_THREADS. I should be able to get back to the PR in the first week of January.

[LSchueler]

Hi again, PR #336 ist finally merged into the main branch and you can set the number of threads directly in GSTools. Can you work with the current version on main or do you need a released version?