Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit numpy thread usage for Transformation classes #2950

Merged
merged 33 commits into from
Apr 10, 2021
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
b8441ff
add threadlimit deco
Sep 19, 2020
604fa1b
add threadlimit deco
yuxuanzhuang Sep 19, 2020
0ed46f9
add dep
yuxuanzhuang Sep 19, 2020
b41a739
runtime change
yuxuanzhuang Sep 23, 2020
3941281
context mananger instead of decor
yuxuanzhuang Oct 13, 2020
326e105
working decor
yuxuanzhuang Oct 16, 2020
b4daa8d
create TransformationBase and thread limit
yuxuanzhuang Oct 16, 2020
d686f06
travis threadpool
yuxuanzhuang Oct 16, 2020
af6bab8
deco to context due to picklibility
yuxuanzhuang Oct 16, 2020
5f985e9
pep
yuxuanzhuang Oct 16, 2020
efbc3f8
docs
yuxuanzhuang Nov 4, 2020
ded5f84
Merge remote-tracking branch 'mda_origin/develop' into trans_single_t…
yuxuanzhuang Nov 4, 2020
e2cd996
change to kwargs
yuxuanzhuang Nov 4, 2020
f084d33
add test for transformation base
yuxuanzhuang Nov 4, 2020
67aeda2
changelog
yuxuanzhuang Nov 4, 2020
d363c0b
appveyor
yuxuanzhuang Nov 4, 2020
34d370e
travis threadpool for arm
yuxuanzhuang Nov 4, 2020
01570c4
remove deco threadlimit
yuxuanzhuang Nov 4, 2020
383b1f5
doc for transformation
yuxuanzhuang Nov 9, 2020
317e5e2
merge to dev
yuxuanzhuang Nov 9, 2020
0fb4d85
typo
yuxuanzhuang Nov 9, 2020
9365e5f
merge to develop
yuxuanzhuang Apr 6, 2021
353bfa9
make threadpoolctl a requirement
yuxuanzhuang Apr 7, 2021
fdd8f52
base documentation
yuxuanzhuang Apr 7, 2021
edb5ef5
doc for transformation fix
yuxuanzhuang Apr 7, 2021
7884210
box dimension rework
yuxuanzhuang Apr 7, 2021
636ae12
add threadpoolctl to azure
yuxuanzhuang Apr 7, 2021
1ea51df
add threadpoolctl to gh ci
yuxuanzhuang Apr 7, 2021
197deb9
merge to develop
yuxuanzhuang Apr 7, 2021
e2ca135
add threadpoolctl to ppc64le
yuxuanzhuang Apr 8, 2021
aba771f
cov notimplement error transformation
yuxuanzhuang Apr 8, 2021
890e2b3
add note for maxthread=1
yuxuanzhuang Apr 8, 2021
17d4cb4
changelog for threadpooltcl
yuxuanzhuang Apr 8, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ env:
- SETUP_CMD="${PYTEST_FLAGS}"
- BUILD_CMD="pip install -e package/ && (cd testsuite/ && python setup.py build)"
- CONDA_MIN_DEPENDENCIES="mmtf-python biopython networkx cython matplotlib scipy griddataformats hypothesis gsd codecov"
- CONDA_DEPENDENCIES="${CONDA_MIN_DEPENDENCIES} seaborn>=0.7.0 clustalw=2.1 netcdf4 scikit-learn joblib>=0.12 chemfiles tqdm>=4.43.0 tidynamics>=1.0.0 rdkit>=2020.03.1 h5py"
- CONDA_DEPENDENCIES="${CONDA_MIN_DEPENDENCIES} seaborn>=0.7.0 clustalw=2.1 netcdf4 scikit-learn joblib>=0.12 chemfiles tqdm>=4.43.0 tidynamics>=1.0.0 rdkit>=2020.03.1 h5py threadpoolctl"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

threadpoolctl isn't a core dependency, so as it is, it will fail when running on minimal dependencies.

- CONDA_CHANNELS='biobuilds conda-forge'
- CONDA_CHANNEL_PRIORITY=True
- PIP_DEPENDENCIES="duecredit parmed"
Expand Down
10 changes: 10 additions & 0 deletions package/MDAnalysis/lib/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,8 @@
import functools
from functools import wraps
import textwrap
from contextlib import ContextDecorator
from threadpoolctl import threadpool_limits

import mmtf
import numpy as np
Expand Down Expand Up @@ -2353,3 +2355,11 @@ def check_box(box):
if np.all(box[3:] == 90.):
return 'ortho', box[:3]
return 'tri_vecs', triclinic_vectors(box)


class threadpool_limits_decorator(threadpool_limits, ContextDecorator):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm probably missing something very obvious, is this being used anywhere anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, not really. It turns out to be not that useful in this case.

def __enter__(self):
return self

def __exit__(self, *exc):
return False
2 changes: 2 additions & 0 deletions package/MDAnalysis/transformations/fit.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@

from ..analysis import align
from ..lib.transformations import euler_from_matrix, euler_matrix
from ..lib.util import threadpool_limits_decorator


class fit_translation(object):
Expand Down Expand Up @@ -207,6 +208,7 @@ def __init__(self, ag, reference, plane=None, weights=None):
self.ref_com = self.ref.center(self.weights)
self.ref_coordinates = self.ref.atoms.positions - self.ref_com

@threadpool_limits_decorator(limits=1)
def __call__(self, ts):
mobile_com = self.mobile.atoms.center(self.weights)
mobile_coordinates = self.mobile.atoms.positions - mobile_com
Expand Down
2 changes: 2 additions & 0 deletions package/MDAnalysis/transformations/rotate.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@

from ..lib.transformations import rotation_matrix
from ..lib.util import get_weights
from ..lib.util import threadpool_limits_decorator


class rotateby(object):
Expand Down Expand Up @@ -162,6 +163,7 @@ def __init__(self,
else:
raise ValueError('A point or an AtomGroup must be specified')

@threadpool_limits_decorator(limits=1)
def __call__(self, ts):
if self.point is None:
position = self.center_method()
Expand Down