Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WPF Parallelization fails #687

Open
sezelt opened this issue Sep 18, 2024 · 0 comments
Open

WPF Parallelization fails #687

sezelt opened this issue Sep 18, 2024 · 0 comments

Comments

@sezelt
Copy link
Member

sezelt commented Sep 18, 2024

There have been reports of the mpire distributed WPF optimization failing with errors like

AttributeError: Can't pickle local object 'WholePatternFit._fit_distributed.<locals>.f

This appears to only affect Windows systems.

I think the fix is to enable dill serialization:

with WorkerPool(
n_jobs=num_jobs,
shared_objects=fit_opts,
) as pool:
results = pool.map(

should be modified to become:

        with WorkerPool(
            n_jobs=num_jobs,
            shared_objects=fit_opts,
            use_dill=True,
        ) as pool:

mpire produces threads differently on Windows vs UNIX and so there can be serialization errors that only show up one one platform when transmitting complicated objects to threads. Unfortunately I do not have a Windows machine to test this on at the moment, so someone else will have to try this and let us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant