Parallel execution of multiple IronPython engines with one runtime

[FSchu]

Description

Hi,
we found a changed behaviour in IP 3.4.1 compared to the IP 2.x implementation.
We are using IP to execute python code in multiple threads.
To improve performance the work threads are precreated and share the same (global) runtime.
After switching to IP 3 we found that addings an assembly via the clr methodes and also imports to e.g. the decimal module sometimes fail. This is different for what happend in IP 2.x, there was no issue, running using mutiple python enginens in parallel threads with one global runtime.

We have been able to mitigate the issue with the addition of a process wide lock object used in the methods responsible for imports and adding assembly references. However, we can't tell for sure that all potential methods are

Steps to Reproduce

1. Create a shared runtime
2. Create multipe thread having their private python engine
3. Ensure that the threads are started at the same time
4. Have the engine execute script code including clr.AddReference calls and "from decimal import Decimal"
5. Attached is a C# sample program (renamed to .cs.txt to allow upload) and a patch file with the changes we made to try to fix the issue
   Program.cs.txt
   mutliThreadedSuport.patch

Expected behavior:
All threads execute the script code without errors

Actual behavior:
Some threads fail to execute the script code with error like "module 'numbers' not in sys.modules" or failure to add the assembly via the clr method. The actual error can be different.

Version Information

• .NET platform used: .NET Framework
• Version 4.6.2
• Windows 11
• 3.4.1 DEBUG (3.4.1.1000) / [.NETFramework,Version=v4.6.2 on .NET Framework 4.8.9282.0 (64-bit)]

[slozier]

Thanks for the report. A few comments:

• In your sample program I'm seeing the same behaviour under IronPython 2.x or 3.x so I'm not sure how it used to work with 2.x? What version of IronPython were you using? Also, which version of the DLR?
• Runtime.GetEngine("Python") does not create per-thread engines. It'll look up the language and engine from dictionaries stores in the runtime and return those. You can output the value of Engine.GetHashCode() to see that your threads are all using the same engine. You can use Python.CreateEngine() to create unique engines. Note that Runtime.GetEngine("Python") doesn't appear to be thead safe as I'm seeing the occasional Failed to load language message in your sample.
• Because you're using the same engine it's trying to do concurrent imports. As far as I know, import has never been thread-safe (not in 2.7 or 3.4). See Import not thread safe ironpython2#661.
• How are you testing that clr.AddReference is failing? I haven't been able to reproduce that one.

[FSchu]

Thanks for the reply.
Actually the test program was only used to reproduce the issue in Py 3.x.
Regarding the clr import we found issues when using the pdfsharp / migradoc dlls via clr imports. Performing this in parallel failed with an error regarding the clr import.
I'll also check, if we can adjust the engine creation without a major performance hit.