FileExistsError: [WinError 183] Cannot create a file when that file already exists 'multi_runs'

[amylee-lixinyi]

Hi
When running parallel EnergyPlus simulation using runIDFs referencing the official document , the abovementioned error message popped up almost every time.
After some trial and error, runIDFs seems to create a folder named 'multi_runs' at the current python file's directory when called, thus a rerun of the same code will end up with error.
Is there a way to solve this apart from manually delate the 'multi_runs' folder every time before parallel processing?
Thank you!

[santoshphilip]

This looks like a bug. I need to reproduce it and see how to fix it. Not going to be easy :-(

[amylee-lixinyi]

The error message pointed to os.mkdir("multi_runs") in

def runIDFs(jobs, processors=1, debug=False):

in run_functions.py

Wish it helps with the troubleshooting

[santoshphilip]

Can you give me the following:

• A sample of your code that broke things
• Version of eppy
• Version of Energyplus
• Operating system (Mac, Linux, Windows)

[amylee-lixinyi]

Of course
Version of eppy: 0.5.63
Version of Energyplus: EnergyPlus 23-2-0
Operating system: Windows
The related code is as following:

import os
import eppy
from eppy.modeleditor import IDF
from eppy.runner.run_functions import runIDFs

def make_eplaunch_options(idf):
    """Make options for run, so that it runs like EPLaunch on Windows"""
    idfversion = idf.idfobjects['version'][0].Version_Identifier.split('.')
    idfversion.extend([0] * (3 - len(idfversion)))
    idfversionstr = '-'.join([str(item) for item in idfversion])
    fname = idf.idfname
    options = {
        'ep_version':idfversionstr, # runIDFs needs the version number
        'output_prefix':os.path.basename(fname).split('.')[0],
        'output_suffix':'C',
        'output_directory':'F:/test/',
        'readvars':True,
        'expandobjects':True
        }
    return options


def main():
    fnames=[]
    IDF.setiddname('C:/EnergyPlusV23-2-0/Energy+.idd') 
    for Building_ID in range(300):
        epw_path = list(pathlib.Path('C:/weather file').glob('*.epw'))[int(Weather[Building_ID])]
        scenario_name.append(epw_path)
        fnames.append("F:/IDF/%s.idf" %Building_ID)
    idfs = (IDF(fnames[BN], scenario_name[BN]) for BN in range(300))
    runs = ((idf, make_eplaunch_options(idf) ) for idf in idfs)
    num_CPUs = 4 
    runIDFs(runs, num_CPUs)

if __name__ == '__main__':
    main()

[santoshphilip]

thanx.

[santoshphilip]

tested sample code in Running in parallel processes using Generators on a Mac.
Running eppy from master branch
Could not recreate issue.

Next Steps

• review and test sample code from @amylee-lixinyi
• test on Windows Machine (maybe windows specific problem)
• test eppy version 0.5.63 (this is an unlikely possibility)

[santoshphilip]

steps in previous comment done on a windows machine. Could not recreate issue.
hmmmm ... what is going on ?

[santoshphilip]

@amylee-lixinyi So far I am unable to recreate the problem. Can you run the following code and post the output. (If you make any changes in the code, post the updated code too)

import os
import eppy
from eppy.modeleditor import IDF
from eppy.runner.run_functions import runIDFs

def make_eplaunch_options(idf):
    """Make options for run, so that it runs like EPLaunch on Windows"""
    idfversion = idf.idfobjects['version'][0].Version_Identifier.split('.')
    idfversion.extend([0] * (3 - len(idfversion)))
    idfversionstr = '-'.join([str(item) for item in idfversion])
    fname = idf.idfname
    options = {
        'ep_version':idfversionstr, # runIDFs needs the version number
        'output_prefix':os.path.basename(fname).split('.')[0],
        'output_suffix':'C',
        'output_directory':'F:/test/',
        'readvars':True,
        'expandobjects':True
        }
    return options


def main():
    fnames=[]
    scenario_name = []
    IDF.setiddname('C:/EnergyPlusV23-2-0/Energy+.idd') 
    for Building_ID in range(300):
        epw_path = list(pathlib.Path('C:/weather file').glob('*.epw'))[int(Weather[Building_ID])]
        scenario_name.append(epw_path)
        fnames.append("F:/IDF/%s.idf" %Building_ID)
    idfs = (IDF(fnames[BN], scenario_name[BN]) for BN in range(300))
    runs = ((idf, make_eplaunch_options(idf) ) for idf in idfs)
    for i, idf in enumerate(idfs):
        print(f"{i}, {idf.idfname}, {idf.epw}")
    # num_CPUs = 4
    # runIDFs(runs, num_CPUs)

if __name__ == '__main__':
    main()

[santoshphilip]

The above post is basically your code with a print statement (runs are not done)

[amylee-lixinyi]

The print statement printed something like this, copied and pasted only the first few lines

0, F:/IDF/0.idf, C:\weather file\weather_0.epw
1, F:/IDF/1.idf, C:\weather file\weather_1.epw
2, F:/IDF/2.idf, C:\weather file\weather_2.epw
2, F:/IDF/3.idf, C:\weather file\weather_3.epw

There is no error message or whatever, but I tried to run runIDFs once more without manually delate the 'multi_runs' folder, the same error still appears.

[santoshphilip]

I would like you to check on a couple of things in the print out:

• The index in the print out is going 0, 1, 2, 2 (It should be 3, 4 etc) - Is that a mistake in your copy and paste ?
• Can you paste some items from the end of the printout ?
• I am seeing backslashes "\" and front slashes "/" in the printout. I would expect only front slashes. Can you check on why that is happening ?

This may not be related to your problem - but still worth checking.

I have some thoughts on the next steps to debug. I'll post them soon. In the meantime, can you check on the above points ?

[amylee-lixinyi]

• The index in the last comment should be 0,1,2,3 instead of 0,1,2,2 sorry for the typo

• Here are a few lines from the end of print out

296, F:/IDF/296.idf, C:\weather file\weather_296.epw
297, F:/IDF/297.idf, C:\weather file\weather_297.epw
298, F:/IDF/298.idf, C:\weather file\weather_298.epw
299, F:/IDF/299.idf, C:\weather file\weather_299.epw

• Querying idf.epw returns WindowsPath('C:/weather file/weather_299.epw') with front slashes "/", print(idf.epw) returns C:\weather file\weather_299.epw. The change of the front slashes to backslashes happenes in print.

[santoshphilip]

OK. WindowsPath may be changing the slashes when it prints. Not worth perusing this. I don't think it matters to the issue at hand.

I post a comment on the steps that can be taken next.

[santoshphilip]

I am having hard time debugging this since I cannot recreate the problem. You may have to do the debugging. See the following documentation:

from eppy.runner.run_functions import runIDFs
print(runIDFs.__doc__)

Wrapper for run() to be used when running IDF5 runs in parallel.

    Parameters
    ----------
    jobs : iterable
        A list or generator made up of an IDF5 object and a kwargs dict
        (see `run_functions.run` for valid keywords).
    processors : int, optional
        Number of processors to run on (default: 1). If 0 is passed then
        the process will run on all CPUs, -1 means one less than all CPUs, etc.
    debug : bool, optional
        The runs are done in folders multi_runs/idf_0, multi_runs/idf_1 etc.
        if debug==False then multi_runs is deleted at the end of this function
        if debug==True multi_runs is not deleted

There is a debug argument where the default value is False. In your code debug=False by default. So it should have deleted the multi_runs. For some reason it is not doing so.

Can you put some print statements within the eppy code and try to discover why the deletion is not happening ?