Concatenate fails when using --skip_nonexistent

[valeriupredoi]

Traceback (most recent call last):
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 752, in _run_task
    output_files = task.run()
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 250, in run
    self.output_files = self._run(input_files)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 473, in _run
    product.apply(step, self.debug)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 342, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 286, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 272, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/_io.py", line 195, in concatenate
    _fix_aux_factories(result)
UnboundLocalError: local variable 'result' referenced before assignment
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/users/valeriu/esmvalcore/esmvalcore/_main.py", line 433, in run
    fire.Fire(ESMValTool())
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/users/valeriu/esmvalcore/esmvalcore/_main.py", line 410, in run
    process_recipe(recipe_file=recipe, config_user=cfg)
  File "/home/users/valeriu/esmvalcore/esmvalcore/_main.py", line 104, in process_recipe
    recipe.run()
  File "/home/users/valeriu/esmvalcore/esmvalcore/_recipe.py", line 1366, in run
    self.tasks.run(max_parallel_tasks=self._cfg['max_parallel_tasks'])
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 674, in run
    self._run_parallel(max_parallel_tasks)
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 717, in _run_parallel
    _copy_results(task, running[task])
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 740, in _copy_results
    task.output_files, updated_products = future.get()
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/users/valeriu/anaconda3/envs/release220/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 752, in _run_task
    output_files = task.run()
  File "/home/users/valeriu/esmvalcore/esmvalcore/_task.py", line 250, in run
    self.output_files = self._run(input_files)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 473, in _run
    product.apply(step, self.debug)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 342, in apply
    self.cubes = preprocess(self.cubes, step, **self.settings[step])
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 286, in preprocess
    result.append(_run_preproc_function(function, items, settings))
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/__init__.py", line 272, in _run_preproc_function
    return function(items, **kwargs)
  File "/home/users/valeriu/esmvalcore/esmvalcore/preprocessor/_io.py", line 195, in concatenate
    _fix_aux_factories(result)
UnboundLocalError: local variable 'result' referenced before assignment
2021-03-10 12:56:37,266 UTC [27481] INFO    
If you have a question or need help, please start a new discussion on https://github.com/ESMValGroup/ESMValTool/discussions
If you suspect this is a bug, please open an issue on https://github.com/ESMValGroup/ESMValTool/issues
To make it easier to find out what the problem is, please consider attaching the files run/recipe_*.yml and run/main_log_debug.txt from the output directory.

While it's trying to concatenate an empty list of cubes because there is no data - have we lost the interface between IO and recipe? Looking into this now...

[bouweandela]

This looks very similar to #988, #917 looks like a likely place where this bug could have been introduced.

[valeriupredoi]

yes, cheers @bouweandela - was pressed on time when I opened this issue and dint get to dig around but I knew there was one issue that we've seen this before - the easy fix is to exit concatenation with empty list return if there are no cubes to concatenate but @jvegasbsc said he gonna fix it proper - ping ping Javi 😁

[bouweandela]

The proper fix would be to create preprocessor tasks containing only variables/datasets where there is sufficient data available when parsing the recipe. I guess some of the more exotic cases were missed when making the error message for missing data nicer. The --skip-nonexistent feature wasn't that well implemented to begin with (see #936). Some diagnostics scripts are really not able to handle this.