Errors running the benchmark #608

ilhembekkr · 2024-02-21T09:40:28Z

ilhembekkr
Feb 21, 2024

Hello, I am trying to use the benchmark for my tests and I had generally Nan, I think I have problems during the execusion could you help ??

Even during the execution, I get some errors

PGijsbers · 2024-02-21T09:45:44Z

PGijsbers
Feb 21, 2024
Maintainer

Hi @ilhembekkr,

From the screenshots, we cannot tell what the issue is. Please have a look at #603, it contains workarounds for some existing issues and may help you with your problem. If, after trying this, you still run into issues, please try to provide a minimal reproducible example that reproduces the problems you experience. That would include a description of which version of the software you are using (both the AMLB version as well as the installed Python and dependencies) and the command used to start the benchmark. Please try to avoid using any custom files/changes as much as possible for this example. Without a MRE, it will be hard for us to help you.

2 replies

ilhembekkr Feb 21, 2024
Author

Hi @PGijsbers, Thank you for your answer,

I am using the last version of amlb as it is without any changes, with python3.8.5
I've installed the requirements and trying to test the benchmark, however I am getting empty results and some errors during execution

for exemple for: python runbenchmark.py AutoGluon
I am getting this output

Running benchmark `AutoGluon` on `test` framework in `local` mode.
Loading frameworks definitions from ['/home/ibekkar/workplace/automlbenchmark/resources/frameworks.yaml'].
Loading benchmark constraint definitions from ['/home/ibekkar/workplace/automlbenchmark/resources/constraints.yaml'].
Loading benchmark definitions from /home/ibekkar/workplace/automlbenchmark/resources/benchmarks/test.yaml.

---------------------------------------------
Starting job local.test.test.kc2.0.AutoGluon.
[MONITORING] [python [26768]] CPU Utilization: 7.0%
Assigning 2 cores (total=32) for new task kc2.
[MONITORING] [python [26768]] Memory Usage: 4.2%
Assigning 492166 MB (total=515633 MB) for new kc2 task.
[MONITORING] [python [26768]] Disk Usage: 89.3%
Running task kc2 on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'classification', 'name': 'kc2', 'openml_task_id': 3913, 'test_server': False, 'fold': 0, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 73162058, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492166, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/0/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'binary', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/0/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/0/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': '2.1.7',
  'auc': nan,
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 0,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/3913',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'logloss': nan,
  'metric': 'auc',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 73162058,
  'task': 'kc2',
  'training_duration': nan,
  'type': 'binary',
  'utc': '2024-02-21T10:06:08',
  'version': '1.0.0'}
Job `local.test.test.kc2.0.AutoGluon` executed in 1.184 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.

---------------------------------------------
Starting job local.test.test.kc2.1.AutoGluon.
Assigning 2 cores (total=32) for new task kc2.
Assigning 492164 MB (total=515633 MB) for new kc2 task.
Running task kc2 on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'classification', 'name': 'kc2', 'openml_task_id': 3913, 'test_server': False, 'fold': 1, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 73162059, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492164, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/1/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'binary', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/1/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/kc2/1/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': '2.1.7',
  'auc': nan,
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 1,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/3913',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'logloss': nan,
  'metric': 'auc',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 73162059,
  'task': 'kc2',
  'training_duration': nan,
  'type': 'binary',
  'utc': '2024-02-21T10:06:09',
  'version': '1.0.0'}
Job `local.test.test.kc2.1.AutoGluon` executed in 1.135 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.

----------------------------------------------
Starting job local.test.test.iris.0.AutoGluon.
Assigning 2 cores (total=32) for new task iris.
Assigning 492163 MB (total=515633 MB) for new iris task.
Running task iris on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'classification', 'name': 'iris', 'openml_task_id': 59, 'test_server': False, 'fold': 0, 'metric': 'logloss', 'metrics': ['logloss', 'acc', 'balacc'], 'seed': 73162058, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492163, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/0/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'multiclass', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/0/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/0/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': '2.1.7',
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 0,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/59',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'logloss': nan,
  'metric': 'neg_logloss',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 73162058,
  'task': 'iris',
  'training_duration': nan,
  'type': 'multiclass',
  'utc': '2024-02-21T10:06:10',
  'version': '1.0.0'}
Job `local.test.test.iris.0.AutoGluon` executed in 1.219 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.

----------------------------------------------
Starting job local.test.test.iris.1.AutoGluon.
Assigning 2 cores (total=32) for new task iris.
Assigning 492159 MB (total=515633 MB) for new iris task.
Running task iris on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'classification', 'name': 'iris', 'openml_task_id': 59, 'test_server': False, 'fold': 1, 'metric': 'logloss', 'metrics': ['logloss', 'acc', 'balacc'], 'seed': 73162059, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492159, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/1/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'multiclass', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/1/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/iris/1/metadata.json`.
Metric scores: { 'acc': nan,
  'app_version': '2.1.7',
  'balacc': nan,
  'constraint': 'test',
  'duration': nan,
  'fold': 1,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/59',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'logloss': nan,
  'metric': 'neg_logloss',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'result': nan,
  'seed': 73162059,
  'task': 'iris',
  'training_duration': nan,
  'type': 'multiclass',
  'utc': '2024-02-21T10:06:11',
  'version': '1.0.0'}
Job `local.test.test.iris.1.AutoGluon` executed in 1.178 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.

-----------------------------------------------------
Starting job local.test.test.cholesterol.0.AutoGluon.
Assigning 2 cores (total=32) for new task cholesterol.
Assigning 492155 MB (total=515633 MB) for new cholesterol task.
Running task cholesterol on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'regression', 'name': 'cholesterol', 'openml_task_id': 2295, 'test_server': False, 'fold': 0, 'metric': 'rmse', 'metrics': ['rmse', 'r2', 'mae'], 'seed': 73162058, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492155, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/0/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'regression', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/0/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/0/metadata.json`.
Metric scores: { 'app_version': '2.1.7',
  'constraint': 'test',
  'duration': nan,
  'fold': 0,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/2295',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'mae': nan,
  'metric': 'neg_rmse',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'r2': nan,
  'result': nan,
  'rmse': nan,
  'seed': 73162058,
  'task': 'cholesterol',
  'training_duration': nan,
  'type': 'regression',
  'utc': '2024-02-21T10:06:13',
  'version': '1.0.0'}
Job `local.test.test.cholesterol.0.AutoGluon` executed in 1.176 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.

-----------------------------------------------------
Starting job local.test.test.cholesterol.1.AutoGluon.
Assigning 2 cores (total=32) for new task cholesterol.
Assigning 492145 MB (total=515633 MB) for new cholesterol task.
Running task cholesterol on framework AutoGluon with config:
TaskConfig({'framework': 'AutoGluon', 'framework_params': {}, 'framework_version': '1.0.0', 'type': 'regression', 'name': 'cholesterol', 'openml_task_id': 2295, 'test_server': False, 'fold': 1, 'metric': 'rmse', 'metrics': ['rmse', 'r2', 'mae'], 'seed': 73162059, 'job_timeout_seconds': 120, 'max_runtime_seconds': 60, 'cores': 2, 'max_mem_size_mb': 492145, 'min_vol_size_mb': -1, 'input_dir': '/home/ibekkar/.cache/openml', 'output_dir': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606', 'output_predictions_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/1/predictions.csv', 'tag': None, 'command': 'runbenchmark.py AutoGluon', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'regression', 'output_metadata_file': '/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/1/metadata.json'})
Running cmd `/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py`
Traceback (most recent call last):

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py", line 23, in <module>

    from frameworks.shared.callee import call_run, result, output_subdir, \

  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/callee.py", line 107, in <module>

    def measure_inference_times(predict_fn: Callable[[DATA_INPUT], Any], files: list[Tuple[int, DATA_INPUT]]) -> dict[int, list[float]]:

TypeError: 'type' object is not subscriptable



Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/ibekkar/workplace/automlbenchmark/amlb/benchmark.py", line 578, in run
    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 16, in run
    return run_autogluon_tabular(dataset, config)
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/__init__.py", line 33, in run_autogluon_tabular
    return run_in_venv(__file__, "exec.py",
  File "/home/ibekkar/workplace/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv
    output, err = run_cmd(cmd, *args,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd
    raise e
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd
    completed = run_subprocess(str_cmd if params.shell else full_cmd,
  File "/home/ibekkar/workplace/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess
    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-zero exit status 1.
Loading metadata from `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/predictions/cholesterol/1/metadata.json`.
Metric scores: { 'app_version': '2.1.7',
  'constraint': 'test',
  'duration': nan,
  'fold': 1,
  'framework': 'AutoGluon',
  'id': 'openml.org/t/2295',
  'info': 'CalledProcessError: Command '
          "'/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python "
          '-W ignore '
          "/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' "
          'returned non-…',
  'mae': nan,
  'metric': 'neg_rmse',
  'mode': 'local',
  'models_count': nan,
  'params': '',
  'predict_duration': nan,
  'r2': nan,
  'result': nan,
  'rmse': nan,
  'seed': 73162059,
  'task': 'cholesterol',
  'training_duration': nan,
  'type': 'regression',
  'utc': '2024-02-21T10:06:14',
  'version': '1.0.0'}
Job `local.test.test.cholesterol.1.AutoGluon` executed in 1.140 seconds.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/autogluon.test.test.local.20240221T100606/scores/results.csv`.
Scores saved to `/home/ibekkar/workplace/automlbenchmark/results/results.csv`.
All jobs executed in 7.295 seconds.
[MONITORING] [python [26768]] CPU Utilization: 13.7%
[MONITORING] [python [26768]] Memory Usage: 4.2%
[MONITORING] [python [26768]] Disk Usage: 89.3%
Processing results for autogluon.test.test.local.20240221T100606
Summing up scores for current run:
               id        task  fold framework constraint      metric  duration     seed                                                                                                                                                                                                     info
openml.org/t/3913         kc2     0 AutoGluon       test         auc       1.2 73162058 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…
openml.org/t/3913         kc2     1 AutoGluon       test         auc       1.1 73162059 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…
  openml.org/t/59        iris     0 AutoGluon       test neg_logloss       1.2 73162058 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…
  openml.org/t/59        iris     1 AutoGluon       test neg_logloss       1.2 73162059 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…
openml.org/t/2295 cholesterol     0 AutoGluon       test    neg_rmse       1.2 73162058 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…
openml.org/t/2295 cholesterol     1 AutoGluon       test    neg_rmse       1.1 73162059 CalledProcessError: Command '/home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/venv/bin/python -W ignore /home/ibekkar/workplace/automlbenchmark/frameworks/AutoGluon/exec.py' returned non-…

PGijsbers Feb 21, 2024
Maintainer

The benchmark makes use of Python type hint syntax introduced in Python 3.9, and so is not compatible with Python 3.8. Please use Python 3.9.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Errors running the benchmark #608

{{title}}

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Errors running the benchmark #608

ilhembekkr Feb 21, 2024

Replies: 1 comment · 2 replies

PGijsbers Feb 21, 2024 Maintainer

ilhembekkr Feb 21, 2024 Author

PGijsbers Feb 21, 2024 Maintainer

ilhembekkr
Feb 21, 2024

Replies: 1 comment 2 replies

PGijsbers
Feb 21, 2024
Maintainer

ilhembekkr Feb 21, 2024
Author

PGijsbers Feb 21, 2024
Maintainer