ValueError: BuilderConfig BuilderConfig(name='alpaca_eval_gpt4_baseline', version=1.0.0, data_dir=None, data_files=None, description='Official AlpacaEval 2.0 evaluation set.') doesn't have a 'trust_remote_code' key.

[ojasraundale]

Hi,

I was trying to run the alpaca_eval on a local model but running the following command throws this error:

alpaca_eval evaluate_from_model <path_to_local_model>:

Traceback (most recent call last):
File "/.venv_vllm/bin/alpaca_eval", line 8, in
sys.exit(main())
File "/.venv_vllm/lib/python3.10/site-packages/alpaca_eval/main.py", line 608, in main
fire.Fire(ALL_FUNCTIONS)
File "/.venv_vllm/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/.venv_vllm/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/.venv_vllm/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/.venv_vllm/lib/python3.10/site-packages/alpaca_eval/main.py", line 280, in evaluate_from_model
df_dataset = utils.load_or_convert_to_dataframe(evaluation_dataset)
File "/.venv_vllm/lib/python3.10/site-packages/alpaca_eval/utils.py", line 245, in load_or_convert_to_dataframe
df = df(**kwargs)
File "/.venv_vllm/lib/python3.10/site-packages/alpaca_eval/constants.py", line 81, in get_alpaca_eval_data
dataset = datasets.load_dataset(
File "/.venv_vllm/lib/python3.10/site-packages/datasets/load.py", line 2129, in load_dataset
builder_instance = load_dataset_builder(
File "/.venv_vllm/lib/python3.10/site-packages/datasets/load.py", line 1852, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
File "/.venv_vllm/lib/python3.10/site-packages/datasets/builder.py", line 373, in init
self.config, self.config_id = self._create_builder_config(
File "/.venv_vllm/lib/python3.10/site-packages/datasets/builder.py", line 560, in _create_builder_config
raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
ValueError: BuilderConfig BuilderConfig(name='alpaca_eval_gpt4_baseline', version=1.0.0, data_dir=None, data_files=None, description='Official AlpacaEval 2.0 evaluation set.') doesn't have a 'trust_remote_code' key.

Removing the parameter "trust_remote_code=True" of get_alpaca_eval_data() from alpaca_eval/constants.py solves the issue temporarily. i.e

def get_alpaca_eval_data(dataset="alpaca_eval_gpt4_baseline"):
    dataset = datasets.load_dataset(
        "tatsu-lab/alpaca_eval",
        dataset,
        cache_dir=DEFAULT_CACHE_DIR,
        token=DATASETS_TOKEN,
        download_mode="force_redownload" if DATASETS_FORCE_DOWNLOAD else None,
        ## Commented the following line.
        # trust_remote_code=True,
    )["eval"]
    return dataset

This is my first time reporting an issue so please let me know if any further details are required or if any formatting needs to be changed.

Thank you.