Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

For HELM testing, is an API, like togetherApiKey, necessary? #44

Open
DOG-wooooof opened this issue Aug 19, 2024 · 0 comments
Open

For HELM testing, is an API, like togetherApiKey, necessary? #44

DOG-wooooof opened this issue Aug 19, 2024 · 0 comments

Comments

@DOG-wooooof
Copy link

DOG-wooooof commented Aug 19, 2024

I followed the instructions in the section Evaluation on tasks from HELM framework(https://github.com/FMInference/H2O/blob/main/h2o_hf/README.md).

However, when I reached Step 3: Evaluate the performance of generated text (refer to helm/command/eval.sh), I encountered the following error message:

        Traceback (most recent call last):
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 170, in make_request_with_retry
    return client.make_request(request)
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/helm/proxy/clients/together_client.py", line 138, in make_request
    raise TogetherClientError("togetherApiKey not set in credentials.conf")
helm.proxy.clients.together_client.TogetherClientError: togetherApiKey not set in credentials.conf

        Request failed. Retrying (attempt #2) in 0 seconds... (See above for error details)
        
        Traceback (most recent call last):
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/retrying.py", line 251, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/helm/proxy/clients/auto_client.py", line 170, in make_request_with_retry
    return client.make_request(request)
  File "/root/miniconda3/envs/myKV4/lib/python3.8/site-packages/helm/proxy/clients/together_client.py", line 138, in make_request
    raise TogetherClientError("togetherApiKey not set in credentials.conf")
helm.proxy.clients.together_client.TogetherClientError: togetherApiKey not set in credentials.conf

        Request failed. Retrying (attempt #3) in 10 seconds... (See above for error details)
    ......

I've read the docs, it seems an API key is required. I’m confused, though—aren’t we supposed to be doing an offline evaluation? Why do we need to connect to certain websites?
please advise If there is any way to fix it, thx!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant