-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cost tracking for openrouter and openai #31
Conversation
@pytest.mark.skipif(not OPENROUTER_API_KEY_AVAILABLE, reason="OpenRouter API key is not available") | ||
def test_get_pricing_openrouter(): | ||
pricing = tracking.get_pricing_openrouter() | ||
assert isinstance(pricing, dict) | ||
assert all(isinstance(v, dict) for v in pricing.values()) | ||
for model in OPENROUTER_MODELS: | ||
assert model in pricing | ||
assert isinstance(pricing[model], dict) | ||
assert all(isinstance(v, float) for v in pricing[model].values()) | ||
|
||
|
||
def test_get_pricing_openai(): | ||
pricing = tracking.get_pricing_openai() | ||
assert isinstance(pricing, dict) | ||
assert all("prompt" in pricing[model] and "completion" in pricing[model] for model in pricing) | ||
assert all(isinstance(pricing[model]["prompt"], float) for model in pricing) | ||
assert all(isinstance(pricing[model]["completion"], float) for model in pricing) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@recursix I was thinking of testing for specific values but i feel like it might be very inconsistent, do you have an opinion on that matter ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can't test the value, it would be too demanding. You can test that it's within a
reasonable range though.
@@ -65,7 +66,7 @@ def __init__( | |||
def obs_preprocessor(self, obs: dict) -> dict: | |||
return self._obs_preprocessor(obs) | |||
|
|||
@openai_monitored_agent | |||
@get_action_decorator |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe call it cost_tracker_decorator?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we put it as part of Agent base class? this way users won't have to add
the decorartor to track the cost
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@get_action_decorator | |
class Agent: | |
def _get_action(self,obs): | |
with cost_tracker(): | |
return self.get_action(obs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'm not sure about this, would we define a new Agent class on top of bgym.Agent ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ahhh crap, I didn't realized it was in browsergym. We could defined TrackedAgent, but hey let's leave it like that for now.
@pytest.mark.skipif(not OPENROUTER_API_KEY_AVAILABLE, reason="OpenRouter API key is not available") | ||
def test_get_pricing_openrouter(): | ||
pricing = tracking.get_pricing_openrouter() | ||
assert isinstance(pricing, dict) | ||
assert all(isinstance(v, dict) for v in pricing.values()) | ||
for model in OPENROUTER_MODELS: | ||
assert model in pricing | ||
assert isinstance(pricing[model], dict) | ||
assert all(isinstance(v, float) for v in pricing[model].values()) | ||
|
||
|
||
def test_get_pricing_openai(): | ||
pricing = tracking.get_pricing_openai() | ||
assert isinstance(pricing, dict) | ||
assert all("prompt" in pricing[model] and "completion" in pricing[model] for model in pricing) | ||
assert all(isinstance(pricing[model]["prompt"], float) for model in pricing) | ||
assert all(isinstance(pricing[model]["completion"], float) for model in pricing) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can't test the value, it would be too demanding. You can test that it's within a
reasonable range though.
|
||
from agentlab.llm.langchain_utils import _convert_messages_to_dict | ||
|
||
TRACKER = threading.local() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ohhhh does this make it thread safe?
Cost tracking for openrouter and openai