Cost tracking for openrouter and openai #31
Conversation
| @pytest.mark.skipif(not OPENROUTER_API_KEY_AVAILABLE, reason="OpenRouter API key is not available") | ||
| def test_get_pricing_openrouter(): | ||
| pricing = tracking.get_pricing_openrouter() | ||
| assert isinstance(pricing, dict) | ||
| assert all(isinstance(v, dict) for v in pricing.values()) | ||
| for model in OPENROUTER_MODELS: | ||
| assert model in pricing | ||
| assert isinstance(pricing[model], dict) | ||
| assert all(isinstance(v, float) for v in pricing[model].values()) | ||
|
|
||
|
|
||
| def test_get_pricing_openai(): | ||
| pricing = tracking.get_pricing_openai() | ||
| assert isinstance(pricing, dict) | ||
| assert all("prompt" in pricing[model] and "completion" in pricing[model] for model in pricing) | ||
| assert all(isinstance(pricing[model]["prompt"], float) for model in pricing) | ||
| assert all(isinstance(pricing[model]["completion"], float) for model in pricing) |
There was a problem hiding this comment.
@recursix I was thinking of testing for specific values but i feel like it might be very inconsistent, do you have an opinion on that matter ?
There was a problem hiding this comment.
can't test the value, it would be too demanding. You can test that it's within a
reasonable range though.
| @@ -65,7 +66,7 @@ def __init__( | |||
| def obs_preprocessor(self, obs: dict) -> dict: | |||
| return self._obs_preprocessor(obs) | |||
|
|
|||
| @openai_monitored_agent | |||
| @get_action_decorator | |||
There was a problem hiding this comment.
maybe call it cost_tracker_decorator?
There was a problem hiding this comment.
should we put it as part of Agent base class? this way users won't have to add
the decorartor to track the cost
There was a problem hiding this comment.
| @get_action_decorator | |
| class Agent: | |
| def _get_action(self,obs): | |
| with cost_tracker(): | |
| return self.get_action(obs) |
There was a problem hiding this comment.
i'm not sure about this, would we define a new Agent class on top of bgym.Agent ?
There was a problem hiding this comment.
Ahhh crap, I didn't realized it was in browsergym. We could defined TrackedAgent, but hey let's leave it like that for now.
| @pytest.mark.skipif(not OPENROUTER_API_KEY_AVAILABLE, reason="OpenRouter API key is not available") | ||
| def test_get_pricing_openrouter(): | ||
| pricing = tracking.get_pricing_openrouter() | ||
| assert isinstance(pricing, dict) | ||
| assert all(isinstance(v, dict) for v in pricing.values()) | ||
| for model in OPENROUTER_MODELS: | ||
| assert model in pricing | ||
| assert isinstance(pricing[model], dict) | ||
| assert all(isinstance(v, float) for v in pricing[model].values()) | ||
|
|
||
|
|
||
| def test_get_pricing_openai(): | ||
| pricing = tracking.get_pricing_openai() | ||
| assert isinstance(pricing, dict) | ||
| assert all("prompt" in pricing[model] and "completion" in pricing[model] for model in pricing) | ||
| assert all(isinstance(pricing[model]["prompt"], float) for model in pricing) | ||
| assert all(isinstance(pricing[model]["completion"], float) for model in pricing) |
There was a problem hiding this comment.
can't test the value, it would be too demanding. You can test that it's within a
reasonable range though.
|
|
||
| from agentlab.llm.langchain_utils import _convert_messages_to_dict | ||
|
|
||
| TRACKER = threading.local() |
There was a problem hiding this comment.
ohhhh does this make it thread safe?
Cost tracking for openrouter and openai