Enforcing cost limits

[jjallaire]

Set a cost limit for a sample or a task and have it automatically terminate when the limit is reached.

Note that there is no general way to determine the pricing of models dynamically (and if we hard code them into the package they are likely to be wrong). Therefore we'll likely want users to let us know what their per-token cost is when using this feature.

[Taytay]

LiteLLM handles pricing of various models with a json file they keep and serve here: https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json

You can add to it or change it at runtime.

I'm fine keep this on the user for now if that makes this easier to build. But BOY, I would love to see a token AND price breakdown after the fact!

[zarSou9]

I wonder if it would be worth switching to use LiteLLM in general