improved retry behavior

[jjallaire]

Various suggestions from @craigwalton-dsit:

Sometimes when running evals against models which have flaky APIs which trigger their is_rate_limit() override to return True, users get confused as to why their eval seems to be "stuck" indefinitely. A few observations:

1. Some provider implementations like google treat numerous errors as is_rate_limit (e.g. 500, 503 and 504) (source). Maybe this override would be better named is_retryable_error.

2. AFAIK it is not that obvious to users that Inspect is busy retrying/waiting to retry failed HTTP requests. The inspect trace anomalies is helpful. It might be even more user-friendly if the "Running samples" UI somehow indicated that a sample was in a retry loop (a bit like how we show "Generating ...").

3. The "rate limits" counter in the UI only updates for actual HTTP 429 errors (not other errors we treat as rate limits/retryable). This might well be the right behaviour, but just wanted to highlight this was one source of confusion. May be resolved by tackling 1.\

[tadamcz]

Previous discussion: #1174