How to handle numerical errors in pmf probabilities summing to 1?

[elray1]

For hubEvals, we're using the scoringutils package, which in turn uses scoringRules to calculate the ranked probability score (rps). Between hubValidations, scoringutils, and scoringRules, there are three different checks that are being done for whether or not class probabilities sum to 1:

• hubverse validations require that class probabilities sum to 1 within a tolerance of +/- the square root of machine epsilon (this is the default tolerance in all.equal in R), which depends on machine architecture, but for me is about 1.5e-8.
• The scoringutils package requires that class probabilities sum to 1 within a tolerance of +/- 1e-10.
• The scoringRules package requires that the sum of the class probabilities minus 1 is <= machine epsilon, which is about 2.2e-16 for me.

There are two issues here:

1. I'm pretty sure the scoringRules validations are more stringent than we can guarantee we'll be able to satisfy. Even normalizing a vector by its sum could yield something that fails this condition:

> set.seed(42)
> x <- abs(rnorm(n = 100, mean = 0.5))
> x <- x / sum(x)
> isTRUE(abs(sum(x) - 1) > .Machine$double.eps)
[1] TRUE
> abs(sum(x) - 1)
[1] 4.440892e-16
> .Machine$double.eps
[1] 2.220446e-16

So I think we should file an issue about this at scoringRules, but I'm not knowledgeable enough about this to be confident in what to suggest that they change their check to. Should it just be sqrt(.Machine$double.eps)?

2. We may end up in a situation where a pmf submission passed hubverse validations, but fails to pass scoringutils and/or scoringRules validations. How should we handle this?

Three ideas:

1. Allow scoringutils and/or scoringRules to throw errors if their more-stringent criteria are not met.
2. Try to call the scoringrules function, but catch errors. If errors are thrown, normalize the class probabilities on behalf of the user and issue a warning instead of an error.
3. Always normalize class probabilities. We're assuming the model outputs this function is being handed passed validations, so the class probabilities come close to 1 and we're doing some minor cleanup on behalf of the package user.

[nickreich]

Regarding options 2 and 3 above for the "ideas" section: It wasn't clear to me if normalizing would solve things, given the example you provide above. It seems like the simplest thing for us to do at hubverse would be idea 1 (just return the errors). How much would that be then putting back on users to correct/normalize things? And is that work feasible. In general, it seems like idea 1 is the simpler and more ideal solution (less for hubverse to maintain down the road).

[zkamvar]

It seems like it's worth bringing this up with the scoringRules maintainers, especially since you cannot guarantee that a normalized vector will sum to one. From what I understand the sqrt(.Machine$double.eps) dates back to numerical recipes (https://scicomp.stackexchange.com/a/14356).

[annakrystalli]

First a question. Have we established that submissions files (i.e. outputs of most models) are more likely than not to fail the scoringRules criteria? Searching the, admittedly limited, number of issues in their repo, I can't see anyone else having reported a problem with their tolerance. So it could be that we could just change our tolerance in hubValidations with minimum disruption.

Having said that, I feel the standard R equality test tolerance should be an acceptable tolerance for R stats packages. As such I also vote for contacting the scoringRules maintainers. Instead of requiring them to change the tolerance, however, perhaps a better suggestion would be for them to introduce a tolerance argument (like the one all.equal() has) that they can set to whatever tolerance they prefer as default but we can use to override.