Proposal to add "Effective Savings Rate" as a KPI

[ecarlin]

Prospoal

I am proposing the F2 adopt a new KPI called Effective Savings Rate (ESR) which measures the overall effectiveness of Rate Optimization efforts (RIs, Savings Plans, CUD, Spot / Pre-emptible, custom pricing, etc.).

Why

There are many "input" metrics that exist today such as utilization, coverage, % spot/preemptible, etc. (which are all good and are current F2 KPIs) but there is no "outcome" based metric that measures actual Rate Optimization performance. ESR allows companies to answer the important question "What is the overall discount I'm achieving with my Rate Optimization efforts?". For example, my use of spot, RIs, SPs, and the EDP I negotiated have all saved me 43.2% on my AWS bill this month. ESR is similar to ROI in that it's a single metric that allows companies to measure, assess, and compare the performance of their investment (or in this case savings) activities.

Formula and Example

Similar to ROI, ESR is a percentage. The formula is straightforward:

ESR = 1 - [(actual price paid + cost to achieve discount) / undiscounted price)]

Here's a simple example: A company is running one instance that costs them $50/mo on-demand. A finops practitioner buys a matching RI which means instead they pay $30/mo. Assuming they use no tools and the cost of their time is zero, their ESR would be 1 - 30/50 = 40%. The company can now say their rate optimization efforts saved them 40% off their cloud bill.

Of course, with an at-scale and dynamic cloud environment, with RIs, SPs, spot/preemptible, changing spot pricing, various coverage rates, various utilization rates, different regional discounts, different instance discounts, different platform discounts, different term commit discounts (1 vs. 3 year), different pre-payment discounts (no, partial, all upfront), different private pricing discounts, CMP and other tooling costs, etc. it's hard to know what the net effect of your rate optimization efforts are. ESR is a simple but powerful formula that lets companies measure this.

Other Factors

• ESR is completely cloud agnostic and is applicable as long as rate optimization options are available (RIs, SPs, CUD, private pricing, etc.)
• ESR is highly benchmark-able. Companies can not only measure their performance but also compare their performance to their peers. This would also allow for a comparison across clouds (I'm making this up, but e.g., on average, GCP customers save more than AWS customers)
• ESR can have different scopes. For example, you might scope it to just compute, or you might scope it to all RI-able services, or to a particular application/division, or to your entire cloud bill.
• Rate Optimization in particular lends itself to an ESR metric because it can be easily baselined. That is, an undiscounted price can objectively be determined. By contrast, in Usage Optimization, it's much more challenging and subjective to establish what you would have paid were the activity not taken. As such, we consider this primarily a Rate Optimization metric, although over time we believe it could potentially be extended to cover Usage Optimization as well.

Origination

Full disclosure: ProsperOps created this metric and we use it very heavily today to assess the performance of our autonomous cloud savings service vs. alternatives. We believe the broad adoption of ESR would greatly benefit companies and the finops community and are happy to contribute it to F2.

Learn More

ProsperOps has published a 3 part blog series on ESR. You can learn more by reading the following blogs:

1. Effective Savings Rate: The most important savings metric you aren't tracking
2. Your financial optimization metrics could be deceiving you
3. Benchmarking Effective Savings Rates: Where does your company rank?

[J0ns0x]

Great one - I also noticed this gap, thank you for posting. We have referred to this as Discount Rate or Effective Discount Rate as it looks like you have in your "why" section and I believe the presenters did in the FinOps Foundation Savings Plan call Aug 2020, would be good to align on the lexicon. For a given set of usage, what is the discount I'm getting at the end of the day compared to on demand rates. This is an important one as it directly captures the financial impacts, compared to other commonly used KPIs around discount mechanisms that are more of a proxy.

[ecarlin]

@J0ns0x Good to know others see the current gap and value of an outcome based savings metric! I totally agree on lexicon alignment. We also use the Effective Discount metric but it measures the discount rate of the savings portfolio itself vs. the overall savings rate. The formula for Effective Discount is similar but slightly different:

Effective Discount = 1 - (Actual Discount Spend / Discount Portfolio On-Demand Equivalent Spend)

Example

(this is an AWS example but the same holds true for GCP and Azure)

• You run 100 m5.large Linux instances in us-east-1 for 30 days
• You have 40 1 year no upfront Linux m5.large Convertible RIs (26% discount)
• You have 40 3 year no upfront Linux m5.large Covertible RIs (49% discount)
• Assume cost to achieve discounts is $0

On-Demand Equivalent (ODE) Spend
Total EC2 ODE spend: $6,912
EC2 ODE covered by 1 year discounts: $2,764.80
EC2 ODE covered by 3 year discounts: $2,764.80

Actual Spend
EC2 on-demand spend: $1,382.40
1 year RI spend: $2,044.80
3 year RI spend: $1,411.20

Metrics
Effective Savings Rate (ESR) = 1 - (Actual Total Spend / Total ODE Spend) = 1 - (4838.40 / 6912) = 30%
Effective Discount = 1 - (Actual Discount Spend / Discount Portfolio ODE Spend) = 1 - (3456 / 5529.60) = 37.5%

Commentary

Effective Discount measures the savings power of the deployed portfolio independent of coverage. In this example, because the company is willing to blend 1 and 3 year commits, they are getting an Effective Discount of 37.5%. If they went all 1 year, it would be 26%. If they went all 3 year, it would be 49%. If they layered in Standard RIs or EC2 Instance SPs (assuming fully utilized), the Effective Discount would increase. If they pre-paid with Partial or All Upfront, the Effective Discount would also increase.

Consider another company with the exact same EC2 usage but they only cover with 10 1 year Convertible RIs and 10 3 year Convertible RIs. Their Effective Discount would be 1 - (864 / 1382.40) = 37.5%. It's exactly the same because they've chosen to deploy an identical percentage mix of 1 and 3 year commitments, although though their coverage is only 20% vs. the original example of 80%.

If we look at ESR, it takes into account total usage which includes the uncovered on-demand spend. Note the ESR of the original example was 30%. While the Effective Discount of what was covered was 37.5%, the overall savings rate including what was left to run on-demand was lower at 30%.

If we look at the ESR for the lower coverage scenario, the Effective Discount is also 37.5% but the ESR would be 1 - (6393.60 / 6912) = 7.5%. The savings power of the portfolio was the same but because it covered less, the ESR was much lower.

Conclusion

ESR measures overall savings rate. Effective Discount measures the savings power of a given portfolio. In order to maximize ESR, you need to both have a high Effective Discount and a high coverage.

This is how ProsperOps has historically defined these 2 metrics and their relationship. Both are important when measuring savings performance. The actual terms themselves are less important than the concepts, although the current names are intuitive imo.

Of course open to thoughts and commentary!

[mshover]

Would like to +1 Effective Savings Rate as a KPI for pricing efficiencies. A couple of minor points to add:

My ideal metric would be:

• Comparable (aka braggable when I'm doing the underlying actions right)
• Easy to measure (ideally both point and trend)
• Meaningful (drives a behavior, else what's the point?)

It would be a metric that I can compare to peers in my industry and see where I rank on a percentile basis, is easy to figure out and trend over time, and that would somehow gauge the discount I've achieved at a given time across all of my pricing efficiencies less waste -- all as % of an optimal discount given my unique environment and risk tolerance so that I'd take action to keep it high over time. The last bit is a pipe dream given the other attributes, especially -- but I think it's important to recognize that the whole purpose of a metric is that it should prompt action. My dream metric is a unicorn/griffin/phoenix (pick your favorite), but ESR is close across all 3 facets. I've taken to using Effective Savings Rate as my chief pricing efficiency metric followed by Waste (unused commitments to include amortization for RIs and SPs as raw $s) as my secondary.

I do agree with @ecarlin on naming as Effective Savings Rate as I perceive the parlance of Discount Rate to be an ROI metric and where each marginal dollar "invested" (at least for a Savings Plan) achieves a lower discount rate -- and is hence less valuable for informing and driving behaviors as well and does not paint as comprehensive of a picture that ESR does.

Last point is on scope. Some organizations will have more tolerance for Spot usage and some will have a scale that allows them to get a larger EDP discount. While it's up to each organization to define their how they'd use the metric and what to include, I'd propose that a "Compare-Scoped" ESR metric be limited to compute only -- then Savings Plans and RIs (to include amortization). I realize that factors like Linux vs Windows or risk tolerance for 1 vs 3 year as well as coverage are but just a few differentiating factors -- but I think it's a rational middle ground. Else, Spot aficionados will take the bragging spoils at FinOps conferences sporting ridiculously unattainable-for-the-rest-of-us Compare-Scoped ESRs.

[vbares]

Also +1 on this metric. We've been using "effective discount" the same way as ESR and resource effective discount as effective discount. I'm moving the team to switch to using ESR and Effective Discount as defined here. The distinction is important and the naming described here is crisper.

[maximeaudet]

Late to the game but +1 for me as well.

Thanks to this thread, I just calculated our % over the years!

[SanjnaSrivatsa]

+1 on this metric. We have been using it in our company quite extensively for a few months and it's a great way to communicate what our Cloud Business Office is able to achieve for our account owners without confusing everyone with different terms and methodologies.