Porting redis throttling improvement related changes

[rmsamitha]

Problem

We have had implemented distributed backend throttling and distributed burst control support with the use of the Redis cluster. This is done by periodically syncing throttling-related parameters such as counter values, first access timestamp, time window, etc. between the Redis server and the Gateway nodes. But the related throttling limits are not accurate due to the periodic sync time. There is a considerable slippage prior to throttling taking place until the sync takes place between gateways and the Redis server. This needs to be reduced to achieve better accuracy at high-load use cases, where conforming to the defined backend throttling limits is critical.

Git issue: wso2/api-manager#1791

Solution:

The presented solution by this PR is a Sync-Async Hybrid model where current asynchronous throttle decision evaluation would take place until exhausting a local quota limit and then the flow would be synchronous. All the gateways will be acknowledged through a pub/sub approach, whenever any gateway node would exhaust the local quota. Once the acknowledgment is received, each gateway would query the Redis server synchronously to fetch the shared counter, so that the throttling decision will be evaluated globally. So there would be no slippage taking place as the counter evaluation will be done through a sync flow when the API request counts come closer to the specified throttle limits. Hence this approach solves the evident problem caused by the existing asynchronous periodically-syncing-based approach.