DG-1986 | Use policy update time as last refresh time

[krsoninikhil]

Change description

This PR contains following 3 small / one liner changes:

1. Use highest updated time from fetched policies as last refresh time instead of audit event time. This will keep the last updated time always equal to a actual policy update time, allowing any missed policies to be fetched in next cycle even if policies are not synced with ES but audit event is generated.

Since this timestamp is used to while fetching next set of policies, earlier method where audit event time was getting used had an edge case where if ES is not synced with policies that are created, policy refresher will not find that policy but since the last updated timestamp is updated with event time, it will be missed in subsequent fetches as well.

2. Currently, deleting an entity doesn't update updateTime of the entity, so if a policy is updated and then deleted, there would two audit events but both will have same value for timestamp which is the time when it was updated. This means then the delete event will not be catched by policy refresher since it has the already tracked timestamp which was generated later on.

3. Adds an exception if policy refresher does not find any policies to create the policy engine. Earlier it was silently getting handled causing problem when delta is applied since there was no policy engine to apply delta to. Throwing exception allows it to keep trying to create engine if no policies are available in the system e.g. in case of atlas reset.

Test Cases

1. Run atlas reset workflow to ensure typdefs are able to seed, ensuring the policy engine is getting created - ✅
2. Execute a workflow after resetting to ensure policies are getting refreshed - ✅
3. Create / update / delete a metadata policy in a Persona to ensure existing flow is untouched - ✅
4. Create / update / delete a metadata policy in a Purpose to ensure existing flow for tag is also untouched - ✅
5. Delete an entity and ensure timestamp is getting updated - ✅

Type of change

• Bug fix (fixes an issue)
• New feature (adds functionality)

Related issues

Fix #1

Checklists

Development

• Lint rules pass locally
• Application changes have been tested thoroughly
• Automated tests covering modified code pass

Security

• Security impact of change has been considered
• Code follows company security practices and guidelines

Code review

• Pull request has a descriptive title and context useful to a reviewer. Screenshots or screencasts are attached as necessary
• "Ready for review" label attached and reviewers assigned
• Changes have been reviewed by at least one other contributor
• Pull request linked to task tracker where applicable

[krsoninikhil]

This will be either first or last policy time.

[sumandas0]

When it will be first and when it will be last?

[krsoninikhil]

@nikhilbonte21 had suggested to use either last or first item from allAtlasPolicies rather than finding the maximum since it is already sorted. So I had noted it down.

But the allAtlasPolicies array is sorted on __timestamp and not on updateTime, and it's an existing method, so edge elements won't give maximum updatTime.