Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DG-1986 | Use policy update time as last refresh time #3967

Merged
merged 4 commits into from
Jan 20, 2025
Merged

DG-1986 | Use policy update time as last refresh time #3967

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b3f6392
Use policy update time as last refresh time
krsoninikhil Jan 9, 2025
55d6e7c
Throw exception if policy engine is null
krsoninikhil Jan 14, 2025
974c50f
Add logs for successful policy engine update
krsoninikhil Jan 17, 2025
49f8eae
Update timestamp and updated by on delete entity operation as well
krsoninikhil Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions auth-agents-common/src/main/java/org/apache/atlas/plugin/service/RangerBasePlugin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ public void setPolicies(ServicePolicies policies) {
if (defaultSvcPolicies == null) {
LOG.error("Could not get default Service Policies. Keeping old policy-engine! This is a FATAL error as the old policy-engine is null!");
isNewEngineNeeded = false;
throw new RuntimeException("PolicyRefresher("+policies.getServiceName()+").setPolicies: fetched service policies contains no policies or delta and current policy engine is null");
} else {
defaultSvcPolicies.setPolicyVersion(policies.getPolicyVersion());
policies = defaultSvcPolicies;
Expand Down Expand Up @@ -419,6 +420,7 @@ public void setPolicies(ServicePolicies policies) {
if (this.refresher != null) {
this.refresher.saveToCache(usePolicyDeltas ? servicePolicies : policies);
}
LOG.info("New RangerPolicyEngine created with policy count:"+ (usePolicyDeltas? servicePolicies.getPolicies().size() : policies.getPolicies().size()));
}

} else {
Expand Down
2 changes: 1 addition & 1 deletion auth-agents-common/src/main/java/org/apache/atlas/plugin/util/PolicyRefresher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,7 +295,7 @@ private void loadPolicy() {
lastUpdatedTimeInMillis = -1;
}
} catch (Exception excp) {
LOG.error("Encountered unexpected exception!!!!!!!!!!!", excp);
LOG.error("Encountered unexpected exception!!!!!!!!!!! Message:" + excp.getMessage() + "Stacktrace: " + excp.getStackTrace().toString(), excp);
}

RangerPerfTracer.log(perf);
Expand Down
2 changes: 2 additions & 0 deletions ...s-common/src/main/java/org/apache/atlas/policytransformer/CachePolicyTransformerImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,8 @@ public ServicePolicies getPoliciesDelta(String serviceName, Map<String, EntityAu

ArrayList<String> policyGuids = new ArrayList<>(policyChanges.keySet());
List<AtlasEntityHeader> allAtlasPolicies = getAtlasPolicies(serviceName, POLICY_BATCH_SIZE, policyGuids);
Date latestUpdateTime = allAtlasPolicies.stream().map(AtlasEntityHeader::getUpdateTime).max(Date::compareTo).orElse(null);
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be either first or last policy time.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it will be first and when it will be last?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nikhilbonte21 had suggested to use either last or first item from allAtlasPolicies rather than finding the maximum since it is already sorted. So I had noted it down.

But the allAtlasPolicies array is sorted on __timestamp and not on updateTime, and it's an existing method, so edge elements won't give maximum updatTime.

servicePolicies.setPolicyUpdateTime(latestUpdateTime);

List<AtlasEntityHeader> atlasServicePolicies = allAtlasPolicies.stream().filter(x -> serviceName.equals(x.getAttribute(ATTR_POLICY_SERVICE_NAME))).collect(Collectors.toList());
List<RangerPolicyDelta> policiesDelta = getRangerPolicyDelta(service, policyChanges, atlasServicePolicies);
Expand Down
2 changes: 2 additions & 0 deletions repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1974,6 +1974,8 @@ private EntityMutationResponse deleteVertices(Collection<AtlasVertex> deletionCa

MetricRecorder metric = RequestContext.get().startMetricRecord("filterCategoryVertices");
for (AtlasVertex vertex : deletionCandidates) {
updateModificationMetadata(vertex);

String typeName = getTypeName(vertex);

List<PreProcessor> preProcessors = getPreProcessor(typeName);
Expand Down
9 changes: 4 additions & 5 deletions webapp/src/main/java/org/apache/atlas/web/rest/AuthREST.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,12 @@ public ServicePolicies downloadPolicies(@PathParam("serviceName") final String s
ServicePolicies ret;
if (usePolicyDelta) {
List<EntityAuditEventV2> auditEvents = getPolicyAuditLogs(serviceName, lastUpdatedTime);
long lastEventTime = auditEvents.isEmpty() ? 0 : auditEvents.get(auditEvents.size() - 1).getCreated();
LOG.info("PolicyDelta: serviceName={}, lastUpdatedTime={}, audit events found={}", serviceName, lastEventTime, auditEvents.size());
LOG.info("PolicyDelta: serviceName={}, lastUpdatedTime={}, audit events found={}", serviceName, lastUpdatedTime, auditEvents.size());
if (auditEvents.isEmpty()) {
return null;
}
Map<String, EntityAuditEventV2.EntityAuditActionV2> policyChanges = policyTransformer.createPolicyChangeMap(serviceName, auditEvents);
ret = policyTransformer.getPoliciesDelta(serviceName, policyChanges, lastEventTime);
ret = policyTransformer.getPoliciesDelta(serviceName, policyChanges, lastUpdatedTime);
} else {
if (!isPolicyUpdated(serviceName, lastUpdatedTime)) {
return null;
Expand Down Expand Up @@ -196,12 +195,12 @@ private List<EntityAuditEventV2> getPolicyAuditLogs(String serviceName, long las
mustClauseList.add(getMap("terms", getMap("typeName", entityUpdateToWatch)));

lastUpdatedTime = lastUpdatedTime == -1 ? 0 : lastUpdatedTime;
mustClauseList.add(getMap("range", getMap("created", getMap("gt", lastUpdatedTime))));
mustClauseList.add(getMap("range", getMap("timestamp", getMap("gt", lastUpdatedTime))));

dsl.put("query", getMap("bool", getMap("must", mustClauseList)));

List<Map<String, Object>> sortClause = new ArrayList<>();
sortClause.add(getMap("created", getMap("order", "asc")));
sortClause.add(getMap("timestamp", getMap("order", "asc")));
dsl.put("sort", sortClause);

int from = 0;
Expand Down
Loading