Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rules engine refactoring #1037

Merged
merged 36 commits into from
May 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
f5e94fe
Refactored validation rule engine in preparation for tranformation en…
basiliskus Apr 23, 2024
28505a5
Fixed instantiation
basiliskus Apr 23, 2024
2c3f69d
Merge branch 'main' into story/1024/transformation-engine
basiliskus Apr 24, 2024
bdd568a
Refactored to make Rule interface more generic
basiliskus Apr 25, 2024
f65cede
Fixed tests
basiliskus Apr 25, 2024
0cf5640
Update transformation_definitions.json
luis-pabon-tf Apr 25, 2024
a08c687
Merge branch 'main' into story/1024/transformation-engine
basiliskus Apr 25, 2024
8631334
Added missing override tags
basiliskus Apr 25, 2024
6214e84
Fixed arg naming
basiliskus Apr 25, 2024
4756364
Use generics to deserialize correct type of Rule implementation
basiliskus Apr 25, 2024
1de28ef
Converted RuleEngine back into a singleton and added parameters
basiliskus Apr 26, 2024
5c8073b
Refactored to have a separate implementation for ValidationRuleEngine
basiliskus Apr 26, 2024
45a3bca
Refactored to pass TypeReference and fix mapping issue
basiliskus Apr 26, 2024
d5490e8
Added missing override tags
basiliskus Apr 26, 2024
8e7398c
Fixed tests in ValidationRuleEngineIntegrationTest
basiliskus Apr 26, 2024
dbd8043
Fixed more tests
basiliskus Apr 26, 2024
d399054
Removed transformation engine implementation to move into new PR
basiliskus Apr 26, 2024
76b7d82
Update ValidationRuleTest.groovy
luis-pabon-tf Apr 29, 2024
27ea75d
Update ValidationRuleEngineTest.groovy
luis-pabon-tf Apr 29, 2024
0f1b275
Update ValidationRuleEngine.java
luis-pabon-tf Apr 29, 2024
5acab7a
Update ValidationRuleEngineTest.groovy
luis-pabon-tf Apr 29, 2024
28b0d7a
Update ValidationRuleEngineTest.groovy
luis-pabon-tf Apr 29, 2024
1f57bb1
Update Rule loading to handle Path for flexibility
saquino0827 Apr 29, 2024
5d4dd9e
Merge branch 'story/1024/transformation-engine' of https://github.com…
saquino0827 Apr 29, 2024
34a2bf9
Fixed remaining test
basiliskus Apr 29, 2024
e137c1d
Fixed actual remaining test
basiliskus Apr 29, 2024
0fc78c5
Changed naming
basiliskus Apr 29, 2024
95d4b2d
Update ValidationRuleEngineTest.groovy
luis-pabon-tf Apr 29, 2024
a27a876
Merge branch 'story/1024/transformation-engine' of https://github.com…
luis-pabon-tf Apr 29, 2024
c0f0093
Added test coverage
basiliskus Apr 29, 2024
2a5f7ab
Updated javadocs
basiliskus Apr 29, 2024
4a5068d
Created trasnformation package and moved transformation engine files
basiliskus Apr 29, 2024
2bd21c8
Changed implementation for rules implementation to inherit from Rule …
basiliskus Apr 29, 2024
6ba1943
Clean up
basiliskus Apr 29, 2024
a03dd0b
Added missing override tag
basiliskus Apr 29, 2024
e9290aa
Create RuleTest.groovy
luis-pabon-tf Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@
import gov.hhs.cdc.trustedintermediary.etor.results.ResultResponse;
import gov.hhs.cdc.trustedintermediary.etor.results.ResultSender;
import gov.hhs.cdc.trustedintermediary.etor.results.SendResultUseCase;
import gov.hhs.cdc.trustedintermediary.etor.ruleengine.RuleEngine;
import gov.hhs.cdc.trustedintermediary.etor.ruleengine.RuleLoader;
import gov.hhs.cdc.trustedintermediary.etor.ruleengine.validation.ValidationRuleEngine;
import gov.hhs.cdc.trustedintermediary.external.database.DatabaseMessageLinkStorage;
import gov.hhs.cdc.trustedintermediary.external.database.DatabasePartnerMetadataStorage;
import gov.hhs.cdc.trustedintermediary.external.database.DbDao;
Expand Down Expand Up @@ -133,7 +133,9 @@ public Map<HttpEndpoint, Function<DomainRequest, DomainResponse>> domainRegistra
PartnerMetadataConverter.class, HapiPartnerMetadataConverter.getInstance());
// Validation rules
ApplicationContext.register(RuleLoader.class, RuleLoader.getInstance());
ApplicationContext.register(RuleEngine.class, RuleEngine.getInstance());
ApplicationContext.register(
ValidationRuleEngine.class,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first value in register should be an interface and the second should be the class that is defining the behavior for an interface

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As commented above, we need to be able to identify two different implementations of the rule engine

Copy link
Contributor

@jorg3lopez jorg3lopez May 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on doing a parent interface (RuleEngine), and two child interfaces that will extend the parent interface (ValidationRuleEngine, TransformationRuleEngine)? With this setup, you can use the child interface to register the implementation for each (ValidationRuleEngineImp, TransformationRuleEngineImp). The only challenge here is the naming for the parent interface, the child interfaces, and the implementations for each child interface. You register both engines with a unique interface for each.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the value we register in the ApplicationContext needs to be an interface, a lot of the existing ones are not. Is there a benefit in creating the interface even if it will have only one implementation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we generally wrap 3rd party dependencies in an interface to reduce dependencies

ValidationRuleEngine.getInstance("validation_definitions.json"));

ApplicationContext.register(SendMessageHelper.class, SendMessageHelper.getInstance());

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import gov.hhs.cdc.trustedintermediary.domainconnector.DomainRequest;
import gov.hhs.cdc.trustedintermediary.etor.metadata.EtorMetadataStep;
import gov.hhs.cdc.trustedintermediary.etor.ruleengine.RuleEngine;
import gov.hhs.cdc.trustedintermediary.etor.ruleengine.validation.ValidationRuleEngine;
import gov.hhs.cdc.trustedintermediary.external.hapi.HapiFhirResource;
import gov.hhs.cdc.trustedintermediary.external.hapi.HapiOrder;
import gov.hhs.cdc.trustedintermediary.wrappers.FhirParseException;
Expand All @@ -20,7 +20,7 @@ public class OrderController {
@Inject HapiFhir fhir;
@Inject Logger logger;
@Inject MetricMetadata metadata;
@Inject RuleEngine ruleEngine;
@Inject ValidationRuleEngine validationEngine;
jcrichlake marked this conversation as resolved.
Show resolved Hide resolved

private OrderController() {}

Expand All @@ -31,7 +31,7 @@ public static OrderController getInstance() {
public Order<?> parseOrders(DomainRequest request) throws FhirParseException {
logger.logInfo("Parsing orders");
var fhirBundle = fhir.parseResource(request.getBody(), Bundle.class);
ruleEngine.validate(new HapiFhirResource(fhirBundle));
validationEngine.runRules(new HapiFhirResource(fhirBundle));
metadata.put(fhirBundle.getId(), EtorMetadataStep.RECEIVED_FROM_REPORT_STREAM);
return new HapiOrder(fhirBundle);
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,28 +1,85 @@
package gov.hhs.cdc.trustedintermediary.etor.ruleengine;

import gov.hhs.cdc.trustedintermediary.context.ApplicationContext;
import gov.hhs.cdc.trustedintermediary.wrappers.HapiFhir;
import gov.hhs.cdc.trustedintermediary.wrappers.Logger;
import java.util.List;

/**
* The Rule interface defines the structure for a rule in the rule engine. Each rule has a name,
* description, warning message, conditions, validations, and methods to check if a resource is
* valid and if the rule applies to a resource.
* Represents a rule that can be run on a FHIR resource. Each rule has a name, description, logging
* message, conditions to determine if the rule should run, and actions to run in case the condition
* is met.
*/
public interface Rule {
String getName();
public class Rule {

String getDescription();
protected final Logger logger = ApplicationContext.getImplementation(Logger.class);
protected final HapiFhir fhirEngine = ApplicationContext.getImplementation(HapiFhir.class);
private String name;
private String description;
private String message;
private List<String> conditions;
private List<String> rules;

/**
* Descriptive message when there's a rule violation Note: When implementing this method, make
* sure that no PII or PHI is included in the message!
* Do not delete this constructor! It is used for JSON deserialization when loading rules from a
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, Jackson requires an empty constructor to de-serialize things, don't have to add this but something to think about

* file.
*/
String getViolationMessage();
public Rule() {}

List<String> getConditions();
public Rule(
String ruleName,
String ruleDescription,
String ruleMessage,
List<String> ruleConditions,
List<String> ruleActions) {
name = ruleName;
description = ruleDescription;
message = ruleMessage;
conditions = ruleConditions;
rules = ruleActions;
}

List<String> getValidations();
public String getName() {
return name;
}

boolean isValid(FhirResource<?> resource);
public String getDescription() {
return description;
}

boolean appliesTo(FhirResource<?> resource);
public String getMessage() {
return message;
}

public List<String> getConditions() {
return conditions;
}

public List<String> getRules() {
return rules;
}

public boolean shouldRun(FhirResource<?> resource) {
return conditions.stream()
.allMatch(
condition -> {
try {
return fhirEngine.evaluateCondition(
resource.getUnderlyingResource(), condition);
} catch (Exception e) {
logger.logError(
"Rule ["
+ name
+ "]: "
+ "An error occurred while evaluating the condition: "
+ condition,
e);
return false;
}
});
}

public void runRule(FhirResource<?> resource) {
throw new UnsupportedOperationException("This method must be implemented by subclasses.");
}
}
Original file line number Diff line number Diff line change
@@ -1,63 +1,13 @@
package gov.hhs.cdc.trustedintermediary.etor.ruleengine;

import gov.hhs.cdc.trustedintermediary.wrappers.Logger;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import javax.inject.Inject;
/**
* The RuleEngine interface defines the structure for a rule engine. Each rule engine has methods to
* load rules, ensure rules are loaded, and run rules on a resource.
*/
public interface RuleEngine {
void unloadRules();

/** Manages the application of rules loaded from a definitions file using the RuleLoader. */
public class RuleEngine {
void ensureRulesLoaded() throws RuleLoaderException;

private static final RuleEngine INSTANCE = new RuleEngine();

final List<Rule> rules = new ArrayList<>();

@Inject Logger logger;
@Inject RuleLoader ruleLoader;

private RuleEngine() {}

public static RuleEngine getInstance() {
return INSTANCE;
}

public void unloadRules() {
rules.clear();
}

public void ensureRulesLoaded() {
if (rules.isEmpty()) {
synchronized (this) {
if (rules.isEmpty()) {
loadRules();
}
}
}
}

private synchronized void loadRules() {
String fileName = "rule_definitions.json";
try (InputStream ruleDefinitionStream =
getClass().getClassLoader().getResourceAsStream(fileName)) {
assert ruleDefinitionStream != null;
var ruleStream =
new String(ruleDefinitionStream.readAllBytes(), StandardCharsets.UTF_8);
rules.addAll(ruleLoader.loadRules(ruleStream));
} catch (IOException | RuleLoaderException e) {
logger.logError("Failed to load rules definitions from: " + fileName, e);
}
}

public void validate(FhirResource<?> resource) {
logger.logDebug("Validating FHIR resource");
ensureRulesLoaded();
for (Rule rule : rules) {
if (rule.appliesTo(resource) && !rule.isValid(resource)) {
logger.logWarning("Rule violation: " + rule.getViolationMessage());
}
}
}
void runRules(FhirResource<?> resource);
}
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
package gov.hhs.cdc.trustedintermediary.etor.ruleengine;

import gov.hhs.cdc.trustedintermediary.wrappers.Logger;
import gov.hhs.cdc.trustedintermediary.wrappers.formatter.Formatter;
import gov.hhs.cdc.trustedintermediary.wrappers.formatter.FormatterProcessingException;
import gov.hhs.cdc.trustedintermediary.wrappers.formatter.TypeReference;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Collections;
import java.util.List;
import java.util.Map;
Expand All @@ -12,21 +18,25 @@
public class RuleLoader {
private static final RuleLoader INSTANCE = new RuleLoader();
@Inject Formatter formatter;
@Inject Logger logger;

private RuleLoader() {}

public static RuleLoader getInstance() {
return INSTANCE;
}

public List<ValidationRule> loadRules(String ruleStream) throws RuleLoaderException {
try {
Map<String, List<ValidationRule>> jsonObj =
formatter.convertJsonToObject(ruleStream, new TypeReference<>() {});
return jsonObj.getOrDefault("rules", Collections.emptyList());
} catch (FormatterProcessingException e) {
public <T> List<T> loadRules(Path path, TypeReference<Map<String, List<T>>> typeReference)
throws RuleLoaderException {
try (InputStream ruleDefinitionStream = Files.newInputStream(path)) {
var rulesString =
new String(ruleDefinitionStream.readAllBytes(), StandardCharsets.UTF_8);
Map<String, List<T>> jsonObj =
formatter.convertJsonToObject(rulesString, typeReference);
return jsonObj.getOrDefault("definitions", Collections.emptyList());
} catch (IOException | FormatterProcessingException e) {
throw new RuleLoaderException(
"Failed to load rules definitions for provided stream", e);
"Failed to load rules definitions for provided path: " + path, e);
}
}
}

This file was deleted.

Loading
Loading