Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

621: Save Metadata from FHIR Sender/Receiver/Number #985

Merged
merged 51 commits into from
Apr 11, 2024
Merged

621: Save Metadata from FHIR Sender/Receiver/Number #985

Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
b8585fc
Start pumping the new metadata into the metadata system
halprin Mar 28, 2024
bf30102
Pipe in the new metadata from annother direction
halprin Mar 28, 2024
ca7a966
Remove unused metadata method that was only ever called in a test
halprin Mar 28, 2024
7c14783
Fix up the with methods in the Metadata POJO
halprin Mar 28, 2024
2b725ae
Removed another unused method from the Metdata POJO
halprin Mar 28, 2024
db0adc7
Merge branch 'story-621-dbdoa_generic' into story-621-save_new_metadata
halprin Apr 1, 2024
90c7b8e
Write new FHIR sender and receiver to the metadata columns
halprin Apr 1, 2024
489f725
Merge branch 'story-621-dbdoa_generic' into story-621-save_new_metadata
halprin Apr 1, 2024
53647a7
Construct the metadata using the new fields from the database
halprin Apr 1, 2024
85589cb
Merge branch 'main' into story-621-save_new_metadata
halprin Apr 2, 2024
b4dc9f0
Fix unit test
halprin Apr 2, 2024
26bb22e
Fix unit tests
halprin Apr 2, 2024
7b45f8c
Fix rest of unit tests
halprin Apr 2, 2024
77f44fe
Merge branch 'main' into story-621-save_new_metadata
halprin Apr 2, 2024
9acc112
Merge branch 'story/621/extract-ids' into story-621-save_new_metadata
saquino0827 Apr 5, 2024
65d0fd2
Update PartnerMetadata, Order/Result Usecase, and Dao classes to use …
saquino0827 Apr 5, 2024
987fe88
Update usage of MessageHdDatatype into PartnerMetadata testing
saquino0827 Apr 5, 2024
2fab464
Add commented test to be written soon
saquino0827 Apr 5, 2024
f716749
Merge branch 'story/621/extract-ids' into story-621-save_new_metadata
saquino0827 Apr 5, 2024
45bef13
Integrate MessageHdDatatype into PartnerMetadataOrchestratorTest
saquino0827 Apr 5, 2024
eea2efc
Integrate MessageHdDatatype with SendMessageHelperTest
saquino0827 Apr 5, 2024
3937309
Merge branch 'main' into story-621-save_new_metadata
saquino0827 Apr 5, 2024
68ad8c4
Merge branch 'main' into story-621-save_new_metadata
luis-pabon-tf Apr 8, 2024
66f97d0
Fixing unit tests to handle MessageHdDatatype and it's return type fo…
saquino0827 Apr 8, 2024
edd0552
change liquibase type for new columns
saquino0827 Apr 8, 2024
ad22d54
Merge branch 'story-621-save_new_metadata' of https://github.com/CDCg…
luis-pabon-tf Apr 9, 2024
92bfa2a
Update PostgresDaoTest.groovy
luis-pabon-tf Apr 9, 2024
40f6698
Merge branch 'main' into story-621-save_new_metadata
luis-pabon-tf Apr 9, 2024
b4931e1
unit test: readMetadata triggers FormatterProcessingException
jorg3lopez Apr 9, 2024
8485cd8
unit test: readMetadatForSender triggers FormatProcessingException
jorg3lopez Apr 9, 2024
d366dfd
unit test: readMetadatForSender triggers SQLException
jorg3lopez Apr 9, 2024
b9c2ea3
unit test: readMetadataForSender happy path works
jorg3lopez Apr 9, 2024
eaf3982
refactoring unit test
jorg3lopez Apr 9, 2024
e424ff8
Update for database handling of the jsonb datatype
saquino0827 Apr 10, 2024
51adfbe
Merge branch 'story-621-save_new_metadata' of https://github.com/CDCg…
saquino0827 Apr 10, 2024
cbf0fb6
Add jackson constructor and remove formatting from DatabasePartnerMet…
saquino0827 Apr 10, 2024
0cd92fa
Update PostgresDao to handle all Jackson formatting and remove it fro…
saquino0827 Apr 10, 2024
2061d30
Refactor PartnerMetadataOrchestratorTest to re-use all MessageHdDatat…
saquino0827 Apr 11, 2024
cc2c378
unit test: sameMetadata triggers FormatProcessingException
jorg3lopez Apr 11, 2024
c975439
Refactor unnecessary jsonb formatting out of PostgresDao and into Dat…
saquino0827 Apr 11, 2024
d11cc18
Merge branch 'story-621-save_new_metadata' of https://github.com/CDCg…
saquino0827 Apr 11, 2024
99fb6f5
Remove comments and fix typo's
saquino0827 Apr 11, 2024
0b1427d
Update DatabasePartnerMetadataStorageTest exception throwing to handl…
saquino0827 Apr 11, 2024
2c97cae
adding helper method to create List<DbColumn>
jorg3lopez Apr 11, 2024
4045075
merge conflicts
jorg3lopez Apr 11, 2024
880cf82
deleted space before list<DbColumn>
jorg3lopez Apr 11, 2024
ec20d90
refactor: added helper method that returns List<DbColumn>
jorg3lopez Apr 11, 2024
10c6dd8
Update DatabasePartnerMetadataStorage to re-add formatting and fix Po…
saquino0827 Apr 11, 2024
58334ae
Merge branch 'story-621-save_new_metadata' of https://github.com/CDCg…
saquino0827 Apr 11, 2024
c1d902d
Fix imports and remove merge conflict file
saquino0827 Apr 11, 2024
b22d5e9
Merge branch 'main' into story-621-save_new_metadata
saquino0827 Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 48 additions & 5 deletions etor/databaseMigrations/metadata.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ databaseChangeLog:
context: metadata
comment: create status type
changes:
- sql:
sql: CREATE TYPE message_status AS ENUM ('PENDING', 'DELIVERED', 'FAILED')
- sql:
sql: CREATE TYPE message_status AS ENUM ('PENDING', 'DELIVERED', 'FAILED')
rollback:
- sql:
sql: DROP TYPE message_status
Expand All @@ -19,7 +19,7 @@ databaseChangeLog:
context: metadata
comment: create partner metadata table
changes:
- createTable:
- createTable:
tableName: metadata
columns:
- column:
Expand Down Expand Up @@ -60,8 +60,8 @@ databaseChangeLog:
context: metadata
comment: create message type
changes:
- sql:
sql: CREATE TYPE message_type AS ENUM ('RESULT', 'ORDER')
- sql:
sql: CREATE TYPE message_type AS ENUM ('RESULT', 'ORDER')
rollback:
- sql:
sql: DROP TYPE message_type
Expand Down Expand Up @@ -109,3 +109,46 @@ databaseChangeLog:
- column:
name: receiving_facility_id
type: varchar(227)
- changeSet:
id: 6
author: samuel.aquino
labels: update-metadata-table
context: metadata
comment: update partner metadata table for order and result linking using jsonb
changes:
- renameColumn:
columnDataType: varchar(227)
oldColumnName: sending_application_id
newColumnName: sending_application_details
tableName: metadata
- renameColumn:
columnDataType: varchar(227)
oldColumnName: sending_facility_id
newColumnName: sending_facility_details
tableName: metadata
- renameColumn:
columnDataType: varchar(227)
oldColumnName: receiving_application_id
newColumnName: receiving_application_details
tableName: metadata
- renameColumn:
columnDataType: varchar(227)
oldColumnName: receiving_facility_id
newColumnName: receiving_facility_details
tableName: metadata
- modifyDataType:
columnName: sending_application_details
newDataType: jsonb
tableName: metadata
- modifyDataType:
columnName: sending_facility_details
newDataType: jsonb
tableName: metadata
- modifyDataType:
columnName: receiving_application_details
newDataType: jsonb
tableName: metadata
- modifyDataType:
columnName: receiving_facility_details
newDataType: jsonb
tableName: metadata
6 changes: 5 additions & 1 deletion etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/messages/MessageHdDataType.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,8 @@
* identifier (ID), and the type of the universal ID. HD reference: <a
* href="https://hl7-definition.caristix.com/v2/HL7v2.5.1/DataTypes/HD">HD-DataType</a>
*/
public record MessageHdDataType(String namespace, String universalId, String universalIdType) {}
public record MessageHdDataType(String namespace, String universalId, String universalIdType) {

// Constructor for Jackson Mapping
public MessageHdDataType {}
}
18 changes: 16 additions & 2 deletions etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/messages/SendMessageHelper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,14 @@ public static SendMessageHelper getInstance() {
private SendMessageHelper() {}

public void savePartnerMetadataForReceivedMessage(
String receivedSubmissionId, int messageHash, PartnerMetadataMessageType messageType) {
String receivedSubmissionId,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thoughts on passing an object that will hold all this data so we can have one parameter?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm down for something like PartnerMetadataMessage or something along those lines. I would like others thoughts on it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmmm, I'm fine passing along the individual arguments, but I wouldn't create a whole new class when a lot of this data just belongs in PartnerMetadata. Today we construct the PartnerMetadata inside PartnerMetadataOrchestrator. I'd be curious to hear people's thoughts on constructing PartnerMetadata in the usecase or somewhere else outside the orchestrator.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having an object instead of many params will be slightly more readable at this point. I don't have super strong feelings on it though

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can bubble up the construction of the PartnerMetadata? That could lead to easier change to that object in the future

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this as is for now and we can discuss this in our engineering block if we want to consider refactoring.

int messageHash,
PartnerMetadataMessageType messageType,
MessageHdDataType sendingApplicationDetails,
MessageHdDataType sendingFacilityDetails,
MessageHdDataType receivingApplicationDetails,
MessageHdDataType receivingFacilityDetails,
String placerOrderNumber) {
if (receivedSubmissionId == null) {
logger.logWarning(
"Received submissionId is null so not saving metadata for received message");
Expand All @@ -30,7 +37,14 @@ public void savePartnerMetadataForReceivedMessage(
String stringMessageHash = String.valueOf(messageHash);

partnerMetadataOrchestrator.updateMetadataForReceivedMessage(
receivedSubmissionId, stringMessageHash, messageType);
receivedSubmissionId,
saquino0827 marked this conversation as resolved.
Show resolved Hide resolved
stringMessageHash,
messageType,
sendingApplicationDetails,
sendingFacilityDetails,
receivingApplicationDetails,
receivingFacilityDetails,
placerOrderNumber);
} catch (PartnerMetadataException e) {
logger.logError(
"Unable to save metadata for receivedSubmissionId " + receivedSubmissionId, e);
Expand Down
101 changes: 79 additions & 22 deletions .../src/main/java/gov/hhs/cdc/trustedintermediary/etor/metadata/partner/PartnerMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package gov.hhs.cdc.trustedintermediary.etor.metadata.partner;

import gov.hhs.cdc.trustedintermediary.etor.messages.MessageHdDataType;
import java.time.Instant;

/**
Expand All @@ -24,7 +25,12 @@ public record PartnerMetadata(
String hash,
PartnerMetadataStatus deliveryStatus,
String failureReason,
PartnerMetadataMessageType messageType) {
PartnerMetadataMessageType messageType,
MessageHdDataType sendingApplicationDetails,
MessageHdDataType sendingFacilityDetails,
MessageHdDataType receivingApplicationDetails,
MessageHdDataType receivingFacilityDetails,
String placerOrderNumber) {

// Below is for defaulting status when null
public PartnerMetadata {
Expand All @@ -40,7 +46,12 @@ public PartnerMetadata(
Instant timeDelivered,
String hash,
PartnerMetadataStatus deliveryStatus,
PartnerMetadataMessageType messageType) {
PartnerMetadataMessageType messageType,
MessageHdDataType sendingApplicationDetails,
MessageHdDataType sendingFacilityDetails,
MessageHdDataType receivingApplicationDetails,
MessageHdDataType receivingFacilityDetails,
String placerOrderNumber) {
this(
receivedSubmissionId,
null,
Expand All @@ -51,26 +62,42 @@ public PartnerMetadata(
hash,
deliveryStatus,
null,
messageType);
}

public PartnerMetadata(String receivedSubmissionId, String hash) {
this(receivedSubmissionId, null, null, null, null, null, hash, null, null, null);
messageType,
sendingApplicationDetails,
sendingFacilityDetails,
receivingApplicationDetails,
receivingFacilityDetails,
placerOrderNumber);
}

public PartnerMetadata(
String receivedSubmissionId, String hash, PartnerMetadataMessageType messageType) {
this(receivedSubmissionId, null, null, null, null, null, hash, null, null, messageType);
String receivedSubmissionId,
String hash,
PartnerMetadataMessageType messageType,
MessageHdDataType sendingApplicationDetails,
MessageHdDataType sendingFacilityDetails,
MessageHdDataType receivingApplicationDetails,
MessageHdDataType receivingFacilityDetails,
String placerOrderNumber) {
this(
receivedSubmissionId,
null,
null,
null,
null,
null,
hash,
null,
null,
messageType,
sendingApplicationDetails,
sendingFacilityDetails,
receivingApplicationDetails,
receivingFacilityDetails,
placerOrderNumber);
}

public PartnerMetadata(String receivedSubmissionId, PartnerMetadataStatus deliveryStatus) {
this(receivedSubmissionId, null, null, null, null, null, null, deliveryStatus, null, null);
}

public PartnerMetadata(
String receivedSubmissionId,
PartnerMetadataStatus deliveryStatus,
PartnerMetadataMessageType messageType) {
this(
receivedSubmissionId,
null,
Expand All @@ -81,7 +108,12 @@ public PartnerMetadata(
null,
deliveryStatus,
null,
messageType);
null,
null,
null,
null,
null,
null);
}

public PartnerMetadata withSentSubmissionId(String sentSubmissionId) {
Expand All @@ -95,7 +127,12 @@ public PartnerMetadata withSentSubmissionId(String sentSubmissionId) {
this.hash,
this.deliveryStatus,
this.failureReason,
this.messageType);
this.messageType,
this.sendingApplicationDetails,
this.sendingFacilityDetails,
this.receivingApplicationDetails,
this.receivingFacilityDetails,
this.placerOrderNumber);
}

public PartnerMetadata withReceiver(String receiver) {
Expand All @@ -109,7 +146,12 @@ public PartnerMetadata withReceiver(String receiver) {
this.hash,
this.deliveryStatus,
this.failureReason,
this.messageType);
this.messageType,
this.sendingApplicationDetails,
this.sendingFacilityDetails,
this.receivingApplicationDetails,
this.receivingFacilityDetails,
this.placerOrderNumber);
}

public PartnerMetadata withTimeDelivered(Instant timeDelivered) {
Expand All @@ -123,7 +165,12 @@ public PartnerMetadata withTimeDelivered(Instant timeDelivered) {
this.hash,
this.deliveryStatus,
this.failureReason,
this.messageType);
this.messageType,
this.sendingApplicationDetails,
this.sendingFacilityDetails,
this.receivingApplicationDetails,
this.receivingFacilityDetails,
this.placerOrderNumber);
}

public PartnerMetadata withDeliveryStatus(PartnerMetadataStatus deliveryStatus) {
Expand All @@ -137,7 +184,12 @@ public PartnerMetadata withDeliveryStatus(PartnerMetadataStatus deliveryStatus)
this.hash,
deliveryStatus,
this.failureReason,
this.messageType);
this.messageType,
this.sendingApplicationDetails,
this.sendingFacilityDetails,
this.receivingApplicationDetails,
this.receivingFacilityDetails,
this.placerOrderNumber);
}

public PartnerMetadata withFailureMessage(String failureMessage) {
Expand All @@ -151,6 +203,11 @@ public PartnerMetadata withFailureMessage(String failureMessage) {
this.hash,
this.deliveryStatus,
failureMessage,
this.messageType);
this.messageType,
this.sendingApplicationDetails,
this.sendingFacilityDetails,
this.receivingApplicationDetails,
this.receivingFacilityDetails,
this.placerOrderNumber);
}
}
27 changes: 24 additions & 3 deletions ...va/gov/hhs/cdc/trustedintermediary/etor/metadata/partner/PartnerMetadataOrchestrator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package gov.hhs.cdc.trustedintermediary.etor.metadata.partner;

import gov.hhs.cdc.trustedintermediary.etor.RSEndpointClient;
import gov.hhs.cdc.trustedintermediary.etor.messages.MessageHdDataType;
import gov.hhs.cdc.trustedintermediary.external.reportstream.ReportStreamEndpointClientException;
import gov.hhs.cdc.trustedintermediary.wrappers.Logger;
import gov.hhs.cdc.trustedintermediary.wrappers.formatter.Formatter;
Expand Down Expand Up @@ -35,7 +36,14 @@ public static PartnerMetadataOrchestrator getInstance() {
private PartnerMetadataOrchestrator() {}

public void updateMetadataForReceivedMessage(
String receivedSubmissionId, String messageHash, PartnerMetadataMessageType messageType)
String receivedSubmissionId,
String messageHash,
PartnerMetadataMessageType messageType,
MessageHdDataType sendingApplicationDetails,
MessageHdDataType sendingFacilityDetails,
MessageHdDataType receivingApplicationDetails,
MessageHdDataType receivingFacilityDetails,
String placerOrderNumber)
throws PartnerMetadataException {
// currently blocked by: https://github.com/CDCgov/prime-reportstream/issues/12624
// once we get the right receivedSubmissionId from RS, this method should work
Expand Down Expand Up @@ -63,7 +71,15 @@ public void updateMetadataForReceivedMessage(
"Unable to retrieve metadata from RS history API, but writing basic metadata entry anyway for received submission ID {}",
receivedSubmissionId);
PartnerMetadata partnerMetadata =
new PartnerMetadata(receivedSubmissionId, messageHash, messageType);
new PartnerMetadata(
receivedSubmissionId,
messageHash,
messageType,
sendingApplicationDetails,
sendingFacilityDetails,
receivingApplicationDetails,
receivingFacilityDetails,
placerOrderNumber);
partnerMetadataStorage.saveMetadata(partnerMetadata);

throw new PartnerMetadataException(
Expand All @@ -82,7 +98,12 @@ public void updateMetadataForReceivedMessage(
null,
messageHash,
PartnerMetadataStatus.PENDING,
messageType);
messageType,
sendingApplicationDetails,
sendingFacilityDetails,
receivingApplicationDetails,
receivingFacilityDetails,
placerOrderNumber);
partnerMetadataStorage.saveMetadata(partnerMetadata);
}

Expand Down
9 changes: 8 additions & 1 deletion etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/orders/SendOrderUseCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,14 @@ public void convertAndSend(final Order<?> order, String receivedSubmissionId)
throws UnableToSendMessageException {

sendMessageHelper.savePartnerMetadataForReceivedMessage(
receivedSubmissionId, order.hashCode(), PartnerMetadataMessageType.ORDER);
receivedSubmissionId,
order.hashCode(),
PartnerMetadataMessageType.ORDER,
order.getSendingApplicationDetails(),
order.getSendingFacilityDetails(),
order.getReceivingApplicationDetails(),
order.getReceivingFacilityDetails(),
order.getPlacerOrderNumber());

var omlOrder = converter.convertToOmlOrder(order);
metadata.put(order.getFhirResourceId(), EtorMetadataStep.ORDER_CONVERTED_TO_OML);
Expand Down
9 changes: 8 additions & 1 deletion etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/results/SendResultUseCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,14 @@ public static SendResultUseCase getInstance() {
public void convertAndSend(Result<?> result, String receivedSubmissionId)
throws UnableToSendMessageException {
sendMessageHelper.savePartnerMetadataForReceivedMessage(
receivedSubmissionId, result.hashCode(), PartnerMetadataMessageType.RESULT);
receivedSubmissionId,
result.hashCode(),
PartnerMetadataMessageType.RESULT,
result.getSendingApplicationDetails(),
result.getSendingFacilityDetails(),
result.getReceivingApplicationDetails(),
result.getReceivingFacilityDetails(),
result.getPlacerOrderNumber());

var convertedResult = converter.addEtorProcessingTag(result);
metadata.put(
Expand Down
Loading
Loading