Large HL7 messages

[AcuteParanoia]

Mirth 4.5.0. I am receiving a huge HL7 message (something like 451,074) and of course it never processes, basically get a partial message and it stops. Smaller HL7 messages are no problem. I believe I need to up memory, but I'm not sure what. The Channel Source buffer size? The administrator.maxheapsize? The -Xmx256m in the service & server files? All of the above? Something else? All thoughts appreciated!

[pacmano1]

451,074 what? Bytes?

[pacmano1]

Not sure there is a great way to calculate optimal heap size. I don't think much about it ever since it is nearly a free resource these days.

[jonbartels]

From the log -

Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.derby.iapi.services.io.DynamicByteArrayOutputStream.expandBuffer(Unknown Source) ~[derby-10.10.2.0.jar:?]
	at org.apache.derby.iapi.services.io.DynamicByteArrayOutputStream.write(Unknown Source) ~[derby-10.10.2.0.jar:?]
	at java.io.DataOutputStream.write(DataOutputStream.java:98) ~[?:?]

This is happening on a single channel thread. So your single channel is running out of memory but it is not killing your whole instance (this is good).

The interesting part of the stack trace is that this is coming from Derby, the default embedded DB with Mirth.

The BEST solution is to send smaller messages.

The next solution would be to run Mirth on a real database. I like Postgres, but SQL Server or MySQL will also work well. Running Mirth on Derby in production is not a good idea because of issues like you are seeing. It puts all of the database server load inside the Mirth runtime. Derby is quite good for testing and small production loads but nearly all Mirth users run Mirth on a real database engine.

A WORKAROUND solution would be to increase the heap memory for Mirth (-Xmx). I feel this is a workaround because I think you will run into other memory and heap related issues with messages this large.

[AcuteParanoia]

Great information. Thank you.

[AcuteParanoia]

Update: We upped the server to 16gb RAM/4 (virtual processors). Logging is on SQL Server now. I changed all -Xmx to 4096m in both mcservice.vmoptions & mcserver.vmoptions. The Administrator Launcher is a Max Heap size 2gb.

When testing with a 20,000 OBX HL7 message it's taking 30 minutes to write the CSV file out. I noticed in Task Manager it was using less than 2gb of RAM and about 25% of the CPU.

Here is a sample OBX line:
OBX|1|TX|MEDICATION||^^50^MG^000081-0597-55^^AZATHIOPRINE \T\ 50MG TAB^1^AZATHIOPRINE 1"^^^6P||||||F

...basically each piece corresponds to column in the CSV file. I've attached 1 of the destination transformers maybe I'm just doing something funky here. I'm thinking I will have to go about this another way.

Med Transform.txt

[jonbartels]

The BEST solution is to send smaller messages.

With extra memory and a solid DB you've set yourself up for success.

The intensive memory usage is because of how Mirth represents HL7 internally. Grab a smaller message and compare the Raw message to the Transformed message in the Source connector? See how Raw is pipe-delimited HL7 but Transformed is XML? Parsing the HL7 to XML is memory intensive, then managing that XML is memory intensive, and the XML is storage-intensive because of the "XML-tax" (all those tags take space).

Is it correct to think of the problem like turning the OBX segments into a CSV? If so, does it then make sense to think of the problem as turning pipe delimited text into CSV?

[AcuteParanoia]

Unfortunately I have no control or say over the message size.

Yes we are converting the OBX segments into CSV. So if I'm understanding you I should just pull it in as rawdata and switch my transformer script to basically pull apart each line. ...If that's true I assume I have to update my filter to accept the message as well. Currently I'm doing msg['OBR']['OBR.4']['OBR.4.1'].toString() equals 'MEDICATION EXTRACT'. I'm not sure how I would redo that rule. I have 7 destinations, 1 for each CSV that can be created (only 1 CSV is created per the HL7 message).

[jonbartels]

Use a regex. Something like OBR\|.*?\|.*?\|.*?\|MEDICATION EXTRACT maybe? Test it at https://regex101.com/r/gFJUV1/1

[AcuteParanoia]

I updated the channels data types to be Raw for both inbound and outbound. I replace the destination transformer with a JavaScript Writer which will break apart the pieces and save to the CSV. I setup a source transformation to create an acknowledgement when NTE is found. Works great if I send message from inside Mirth Administrator. ...but with live system it seems to never disconnect - I assume because it never gets the acknowledge back. I think I may have to create another destination with a TCP response. Is that correct or is there a setting I'm missing in the source I need to set.

I've included my script, just rename the txt to xml. The other file is just a few HL7 samples. Far short of the 450,000 OBXs I'm expecting to deal with. Just shove me in the right direction. I'm almost there LOL

DSS RXFRAMEWORK TEST 6.txt
HL7 MED2.txt

[tonygermano]

You could probably replace all of that code with

responseMap.put('TCP Response', ACKGenerator.generateAckResponse(msg, 'AA', ''))

Then, like @jonbartels said, where are you iterating over the lines to convert the OBX segments to csv rows?

[AcuteParanoia]

Jon/Tony -

I have a destination setup as a JavaScript Writer with the following code below. That is where I iterate thru lines. I need to check on the ACK. I was under the impression our system wanted more than just MSA|AA.

function replaceSpecialCharacters(inputString) {
if (inputString === undefined) {
return '';
}
// Replace '\R' with ''
const replacedString1 = inputString.replaceAll('\\R\\', '');
// Replace '\E' with ''
const replacedString2 = replacedString1.replaceAll('\\E\\', '\\');
// Replace '\F' with '|'
const replacedString3 = replacedString2.replaceAll('\\F\\', '|');
// Replace '\T' with '&'
const replacedString4 = replacedString3.replaceAll('\\T\\', '&');
// Replace '/S/' with '^'
const finalString = replacedString4.replaceAll('\\S\\', '^');
return finalString;
}

var csv = 'MED_ALIAS, CDC_RESPONSE, STRENGTH, UNITS, EXAMPLE_NDC, GPI, MED_DESC, MED_ID, MED_NAME, NOTE, RXCUI, CS_SCHEDULE\n';
var lines = connectorMessage.getRawData().split('\n');
for each (line in lines) {
if (line.startsWith('OBX')) {
var segments = line.split('\|'); // Escape the pipe character
//logger.info("segments: " + segments);
if (segments[5]) {
var subSegment5 = segments[5];

        // Split subSegment5 into pieces
        var pieces = subSegment5.split('\\^');
        
        // Place each piece into a temporary variable
        var MedAlias = pieces[0];
        var CDCResponse = pieces[1];
        var Strength = pieces[2];
        var Units = pieces[3];
        var ExampleNDC = pieces[4];
        var GPI = pieces[5];
        var MedDesc = replaceSpecialCharacters(pieces[6]);
        var MedID = pieces[7];
        var MedName = replaceSpecialCharacters(pieces[8]);
        var Note = replaceSpecialCharacters(pieces[9]);
        var RXCUI = pieces[10];
        var CSSchedule = pieces[11];
        
        // Surround each piece with double quotes
        pieces = [MedAlias, CDCResponse, Strength, Units, ExampleNDC, GPI, MedDesc, MedID, MedName, Note, RXCUI, CSSchedule].map(function(piece) {
            piece = piece ? replaceSpecialCharacters(piece) : '';
            return '\"' + piece + (piece.endsWith('"') ? '\"' : '') + '\"';
        });
        
        var newSubSegment5 = pieces.join(',');
        
        csv += newSubSegment5 + '\n'; // Add the modified subSegment5 to the CSV
    }
}

}

// Generate the filename with the current date and time
var date = new Date();
var filename = 'EMR_MEDICATIONS_' + date.getFullYear() + '-' + (date.getMonth()+1) + '-' + date.getDate() + ' ' + date.getHours() + '.' + date.getMinutes() + '.' + date.getSeconds() + '.csv';

// Write the CSV string to a file
var filePath = 'D:\WORK\MIRTH\OUTPUT\' + filename; // Your directory path
var file = new java.io.File(filePath);
var writer = new java.io.FileWriter(file);
writer.write(csv);
writer.close();

[AcuteParanoia]

Does this make sense or do you guys think I'm off on this? BTW...Thanks for your time and knowledge!

[tonygermano]

It makes sense to me. Since you are working with huge messages and trying to minimize memory usage, you may want to read the string a line at a time instead of splitting the whole thing at once into an array.

When you're done, I'd probably store the csv string as a mirth attachment rather than writing it to the file. Then you can send it with any type of destination connector you'd like.

[snarayanan99]

I had a similar requirement. I parsed the large input HL7 message as an "Attachment" and set the mcservice and administrator heap size to 8192. That seems to work and i am able to parse 40 MB messages without any issues