I am using Mirth Connect v3.5.2.b204 to read large comma separated value (147mb) file to convert a rows into HL7 v2.4 messages.
Source: I have configured a file reader connector type that pulls records from a directory which contains the .csv file. After processing the file should be moved to a processed file directory.
Destination: The connector type is pointed to a TCP Sender where the converted .csv records to HL7 messages are pushed.
Issue: Mirth appears to keeping reading the file even after its finished processing the last record (number 700k). Therefore the record count under received keeps incrementing and re-sending already processed messages. Please note this channel issue has not occurred on smaller files e.g. 40MB
Resolution: I stopped the mirth service, removed the .csv from source folder and restarted the mirth service. Since the file reader had no file to read the count seized to increase and conversion processing has continued.
Queries: 1. What does this error mean? 2. Are converted messages still viable? looking at samples the conversion of .csv row to HL7 is performing the correct formatting.
Error Message from Mirth Logs:
ERROR 2018-04-30 16:32:07,581 [File Reader Polling Thread on data import (342c3cbc-433b-4965-8323-c2aebff38765) < 342c3cbc-433b-4965-8323-c2aebff38765_Worker-1] com.mirth.connect.connectors.file.FileReceiver: Error reading file D:\Mirth_source\Full Data 2 2018_04_27.txt
Java heap space
ERROR 2018-04-30 16:32:07,604 [File Reader Polling Thread on Patient import (342c3cbc-433b-4965-8323-c2aebff38765) < 342c3cbc-433b-4965-8323-c2aebff38765_Worker-1] com.mirth.connect.connectors.file.FileReceiver: Error processing file in channel: 342c3cbc-433b-4965-8323-c2aebff38765
com.mirth.connect.connectors.file.FileConnectorException: Error reading file D:\Mirth_source\Full Data 2 2018_04_27.txt
Java heap space
at com.mirth.connect.connectors.file.FileReceiver.processFile(FileReceiver.java:371)
at com.mirth.connect.connectors.file.FileReceiver.processFiles(FileReceiver.java:247)
at com.mirth.connect.connectors.file.FileReceiver.poll(FileReceiver.java:203)
at com.mirth.connect.donkey.server.channel.PollConnectorJob.execute(PollConnectorJob.java:49)
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedReader.getChar(DelimitedReader.java:551)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedReader.getColumnValue(DelimitedReader.java:438)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedReader.getRecord(DelimitedReader.java:232)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedBatchAdaptor.getMessage(DelimitedBatchAdaptor.java:152)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedBatchAdaptor.getMessageFromReader(DelimitedBatchAdaptor.java:105)
at com.mirth.connect.plugins.datatypes.delimited.DelimitedBatchAdaptor.getNextMessage(DelimitedBatchAdaptor.java:96)
at com.mirth.connect.donkey.server.message.batch.BatchAdaptor.getMessage(BatchAdaptor.java:44)
at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchBatchMessage(SourceConnector.java:231)
at com.mirth.connect.donkey.server.channel.SourceConnector.dispatchBatchMessage(SourceConnector.java:196)
at com.mirth.connect.connectors.file.FileReceiver.processFile(FileReceiver.java:331)
You may want to look into using Batch Processing. You can enable it by setting Process Batch to Yes on the source settings.
This will cause the channel to read each record from the CSV in as a separate message, one at a time. When using a File Reader, it also means that the entire file will not be read into memory all at once.