We have to create a responder application as a windows service which will take messages off the queue, acknowledge the sender that we received them and that they are validated by sending a response back within 5 seconds. After we get another acknowledgement from the sender that we are okay to process the message, we process the message and send the result of that request back to them within another (5 sec). If at any point in that cycle either they or we do not get the message off the queue and respond within the time allotted the message will expire and we will need to re-try the message.
What we are having trouble with determining when messages will expire and need to be re-sent. I've been reading about putting them on the dead-letter queue, but I'm unsure that the purpose of it matches what we need. The dead-letter queue is not for items which just timeout, but failed in transmission. I've also read about using the Report option on the MQMessage itself to generate a report and possibly move it to a different queue which can be monitored. We'd then need to re-try the items that are on the new queue.
Are those my only two options if I need to monitor every incoming and outgoing message to ensure that we get a response from the other MQ server we are communicating with?
I'd do it with message expiry and activity reporting in MQ, but that doesn't make it unnecessary for your applications to have a log of what they sent and received.
On every message you send, you could set an expiry time (meaning that the message will be discarded, if it is removed from the queue outside of that time frame). And by also setting the reporting option for getting reports when the message is discarded, the sender of the message (actually you can specify any queue for the reports not just one monitored by the sender) will get a report message identifying the discarded message, which can then be resent.
However the discarded message won't be put to any queue, the sending application needs to be able to resend the message from scratch.
And the processing side, when it's waiting on the 2nd message, it will need to have it's own "timer" for handling timeouts.
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ref.dev.doc/q097490_.htm
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.mon.doc/q036600_.htm
All that said, it seems to me that you are not using MQ as intended and are trying to mimic a synchronous communication. MQ can gurantee exactly once delivery, so I'd say that if you have a properly configured and sized MQ route between your applications and properly set up message monitoring, then all this ack/nack is unnecessary.