According to this article in TIBCO JMS undelivered message are moved by default to $sys.undelivered (DLQ). The article states:
Generally, a , sometimes referred to as an undelivered-message queue, is a holding queue for messages that cannot be delivered to their destination queues.
In TIBCO EMS, if a message expires or has exceeded the value specified by the maxRedelivery property on a queue, the server checks the message's JMS_TIBCO_PRESERVE_UNDELIVERED property.
If JMS_TIBCO_PRESERVE_UNDELIVERED is set to true, the server moves the message to the undelivered message queue, $sys.undelivered; If JMS_TIBCO_PRESERVE_UNDELIVERED is set to false, the server will delete the message.
This undelivered message queue is a system queue that is always present and cannot be deleted.
Is it possible to put automatically undelivered message of a queue in a dedicated DLQ instead of the $sys.undelivered queue?
Based on this documentation you should be able to create a selector to filter the messages sent to $sys.undelivered and route them to a different queue.