Or is that behavior only available when the Function errors out.
This is for a Queue-triggered function.
From my testing, yes it seems that if an azure function times out the message is returned to the queue indefinitely.
Here's my message info 8 hours after it was submitted:
INSERTION TIME Mon, 09 Jul 2018 16:04:33 GMT
EXPIRATION TIME Mon, 16 Jul 2018 16:04:33 GMT
DEQUEUE COUNT 29
This MSDN page https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-error-pages says "The two triggers that have retry support are Azure Queue storage and Azure Blob storage. By default, these triggers are retried up to five times. After the fifth retry, both triggers write a message to a special poison queue."
This error behaviour does not hold true in the event of a function timeout; my message is at 29 retries and counting!