RabbitMQ uses a non-negative long (63-bit integer, because non-negative only) called a delivery tag to store how many messages have been sent over a channel. What happens if you send (2^63)+1 messages over a channel?
According to my back-of-napkin calculations, assuming a maximum publish rate of 53,710 messages per second, you would have to be publishing for a total of 7.06 x 10^13 years, which is four orders of magnitude greater than the age of the known universe.
Stated a different way, if we assume that somehow you could publish 3 messages per processor cycle, and that the Intel processor could process 7,000,000,000 messages per second, it would still take nearly 84 years.
Therefore, it is safe to conclude you will run into other issues first. But, if your RabbitMQ server manages to stay up and running that long, you deserve a prize.
But in all seriousness, if this were somehow to happen, I imagine it depends on how Erlang handles integers. This post indicates that Erlang will run out of memory - I don't fully understand how they accomplish that, but maybe then the whole system blows up? Who knows. In c#, the ints simply roll after overflow.