I am looking for better tools than Wireshark for this. The problem with Wireshark is that it does not format the data layer (which is the only part I am looking at) cleanly for me to compare the different packets and attempt to understand the third party encoding (which is closed source).
Specifically, what are some good tools for viewing data, and not TCP/UDP header information? Particularly, a tool that formats the data for comparison.
To be very specific: I would like a program that compares multiple (not just 2) files in hex.
Your best bet, frankly, is to roll your own.
Get a scripting language that you're comfortable with and start hacking away at the problem. First write up a simple multiway comparison, but as soon as you start finding patterns that you think are significant (or that you think might be) go back and add them to the code--eliminate them from the output, highlight them, translate them into another notation, replace them with their "meaning" or some high level description of their role--whatever seems appropriate. If you can't decide, make it an option.
You are asking for visualization software because you want something to help you form and internalize an understanding of their encoding. But the comparisons you are going to make are only part of the process you'll be using (which is essentially the scientific method)--you'll also be forming and revising guesses about what various parts of the packets mean, how they interact, etc.
No pre-built tool is going to help you there, but a good scripting language (say python, ruby, or even perl) will help a lot. As you form a theory, code it up and try it out. Mungle your code around, trying out different ideas as they occur to you, building up a bag of tricks customized to this problem as you go along.
-- MarkusQ
P.S. Don't fall into the trap of trying to use C or Java or something to do this. You'll be playing fast and loose, and should have a tool that doesn't require variable declarations, compilation, etc. There will be plenty of opportunity to tighten it up and rewrite it once you understand how it works.