I am working on analyzing H264 video data being streamed over a network. Right now, I am able to successfully extract and analyze the raw H264 for UDP. This process is going to be ALOT harder for the TCP/RTSP because of fragmentation and multiplexing.
Is the video compression / encoding any different on the TCP/RTSP multiplexed stream compared to the UDP stream?
It's only slightly harder as you typically have to demultiplex the audio and video, as well as the RTCP reports on the TCP connection. Fragmentation is not an issue.
Is the video compression / encoding any different on the TCP/RTSP multiplexed stream compared to the UDP stream?
No differences at all. The multiplexing of RTP/RTCP packets is defined in RFC2326.
As far as tools go, you can use openRTSP from http://www.live555.com which handles the transport for you (RTP over RTSP via the -t command line argument) and writes the frames to file.
With reference to Ainitak's comment, it's not that complex: there's a 4 byte header, '$' followed by the channel id, followed by the 2 byte length of the following RTP/RTCP packet. It's not too tricky to parse this.