Please don't say bonjour. The best guess I can gather is XMPP, but I don't think thats right
iChat does use Bonjour (which is a multicast-DNS based service discovery protocol) to find other nodes when they're on the same LAN. For remote chat it uses XMPP.
It uses SIP (and RTP?) for the actual chat and audio / video sessions.