In OTR messaging the encryption key is published after an encrypted message has been read by the recipient, so there is apparently no non-repudiation. In RSA the private key is obviously not published, but nobody can prove that it was not compromised in the past. Alice can always deny having sent and signed a particular message because of this.
I don't understand why RSA can legally prove who sent a message but OTR prevents this.
RSA can't prove that Alice sent something, but an RSA signature that was done with Alice's key can be used by a third party to find out that something was indeed signed by Alice's key. Sure, she can still say "Someone must have stolen my key, I never signed that!", but it might not be very plausible. What OTR offers is plausible deniability: Everybody knows that the signature could have been done by anyone, so it's plausible that the messages could be crafted by a malicious party.