We have RESTFul Web Api using OAuth2 for authentication. Is it practical to use self signed certificate for SSL/TLS encryption? We provide .Net/Java SDK where the certificate validation error for not being a trusted CA issued will be ignored, so the customers won't see any error unless he/she tries to call the Api directly from HTTP request. This solution looks good to us but are there any risks or disadvantages?
Validation of the certificate is done to make sure that the client is talking to the expected server, i.e. authentication. Validation is the essential part of defending against man in the middle attacks, i.e. against an attacker in the path of the traffic doing sniffing and modifying the data.
Thus while you can use self-signed certificates instead of certificates signed by a publicly trusted CA you still need to properly validate the certificates and not simply disable validation. For self-signed certificates this can be done by checking if the fingerprint of the certificate or its public key matches the expectation. It would not be enough to just check the subject of the certificate because an attacker could easily create certificates with the same subject and use it inside attacks.
For more information on how to do this properly including sample code see OWASP: Certificate and Public Key Pinnning.