I have done some research on this but still have some trouble connecting the dots as to what exactly happens when the server send its certificate to the client in terms of verifying signature and root certificates.
When you create a CSR, it generates a private key along with it, and you send it to the CA to generate the cert and sign it, but doesn't the CA use it's own private key to sign? So what is the point of the private key you generated with your CSR?
When server sends its certificate for client to validate, how exactly does client validate that it is a valid CA cert. It has a collection of trusted CA certs, ok - but how exactly are they used to verify that it was a valid CA that signed the server's certificate using the signature and public key of the server certificate? What things are compared to make sure it was not forged?
Is there any point in encrypting your internal self signed certs? How about an internal root cert? Or is the private key the only one worth encrypting?
If we don't keep a database of encrypted data for our web service (over SSL) for example, would we ever care about storing our own private key once we generated the self signed cert, and if we do, they why?
When you create a CSR, it generates a private key along with it
Or you have already generated your own private key.
and you send it to the CA to generate the cert and sign it
You send the CSR. You don't send your private key. It's private. You don't send it to anyone.
but doesn't the CA use it's own private key to sign?
Yes.
So what is the point of the private key you generated with your CSR?
It pairs with the public key contained in the certificate and it is part of the process used to prove that you and only you own that certificate, as only you can generate digital signatures with that private key that can be verified by the public key in the certificate.
When server sends its certificate for client to validate, how exactly does client validate that it is a valid CA cert. It has a collection of trusted CA certs, ok - but how exactly are they used to verify that it was a valid CA that signed the server's certificate using the signature and public key of the server certificate? What things are compared to make sure it was not forged?
The certificate itself is verified, by verifying its digital signature; it is checked for being within its validity period; and then an attempt is made to form a certificate chain using the alleged signer of the certificate (the 'issuer') and the trusted certificates in the collection.
Is there any point in encrypting your internal self signed certs?
No. They are public documents. Only the private key is private, and that isn't in the certificate.
How about an internal root cert?
No.
Or is the private key the only one worth encrypting?
Yes.
If we don't keep a database of encrypted data for our web service (over SSL) for example, would we ever care about storing our own private key once we generated the self signed cert, and if we do, they why?
Because it's private. It is a critical part of your identity. If you leak it, others can impersonate you.