I'm interested in Java-NSS libraries, and I'm reading the Sun's P11 Guide. I am confused on the following:
What is the difference between using a PKCS12 keystore and a PKCS11 keystore?
A keystore is just a keystore, right? Are there some differences? Can they be used interchangeably in any aspect?
PKCS#12 is a file format (often called .p12 or .pfx) where you can store a private key and certificates. It's used for converting/transporting keys and certificates, mainly. If you export a private key + certificate from your browser, it's likely going to be in that format.
PKCS#11 is an interface, usually used to talk to hardware cryptographic tokens (often smart-cards or USB-tokens, which effectively are smart-cards embedded in a reader). This interface has a number of operations to make use of the keys and certificates. Some tokens are able to sign using the private key they contain, without the key being able to leave the device. The point of this interface is to treat what handles the keys and certificates as a separate entity, without having to do the cryptographic operations that PKCS#11 offer (more specifically, the ones related to the private key).
When you use PKCS#11 with NSS, you're effectively using NSS as a black-box wrapped behind the PKCS#11 layer (it's effectively a software provider for what a PKCS#11 hardware token would be). There is a slight difference in the way Java uses NSS via PKCS#11 in that it doesn't require a PKCS#11 shared library (compared to other PKCS#11 libraries), so as such, it's not PKCS#11 strictly speaking, although it's very similar.
In Java, you may be able to get an RSAPrivateKey instance from a PKCS#11 store, use it to sign and decipher, without ever being able to get anything from its modulus. The security provider handling it will do the signing/deciphering via the library (and thus via the token, if that library is supported by a hardware token).
Coming back to the KeyStore in Java, it's an API that can allow you to load and use keys and certificates from files (you get various files formats such as JKS, PKCS#12, PEM, depending on your security provider) or from other underlying APIs (such as PKCS#11, more or less merged with NSS in the Sun provider, or the KeychainStore if you're on OSX and want to use the KeyChain as a KeyStore).