How an RSA key literally is stored using the formats PKCS#1 and PKCS#8? What is the difference between the PKCS formats vs encodings (DER, PEM)? From what I understand, PEM is more human readable. Is PEM/DER for keys/certs similar to UTF-8/16 for characters? What is the significance of DER/PEM?
PKCS#1 and PKCS#8 (Public-Key Cryptography Standard) are standards that govern the use of particular cryptographic primitives, padding, etc. Both define file formats that are used to store keys, certificates, and other relevant information.
PEM (Privacy-Enhanced Mail) and DER (Distinguished Encoding Rules) are a little bit more interesting. DER is the ASN.1 encoding for keys and certificates etc., which you'll be able to Google plenty about. Private keys and certificates are encoded using DER and can be saved directly like this. However, these files are binary and can't be copied and pasted easily, so many (if not most?) implementations accept PEM encoded files also. PEM is basically base64 encoded DER: we add a header, optional meta-data, and the base64 encoded DER data and we have a PEM file.