I'm after some concrete advice on how best to prevent (or at least deter) unauthorised software upgrades on an embedded system. It doesn't need to be bullet-proof and we can assume for now that the system itself is sufficiently locked down so that no-one can get unauthorised access to it.
My plan is to basically have an installer process running on the system which would receive update packages from anywhere but that could ensure those packages came from a trusted source (i.e., me) before attempting to install them.
In simple form, the update package would have the actual installation package, plus a matching digital signature that could only be generated by myself. Moreover, the signatures would be purely self-generated with no external authorities involved.
So these are my thoughts on the possible process:
Generate a private/public key pair and distribute the public key along with the embedded system itself.
When creating a software install package, pipe the contents of the package (or an MD5 of the package) through a signature generator using our private key.
Distribute the software install package along with that signature.
Have the installer check the signature against the software install package (using the public key it already has) and only install if there's a match.
If anyone can find any problems with this scheme, I'd appreciate the details, along with any specific advice on how to avoid them. In addition (though this is not the primary purpose of the question), any advice on tools to generate the keys would be appreciated.
I do not see any apparent problems with your solution. I can suggest improvements that you may have already taken into account
If the embedded software is sufficiently locked, it is not necessary to take additional measures to protect the integrity of the public key distributed with the software (e.g. by signing the installer itself and obfuscate, that could be a headache)
I've considered a TLS connection to download the updates, but it would not really needed, because packages are going to be protected with a digital signature
I suggest encapsulating the public key in an X509 certificate. This way you can control the period of validity and even a possible revocation if the private key has been compromised. In this case you will need a hierarchical Certificate Authority, with a root certificate that issues the signing certificates. Include in the truststore of the installer the public part of the root certificate. Then using a different signing certificate after expiration/revocation will be transparent to installer.
The root certificate has a long duration and a large key size (and should be conveniently secured), and the signing certificates have a shorter duration and can use a smaller key.
With this CA you could also generate a TLS certificate if you need some additional service: e.g check available updates. In this case include the certificate in the truststore of the installer to avoid man-in-the-middle attacks (SSL-pinning).
You can sign the full distribution or a hash. It does not affect security (see https://crypto.stackexchange.com/questions/6335/is-signing-a-hash-instead-of-the-full-data-considered-secure) but do not use MD5 because has extensive vulnerabilities. Use a SHA-2 function.
To generate the keys you can use openssl in command line or use the GUI application KeyStore-Explorer