I have an existing Java web application running on Linux using embedded Jetty. The application is loaded using jsvc, which runs as root, listens on port 443, and relays requests to the Java app, which is run under a less privileged user "appname" on port 8443.
Currently, the application reads an encryption key from a file we'll call "secrets.properties". It is writable by "root", and readable by "appname" (technically, by members of the "appname" group). My preference, however, would be that the file is only readable by "root", and that jsvc reads the file and passes the contents of that file (or even just a single property) to the application. My goal is that if someone were able to subvert the app and gain system access with the app's "appname" account, that they couldn't retrieve the key.
Is that possible without having the key visible to someone running "ps -ex"?
They may not be able to retrieve the key, but they would still be able to use it
Since the attacker now controls the application, they could send any messages they want and whatever is signing/encrypting them would, believing that the application was uncompromised , dutifully sign or encrypt them. There is little point in doing this if you are merely trying to authenticate data coming from the application.
No matter what, you'd be smart to revoke your keys if the server was compromised, so it doesn't save you their either.
Now, if your worried about the compromise of a long lived key allowing an attacker to read previously sent (but no longer stored in the app) messages , thats a different problem. I would suggest using a crypto system with perfect forward secrecy. TLS/SSL has this if you use authenticated Diffie-Helmen key exchange which are the schemes with ECDHE or DHE in them(e.g. ECDHE-ECDSA-AES128-GCM-SHA256 or TLS-DHE-RSA-with-AES-256-CBC-SHA)
If for some reason your still worried about keys being compromised, you could write a proxy. It either could run as root or run as a different group and have the key handed to it. For TLS/SSL this is easy. For some custom encryption/signing system, who knows . But again, there is very little reason to want to do this.