PrivilegedActionException trying to invoke a signed Java applet from JavaScript

First off, I am not a Java developer. I had to create a Java applet to invoke some code in a native DLL I wrote from the browser.

I use JNA to load a native DLL and invoke its methods.
I have signed the applet using a self-signed certificate.
The browser asks me whether or not to allow the execution of the applet.
The applet code which loads my DLL is enclosed within AccessController.doPrivileged block.

Like this:

public String Test()
{
    pHelper = AccessController.doPrivileged(new PrivilegedAction<IHelper>() 
    {
        @Override
        public IHelper run() 
        {
            return (IHelper)Native.loadLibrary("Helper", IHelper.class);
        }
    });

    return "test";
}

The code works fine when debugged inside Eclipse.

It does not work when invoked from JavaScript. Causes PrivilegedActionException.

If I remove the entire AccessController.doPrivileged block and leave return "test" only, the code runs when invoked from JavaScript. Any code that doesn't require privileges runs fine when invoked from JavaScript.

Tested from Chrome version 40.something and Firefox 36 on Windows 8.1 64-bit. The native DLL is 32-bit as well as JRE used to run the applet.

Any tips?

Solution

I have never resolved this particular mystery. However, I was able to find a workaround, thanks to my applet design specification which doesn't require exposing any applet methods which need to be invoked to perform privileged operations.

I have found that executing privileged operations inside the applet init() function will work. Only privileged operations executed by invoking from JavaScript seem to cause problems. Consider the following code.

public class MyApplet extends JApplet {
    private IHelper pHelper = null;
    private MyReturnedInfo pInfo = null;

    public void init() {
        pHelper = (IHelper)Native.loadLibrary("Helper", IHelper.class);
        if (pHelper != null) {
            pInfo = pHelper.GetInfo();
        }
    }

    public String GetInfoString() {
        if (pInfo != null) {
            // need to call toString to convert from native wide char to something JavaScript will be able to interpret
            return pInfo.MyInfoString.toString(); 
        }
        return null;
    }
}

Upon loading this applet, calling document.myApplet.GetInfoString() from JavaScript (providing the applet has an ID "myApplet") will return the required information.

Interestingly though, after signing the applet with a certificate issued by a trusted authority such as VeriSign, even this would not work in IE, while it would work properly in FF and Chrome. I have seen signed Java applets which work fine when called from JavaScript in IE, but I guess my applet is special because it requires all-permissions attribute in the manifest and IE probably doesn't like that. It's a guess. However, I have never found the real reason for that either, because I was able to resort to another workaround. :) If you are reading this answer then I bet you are interested in it as well.

Java applets allow us to provide additional parameters which we are able to obtain by calling this.getParameter() from inside the init() function. Also, if we allow the applet to call JavaScript functions from our HTML document by using mayscript attribute, we can easily combine these two facts to provide the JavaScript function for applet to call after the information from our native DLL has been obtained.

Let's say that in our HTML, we define the JavaScript like this.

<script type="text/javascript" src="https://www.java.com/js/deployJava.js"></script>
<script type="text/javascript">
    var attributes = {
        id: "myApplet",
        name: "myApplet",
        code: "MyApplet.class",
        mayscript: "true",
        scriptable: "true",
        archive: "/path(s)/to/jar(s)",
        width: 0,
        height: 0
   };

    var params = {
        "AppletReady": "appletInitialized",
    };

    // For convenience, it's easier to deploy the applet using deployJava,
    // so it writes the applet HTML tag for us after checking if Java is installed.
    // We have included it above.
    deployJava.runApplet(attributes, params, "1.8.0");

    function appletInitialized(myString, someOtherArgument) {
        // do something with your parameters
        // NOTE: do NOT call alert() from this function! 
        // Because it will most likely cause your browser to freeze, 
        // I've found that's also one of the things Java doesn't like.
    };
</script>

Then, we modify the Java applet code to look like this.

public class MyApplet extends JApplet {
    private IHelper pHelper = null;
    private MyReturnedInfo pInfo = null;

    public void init() {
        // Read the AppletReady parameter as passed from JavaScript
        String paramKey = "AppletReady";
        String jsLoadedCallback = this.getParameter(paramKey);

        // Load the library and get the information
        pHelper = (IHelper)Native.loadLibrary("Helper", IHelper.class);
        if (pHelper != null) {
            pInfo = pHelper.GetInfo();
            if (pInfo != null && jsLoadedCallback != null) {
                // Get the window which contains "this" applet
                JSObject jsObject = JSObject.getWindow(this);

                // Call the provided JavaScript function.
                // You can use as many parameters as you need.
                jsObject.call(jsLoadedCallback, new Object[] {
                        pInfo.MyInfoString.toString(),
                        pInfo.SomeOtherStringMaybe.toString()
                });
            }
        }
    }
}

However, if you need the applet to call your native DLL methods dynamically during runtime (I.E. you require the applet to expose functions which need to be called to perform privileged operations dynamically) this solution will not work for you and you are out of luck, at least if using JNA.