In this document: http://download.oracle.com/javase/6/docs/technotes/guides/standards/ it is describing only a single non programmatic way to override endorsed standards in Java (e.g. to provide a different TransformerFactory implementation etc...). by putting the jars in <java-home>\lib\endorsed
Is there a way to spefify an endorsed jar programmatically? (looking at Tomcat's endorsed dir, I assume it is possible, after all Tomcat is written in Java)
Tomcat also just sets the system property when starting the JVM, so uses the regular / documented way, as documented in the classloader 'how-to' ("Tomcat utilizes this mechanism by including the system property setting -Djava.endorsed.dirs=$JAVA_ENDORSED_DIRS in the command line that starts the container.")
Therefore, it's not so easy to do it programmatically. I assume setting the system property after the JVM has been started has no effect (I didn't test this).
However, it is still possible: you just have to start another JVM from within your application, and set the system property there. Your application would have to first check if the endorsed dir is set, if yes run normally, and if not: use Runtime.getRuntime().exec(cmdarray) to 'start itself again' (with the system property set correctly of course). This 'start itself' is a bit complicated to get right, but it's possible. The company I work for (www.day.com) uses it for all products, we call it 'quickstart'. I believe we don't actually set the endorsed dir, but we set the max memory (-Xmx..) if the current setting isn't sufficient.