Sometimes, after server restart all applications fail because of missing ExtLib libraries. They should be installed during HTTP startup from NSF update site. Server log shows:
HTTP JVM: CLFAD0330I: NSF Based plugins are being installed in the OSGi runtime. For more information please consult the log
HTTP JVM: null. For more detailed information, please consult error-log-0.xml located in /data/domino/workspace/logs
... repeated ~30x
Content of the error-log-0.xml is no help - fails show only as empty line. Thankfully there is another log file startup.log containing stack trace of the problem:
!ENTRY org.eclipse.osgi 4 0 2015-04-24 09:20:02.450 !MESSAGE Error installing bundle: osginsf:update/extlib.nsf/E7D8A0430DEAFDF2C1257B4F003FF202/com.ibm.xsp.extlib.core_8.5.3.20130315-0724.jar !STACK 0 java.io.IOException at com.ibm.domino.internal.protocol.osginsf.OSGiNSFUrlConnection.getInputStream(OSGiNSFUrlConnection.java:133) at java.net.URL.openStream(URL.java:1022) at org.eclipse.core.runtime.adaptor.EclipseStarter.installBundles(EclipseStarter.java:1105) at org.eclipse.core.runtime.adaptor.EclipseStarter.loadBasicBundles(EclipseStarter.java:646) at org.eclipse.core.runtime.adaptor.EclipseStarter.access$1(EclipseStarter.java:623) at org.eclipse.core.runtime.adaptor.EclipseStarter$1.bundleChanged(EclipseStarter.java:308) at
... stripped ...
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at com.ibm.domino.http.bootstrap.osgi.LaunchRCP.startRCP(LaunchRCP.java:342) at com.ibm.domino.http.bootstrap.osgi.LaunchRCP.launchOSGIFramework(LaunchRCP.java:172) at com.ibm.domino.http.bootstrap.BootstrapOSGIClassLoader.launchOSGIFramework(BootstrapOSGIClassLoader.java:79) at com.ibm.domino.http.bootstrap.BootstrapOSGIClassLoader.loadClass(BootstrapOSGIClassLoader.java:63) at java.lang.ClassLoader.loadClass(ClassLoader.java:638) at com.ibm.domino.http.bootstrap.BootstrapClassLoader.findClass(BootstrapClassLoader.java:79)
!ENTRY org.eclipse.update.configurator 4 0 2015-04-24 09:20:02.762
!MESSAGE D:\Lotus\Domino\data\domino\workspace.config\domino\eclipse\plugins\plugins is not a valid plugins directory.
What catched my eye is the last line - the mentioned plugin directory does not exist (one extra '/plugins' directory).
The only workaround is to recreate update site from scratch (delete all content and reimport local update site).
BTW, I have found very old discussion, with no solution on ExtLib project site.
What is the cause of failed update? I can't tell whether IO exception is caused by wrong file path or dead URL to update site (network misconfiguration).
Problem was solved by increasing disk size for Domino binaries - it went out of space (and nobody noticed) by growth of Windows directory (Windows updates). Data and translog directories use different drives, so server was running with no "out of space" errors.
It seems IO Exception was thrown during unsuccessful disk write.