I have developed a web service for Portal 7.0 and deployed the application into an environment. The installation and start up of the application is fine, however, when the web service is invoked, the following error appears in the SystemOut.log a few hundred times. The server then starts reporting other errors about 'Too many open files' until the application is stopped or the services restarted.
[7/29/14 8:45:56:647 EST] 00000057 WSClassFinder E WSWS7058E: A JAXBContext could not be created correctly because of the following error: java.util.zip.ZipException: error in opening zip file
[7/29/14 8:45:56:647 EST] 00000057 WSClassFinder E WSWS7058E: A JAXBContext could not be created correctly because of the following error: java.util.zip.ZipException: error in opening zip file
Strange thing is we have a development and production environment that are basically identical (Same Linux OS version + WAS and Portal 7.0.0.23). Initially the web service worked fine in dev, but an attempt to use it in production failed (Not a good look). After rebooting dev, the problem started to occur there as well (at least it is now consistent).
I also tried to build the most basic of basic web service applications (it just returns a single static string) and even that failed with exactly the same result when invoked. I have a separate development environment where I do most of my work which is separate to the one above, but with the same app versions. This one has always worked fine. This suggests it is a configuration or server issue.
I have used tracing on the com.ibm.ws.classloader and java.util.zip without any very useful information. Class loader didn't seem to clearly indicate the root of the problem and java.util.zip didn't report anything in the trace.
Unfortunately I'm stuck with where to look for options to correct this. It could be an OS file permissions problem, but its not clear where this may be. There is an upgrade of this environment planned, but because it was previously working on this build version (on the dev server) I'd really like to get to the root cause before pursuing the upgrade.
Any suggestions or pointers would be most appreciated.
The suggestions about the open file limit restrictions were right all along, we were just looking in the wrong place. The 'default' ulimit settings were valid, but these are not the ones that are applied when the WebSphere Portal server is started via an init.d script. The very last section in this article identified the issue we were experiencing:
http://www-01.ibm.com/support/docview.wss?uid=swg21469413
We verified this was the case by using the details in this article:
http://www-01.ibm.com/support/docview.wss?rs=0&uid=swg21407889
All that was required was the update the init.d script to include the line 'ulimit -n xxxxx' with our required limit. We had used the WASService scripts to generate the init.d script and thought that it shouldn't need further modification. Maybe we missed a point in the use of the WASService script that handles this.