I am writing a connection pool using Executors. My problem is that when the database goes down, all the connections in pool become invalid connections.
One way would be to refresh the connection pool at a regular interval, or to check the validity of the connections. What is the best way to handle this type of problem?
First of all, if you don't absolutely, definitely, no way around, HAVE TO write your own, this is real typical case of reinventing a perfectly fine wheel. There are lots of fine and dandy connection pool implementations available on the interwebz.
I (among many others) use Apache DBCP: http://commons.apache.org/dbcp/
If you have some kind of uber-special database, you could at least rely on an existing pooling solution like Commons Pool: http://commons.apache.org/pool/
You know, using apache stuff allows you to sit with the cool kids ;)
Well then, if you really have to write your own I'd suggest this:
Whenever a connection is requested, check it for validity. No way arround this. If it's not valid anymore, either immediately drop all connections in the pool (assuming all of them are invalid), or just get the next one and just drop the one you checked. Repeat until you find a valid one or you have to grow the pool. Growing the pool will need to check for validity and abort if new connections can't be established (obviously). You would have some fine infinity-looping on your hand otherwise.
A secondary thread who checks on the pool and cleans it up and/or grows it as needed seems fine too. Just don't rely on it alone as it may very well miss an invalid connection and you would still serve unusable pool elements.
The thread (or threads) may only serve to speed things up (checking/growing the pool), but are not suited to do the job for themselves. You will have to check/grow on request, secondary threads only assist in avoiding that.