I've used URLconnection.connect and BufferedInputStream(connection.getInputStream()) to download pages, but unfortunately it seems it is unable to download any 404 pages or any other error-message pages, instead it throws an exception. Is there a way to change this behavior and get the error message that is displayed in a browser?
You are right. The implementation of HttpURLConnection throws exception when any http status except 2xx series is returned.
You should use Jakarta HTTP Client to be able to get access to input stream (and therefore download any content) when error status is returned.