I am trying to reach https://nominatim.openstreetmap.org/ through my java application for some addresses retrieval (currently debugging inside IntelliJ).
Basically on my browser (i.e.: Chrome) and Postman I am able to reach the pointed address, while I am not able to reach it during app execution. I am also able to test succesfully the connection to the pointed URL through IntelliJ Test Connection inside File > Settings > System Settings > HTTP Proxy via checking Auto-detect proxy settings or via setting a valid proxy address (I am behind a VPN when the problem occurs).
I checked for a proxy (via looking the address through wpad URL) and set the https.proxyHost and https.proxyPort JVM options (also through System.setProperty inside the code) properly, but still I get the following error after some seconds:
org.apache.http.conn.HttpHostConnectException: Connect to nominatim.openstreetmap.org:443 [nominatim.openstreetmap.org/130.117.76.9] failed: Connection timed out: connect
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:136) ~[httpclient-4.3.4.jar:4.3.4]
at linear.skywalker.subjectupload.handlers.HttpClientHandler.executeGet(HttpClientHandler.java:173) ~[classes/:?]
at linear.skywalker.subjectupload.randomsubject.AddressGenerator.getProvinceDescriptor(AddressGenerator.java:55) ~[classes/:?]
at linear.skywalker.subjectupload.randomsubject.GenerateRandomSubject.generateRandomAddress(GenerateRandomSubject.java:206) ~[classes/:?]
at linear.skywalker.subjectupload.randomsubject.GenerateRandomSubject.generate(GenerateRandomSubject.java:87) ~[classes/:?]
at linear.skywalker.subjectupload.handlers.SubjectHandler.generateRandomSubject(SubjectHandler.java:125) ~[classes/:?]
at linear.skywalker.subjectupload.Orchestrator.mainProcedure(Orchestrator.java:138) [classes/:?]
at linear.skywalker.subjectupload.Orchestrator.orchestrateExecution(Orchestrator.java:211) [classes/:?]
at linear.skywalker.subjectupload.Starter.main(Starter.java:75) [classes/:?]
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method) ~[?:1.8.0_221]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79) ~[?:1.8.0_221]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_221]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_221]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_221]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[?:1.8.0_221]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_221]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_221]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:239) ~[httpclient-4.3.4.jar:4.3.4]
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123) ~[httpclient-4.3.4.jar:4.3.4]
... 19 more
[INFO ] 2020-12-10 17:07:03.708 [main] Orchestrator - Subject at row [1] creation finished
[INFO ] 2020-12-10 17:07:03.708 [main] Orchestrator - Finished process creation
Disconnected from the target VM, address: '127.0.0.1:57975', transport: 'socket'
Process finished with exit code 0
I am using, as you can see from the exception, Apache HttpClient that is instantiated through HttpClientBuilder.create().build().
Disconnecting from VPN makes the code working, so I suspect something is wrong with my settings, or I am missing some configurations.
The point where I get the error is the following:
HttpGet httpGet = new HttpGet(NATIM_SEARCH_BASE + "?q=" + province.replace(" ", "%20") + "+Italia"
+ "&format=json");
httpGet.setHeader("User-Agent", <timestamp>);
httpGet.setHeader("Referer", "http://www.example.com/1");
httpGet.setHeader("Host", "localhost");
return new JSONArray(httpClientHandler.executeGet(httpGet));
where executeGet just calls the execute on the HttpClient object with a pre-built ResponseHandler.
Does anybody know if I need to perform additional tuning or settings on the code?
And: is there any tutorial or working examples for this situation?
Best regards, Alberto
After some trials and investigation, I found out how to overcome this problem:
CloseableHttpClient closeableHttpClient = HttpClients.createDefault();
hostTarget = new HttpHost(targetHost, targetPort, targetScheme);
hostProxy = new HttpHost(proxyHost, proxyPort, proxyScheme);
RequestConfig config = RequestConfig.custom()
.setProxy(hostProxy)
.build();
HttpGet request = new HttpGet(requestUrl);
request.setConfig(config);
CloseableHttpResponse response = closeableHttpClient.execute(hostTarget, request);
in requestUrl we need to specify the URL portion containing the reuested resource (eg.: /search?q=Rome+Italy&format=json).
This can be also found in the official documentation example here.