Working on Storm Crawler 1.12.1 and Elastic search 6.5.x. My Crawler is running on http://localhost:8080 and Elastic search running on https://localhost:9200.I am trying to crawl a website.During URL injection process I am facing javax.net.ssl.SSLHandshakeException: General SSLEngine problem error and check the detailed error here.
I tried for OKHttp and added https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol" in crawler-conf.yaml.
How can I turn off the certificate verification temporarily.
As @julien-nioche indicated in his comment: StormCrawler does not fetch in the process of URL injection.
The SSLHandshakeException most likely originates from your ElasticSearch instance. Is your ElasticSearch really running on HTTPS as indicated by https://localhost:9200?
The stacktrace indicates that the certificate path could not be validated.
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
To resolve this issue, you could add the certificate to your local key-store. For details on this subject, you can take a look at this answer. Another possibility would be to disable HTTPS on the ElasticSearch instance for your local development setup.