Search code examples
elasticsearchlogstashelastic-cloud

Logstash Elastic Cloud 401 Unauthorized error


Official logstash elastic cloud module

Official doc for starting with

My logstash.yml looks like:

  cloud.id: "Test:testkey"
  cloud.auth: "elastic:password"

With 2 spaces in front and no space at end, within ""

This is all I have in logstash.yml and nothing else,
And I am getting:

[2018-08-29T12:33:52,112][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://myserverurl:12345/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'https://myserverurl:12345/'"}

And the my_config_file_name.conf looks like:

input{jdbc{...jdbc here... This works, as I see data in windows console}}
output {
    stdout { codec => json_lines }
    elasticsearch {
    hosts => ["myserverurl:12345"]
    index => "my_index"
    # document_id => "%{brand}"
    }

What I am doing is hitting bin/logstash on windows cmd,
It loads data from database that I have configured in input of conf file and then shows me error, I want to index my data from MySQL to elasticsearch on Cloud, I took 14 days trial and created a test index, for learning purpose as I later have to deploy it. My Pipeline looks like:

  - pipeline.id: my_id
    path.config: "./config/conf_file_name.conf"
    pipeline.workers: 1

If logs won't include senistive data, I can also provide them.
Basically I wan't to sync (schedule check) my MYSQL data with ElasticSearch on cloud i.e. AWS


Solution

  • The output shall be:

    elasticsearch {
        hosts => ["https://yourhost:yourport/"]
        user => "elastic"
        password => "password"
        # protocol => https
        # port => "yourport"
        index => "test_index"
        # document_id => "%{table_id}"
    

    # - represent comments

    as stated at: Configuring logstash with elastic cloud docs

    The document provided while deploying app does not provide config for jdbc, jdbc as well need user and password even if defined in settings file i.e. logstash.yml