Logstash can't index into ES: `action [indices:admin/auto_create] is unauthorized for user`

Question

I created a user demoTester with the following roles:

/usr/share/elasticsearch/bin/elasticsearch-users useradd demoTester -p demoTester -r kibana_admin,logstash_admin,beats_admin,logstash_system,monitoring_user,watcher_admin,editor,machine_learning_admin

When I run my deployment script, I can see that Logstash is listening on port 5044 and the logs are being sent, but the user demoTester can't index into ES. I have read the documentation on how to create privileges, but the examples are not clear to me. I am not creating via the Kibana UI, I am automating everything through a script.

error=>{"type"=>"security_exception", "reason"=>"action [indices:admin/auto_create] is unauthorized for user [demotester] with roles [watcher_admin,editor,monitoring_user,logstash_system,beats_admin,machine_learning_admin,kibana_admin,logstash_admin] on indices [demo-2022.10.27], this action is granted by the index privileges [auto_configure,create_index,manage,all]"}}

Here's my logstash conf file:

input {
  beats {
    port => 5044
  }
}
output {
    elasticsearch {
      ssl => true
      ssl_certificate_verification => true
      cacert => '/etc/logstash/certs/http_ca.crt'
      user => demoTester
      password => demoTester
      hosts => ["https://10.0.8.19:9200"]
      index =>"demo-%{+YYYY.MM.dd}"
    }
}

Answer 1

The demoTester user does not have the create_index privilege for the demo-2022.10.27 index.

The easy way is to add the role superuser to the demoTester user but use that is only for demo purposes.

The secure way is to create a role with create_index privilege for the demo* indices and assign that role to your demoTester user.

To create the role you can call the _security/role api (https://www.elastic.co/guide/en/elasticsearch/reference/8.4/security-api-put-role.html) with the re

POST /_security/role/my_admin_role
{
    "indices" : [
      {
        "names" : [
          "demo*"
        ],
        "privileges" : [
          "create_index",
          "write",
          "create"
        ],
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
        "enabled" : true
    }
}

and after that assign the role to de demoTester user.