Failed to connect to backoff(elasticsearch 401 Unauthorized: Unauthorized)

I have recently installed opendistro elasticsearch on a dedicated host and also setting up APM server on another host.

I have the following Error response when starting up apm-server via systemctl

apm apm-server[3220]: 2021-05-14T13:42:24.221Z#011ERROR#011[publisher_pipeline_output]#011pipeline/output.go:154#011Failed to connect to backoff(elasticsearch(http://192.168.56.XXX:9200)): 401 Unauthorized: Unauthorized

I have the following 2 essential configuration in my apm-server configuration

apm-server:
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "0.0.0.0:8200"

output.elasticsearch:
  hosts: ["elasticserver:9200"]

I have checked:

indentations etc and seems to be ok
on my opendistro elastic host security configuration file /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml

I have the following section which is set to true by default, others such as LDAP, kerberos etc are set to false.

     basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 4
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

When I perform a curl command with the super user admin account from the apm server I also get the following similar error.

curl -U admin:admin http://elasticserver:9200
Unauthorizedroot@apm

This does sound like I need to create an internal user with a few backend roles (with rights to apm indices) assigned to it. I'm eager to see what y'all folks think.

I look forward to your responses, much obliged.

Solution

For basic authentication you do need to create an internal user, hash the password using the script on the master node and then map it to the "all_access" role