How to run container of beat that required authentication from Elasticsearch

Question

The main purpose: I want to use Logstash for collecting logs files that rely on remote server.

My ELK stack were created by using docker-compose.yml

version: '3.3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.1
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - '/share/elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro'
    environment:
      ES_JAVA_OPTS: "-Xmx512m -Xms256m"
      ELASTIC_PASSWORD: changeme
      discovery.type: single-node
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1
  logstash:
    image: docker.elastic.co/logstash/logstash:7.5.1
    ports:
      - "5000:5000"
      - "9600:9600"
    volumes:
      - '/share/elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro'
      - '/share/elk/logstash/pipeline/logstash.conf:/usr/share/logstash/pipeline/logstash.conf:ro'
    environment:
      LS_JAVA_OPTS: "-Xmx512m -Xms256m"
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1
      
  kibana:
    image: docker.elastic.co/kibana/kibana:7.5.1
    ports:
      - "5601:5601"
    volumes:
      - '/share/elk/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:ro'
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1

networks:
  elk:
    driver: overlay

and then I want to install a filebeat at the target host in order to send log to the ELK host.

docker run docker.elastic.co/beats/filebeat-oss:7.5.1 setup \
-E setup.kibana.host=x.x.x.x:5601  \
-E ELASTIC_PASSWORD="changeme" \
-E output.elasticsearch.hosts=["x.x.x.x:9200"]

but once hit the enter, the error occurs

Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://x.x.x.x:9200: 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}]

Also tried with -E ELASTICS_USERNAME="elastic" the error still persists

Answer 1

You should disable the basic x-pack security which is by default enabled in Elasticsearch 7.X version, under environment variable of ES docker image, mentioned below and start ES docker container.

xpack.security.enabled : false

After this, no need to pass ES creds and you can also remove below from your ES env. var:

ELASTIC_PASSWORD: changeme