Search code examples
spring-bootdockerelasticsearchlogstashkibana

Cannot see Log information in Kibana running in Docker Container in my Spring Boot App (ELK)

I have an issue to show logger information Kibana running in Docker Container in my Spring Boot App. After I ran this command(docker-compose -d), all containers covering ELK ran in Docker. I opened Kibana with this url (http://localhost:5601). Then I created an index related to Logstash. I couldn't see any logger related with my project information in Kibana.

Here is the screenshot shown below.

How can I fix it?

Here is my logstash.conf file shown below.

    input {
        tcp {
                port => 5000
        }
        file {
                path => "C:/Users/{username}/IdeaProjects/SpringBootElk/Springboot-Elk.log"
                sincedb_path => "/dev/null"
                start_position => "beginning"
        }
}
output {
        stdout{
                codec => rubydebug
        }
        elasticsearch {
                hosts => "elasticsearch:9200"
        }
}

Here is logstash conf in docker-compose.yml

services:
  logstash:
    image: docker.elastic.co/logstash/logstash:7.15.2
    user: root
    command: -f /etc/logstash/conf.d/
    volumes:
      - ./elk/logstash/:/etc/logstash/conf.d/
      - ./Users/{username}/IdeaProjects/SpringBootElk/Springboot-Elk.log:/Springboot-Elk.log
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

Here is my Project File : My Project

enter image description here

Solution

  • You probably need to add another volume in your docker-compose configuration in order to map the C:/Users/... folder of your local host to the Docker container, otherwise Logstash running inside a Docker container will not be able to see your filesystem.

    volumes:
  - ./elk/logstash/:/etc/logstash/conf.d/
  - "C:/Users/{username}/IdeaProjects/SpringBootElk:/tmp/logs"

    Note: if it doesn't work for your Windows environment, you might want to check this thread.

    Then you can modify your file input like this:

    file {
        path => "/tmp/logs/*"
        sincedb_path => "/dev/null"
        start_position => "beginning"
}

    UPDATE: So you have two modes...

    1. you run the app from IntelliJ and the logs are produced on your localhost and it works
    2. you run the app from docker-compose and the logs are produced in the app's container and Logstash which runs in another container doesn't see them

    In this case, you need to share a volume between containers. Below, we define a shared volume called app-logs. The app container will produce logs in /path/to/Springboot-Elk.log (you might need to adjust the path according to your application configuration) and the logstash container will see them in /tmp/logs.

    So now your Logstash container is able to read both the logs produced by your application running on localhost and from your application running from its own container.

    services:
  logstash:
    ...
    volumes:
      - ./elk/logstash/:/etc/logstash/conf.d/
      - "C:/Users/{username}/IdeaProjects/SpringBootElk:/tmp/logs"
      - app-logs:/tmp/logs
    ...
  app:
    ...
    volumes:
      - app-logs:/path/to/Springboot-Elk.log
    ...

volumes:
  app-logs: