Search code examples
spring-bootdockerspring-cloudnetflix-eurekaservice-discovery

Spring Boot Actuator shows status down for microservices

I am trying do add Eureka Service for my microservices however even if microservices are running and messaging is working properly Eureka cannot discover service because they are down? This it the output of http://localhost:8080/actuator/health

{
  "status": "DOWN",
  "components": {
    "binders": {
      "status": "UP",
      "components": {
        "rabbit": {
          "status": "UP",
          "details": {
            "version": "3.7.8"
          }
        }
      }
    },
    "coreServices": {
      "status": "DOWN",
      "details": {
        "movie": {
          "status": "DOWN",
          "details": {
            "error": "io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: movie/172.21.0.8:80"
          }
        },
        "reviews": {
          "status": "DOWN",
          "details": {
            "error": "io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: review/172.21.0.9:80"
          }
        },
        "recommendations": {
          "status": "DOWN",
          "details": {
            "error": "io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: recommendation/172.21.0.6:80"
          }
        }
      }
    },
    "discoveryComposite": {
      "description": "Discovery Client not initialized",
      "status": "UNKNOWN",
      "components": {
        "discoveryClient": {
          "description": "Discovery Client not initialized",
          "status": "UNKNOWN"
        }
      }
    },
    "diskSpace": {
      "status": "UP",
      "details": {
        "total": 84014424064,
        "free": 28400906240,
        "threshold": 10485760
      }
    },
    "ping": {
      "status": "UP"
    },
    "rabbit": {
      "status": "UP",
      "details": {
        "version": "3.7.8"
      }
    },
    "reactiveDiscoveryClients": {
      "description": "Discovery Client not initialized",
      "status": "UNKNOWN",
      "components": {
        "Simple Reactive Discovery Client": {
          "description": "Discovery Client not initialized",
          "status": "UNKNOWN"
        }
      }
    }
  }
}

Why is that? by running docker ps all of them are available:

(base) mat@mat-Vostro-5471:~/Projects/movie-rest-services$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                    PORTS                                                                                        NAMES
376da889ce8c        movie-rest-services_review            "java -jar /app.jar"     23 minutes ago      Up 23 minutes             8080/tcp                                                                                     movie-rest-services_review_1
f4fe8c2668f2        movie-rest-services_movie             "java -jar /app.jar"     23 minutes ago      Up 23 minutes             8080/tcp                                                                                     movie-rest-services_movie_1
b2fa74bdb3ea        movie-rest-services_movie-composite   "java -jar /app.jar"     23 minutes ago      Up 23 minutes             0.0.0.0:8080->8080/tcp                                                                       movie-rest-services_movie-composite_1
d45062ca4e0c        movie-rest-services_recommendation    "java -jar /app.jar"     23 minutes ago      Up 23 minutes             8080/tcp                                                                                     movie-rest-services_recommendation_1
a16db364e277        movie-rest-services_eureka            "java -jar /app.jar"     23 minutes ago      Up 23 minutes             0.0.0.0:8761->8761/tcp                                                                       movie-rest-services_eureka_1
fd6b45ade9c3        rabbitmq:3.7.8-management             "docker-entrypoint.s…"   23 minutes ago      Up 23 minutes (healthy)   4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   movie-rest-services_rabbitmq_1
68c9e3e92256        mongo:3.6.9                           "docker-entrypoint.s…"   23 minutes ago      Up 23 minutes (healthy)   0.0.0.0:27017->27017/tcp                                                                     movie-rest-services_mongodb_1
2ad058b543d8        mysql:5.7                             "docker-entrypoint.s…"   23 minutes ago      Up 23 minutes (healthy)   0.0.0.0:3306->3306/tcp, 33060/tcp                                                            movie-rest-services_mysql_1

all of my microservices including composite have spring.application.name and eureka configuration set like:

spring.application.name: movie

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8761/eureka/
    initialInstanceInfoReplicationIntervalSeconds: 5
    registryFetchIntervalSeconds: 5
  instance:
    leaseRenewalIntervalInSeconds: 5
    leaseExpirationDurationInSeconds: 5

and Eureka service application looks like:

server:
  port: 8761

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/
  # from: https://github.com/spring-cloud-samples/eureka/blob/master/src/main/resources/application.yml
  server:
    waitTimeInMsWhenSyncEmpty: 0
    response-cache-update-interval-ms: 5000

management.endpoints.web.exposure.include: "*"

I am checking if microservice is up or down like (I will stick to movie example described above):

private Mono<Health> getHealth(String url) {
    url += "/actuator/health";
    log.info("Will call the Health API on URL: {}", url);
    return getWebClient().get().uri(url).retrieve().bodyToMono(String.class)
            .map(s -> new Health.Builder().up().build())
            .onErrorResume(ex -> Mono.just(new Health.Builder().down(ex).build()))
            .log();
}

where url is: private final String MOVIE_SERVICE_URL = "http://movie";

Which will give output of:

movie-composite_1  | 2020-02-11 18:03:07.251  INFO 1 --- [or-http-epoll-6] c.g.m.c.m.s.MovieCompositeIntegration    : Will call the Health API on URL: http://movie/actuator/health
movie-composite_1  | 2020-02-11 18:03:07.253  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.4             : onSubscribe(FluxOnErrorResume.ResumeSubscriber)
movie-composite_1  | 2020-02-11 18:03:07.253  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.4             : request(unbounded)
movie-composite_1  | 2020-02-11 18:03:07.256  INFO 1 --- [or-http-epoll-6] c.g.m.c.m.s.MovieCompositeIntegration    : Will call the Health API on URL: http://recommendation/actuator/health
movie-composite_1  | 2020-02-11 18:03:07.257  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.5             : onSubscribe(FluxOnErrorResume.ResumeSubscriber)
movie-composite_1  | 2020-02-11 18:03:07.257  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.5             : request(unbounded)
movie-composite_1  | 2020-02-11 18:03:07.260  INFO 1 --- [or-http-epoll-6] c.g.m.c.m.s.MovieCompositeIntegration    : Will call the Health API on URL: http://review/actuator/health
movie-composite_1  | 2020-02-11 18:03:07.261  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.6             : onSubscribe(FluxOnErrorResume.ResumeSubscriber)
movie-composite_1  | 2020-02-11 18:03:07.261  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.6             : request(unbounded)
movie-composite_1  | 2020-02-11 18:03:07.265  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.4             : onNext(DOWN {error=io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: movie/172.21.0.8:80})
movie-composite_1  | 2020-02-11 18:03:07.266  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.4             : onComplete()
movie-composite_1  | 2020-02-11 18:03:07.266  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.5             : onNext(DOWN {error=io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: recommendation/172.21.0.6:80})
movie-composite_1  | 2020-02-11 18:03:07.266  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.5             : onComplete()
movie-composite_1  | 2020-02-11 18:03:07.267  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.6             : onNext(DOWN {error=io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: review/172.21.0.9:80})
movie-composite_1  | 2020-02-11 18:03:07.273  INFO 1 --- [or-http-epoll-6] reactor.Mono.OnErrorResume.6             : onComplete()

Can you give me any hint on this? All test are passing, and mvn clean install also is successful.

Solution

  • Looking at your docker ps results, it seems like the movie service is listening at port 8080.

    In that case, you will need to reach that service at http://movie:8080/actuator/health. Same go for the other services!

    Explanation:

    • Each container will be able to listen to any port that it wishes without interfering with other containers or host machine. For example, movie and review service can both listen on port 8080 (similar behavior to different machine/VM in the same network)
    • In order for containers to communicate with each other, they need to be in the same network and other containers need to reach them at the port that they are listening to e.g. movie:8080 or review:8080
    • However, you will not be able to reach the containers at those port directly from your host machine because your localhost and the containers are not in the same network. In order to communicate, you will need to do something call port mapping like in the case of your movie-composite service 0.0.0.0:8080->8080/tcp, you are mapping the port 8080 that the container is listening to to the port 8080 of your localhost. Hence, you are able to open this link on the browser http://localhost:8080/actuator/health