Spring Integration Java DSL: How to loop the paged Rest service?

Question

How to loop the paged Rest service with the Java DSL Http.outboundGatewaymethod?

The rest URL is for example

http://localhost:8080/people?page=3

and it returns for example

"content": [
    {"name": "Mike",
     "city": "MyCity"
    },
    {"name": "Peter",
     "city": "MyCity"
    },
    ...
 ]
"pageable": {
    "sort": {
        "sorted": false,
        "unsorted": true
    },
    "pageSize": 20,
    "pageNumber": 3,
    "offset": 60,
    "paged": true,
    "unpaged": false
},
"last": false,
"totalElements": 250,
"totalPages": 13,
"first": false,
"sort": {
    "sorted": false,
    "unsorted": true
},
"number": 3,
"numberOfElements": 20,
"size": 20
}

where the variable totalPages tells the total pages amount.

So if the implementation

        integrationFlowBuilder
          .handle(Http
            .outboundGateway("http://localhost:8080/people?page=3")
            .httpMethod(HttpMethod.GET)
            .expectedResponseType(String.class))

access one page, how to loop all the pages?

Answer 1

The easiest way to do this is like wrapping the call to this Http.outboundGateway() with the @MessagingGateway and provide a page number as an argument:

@MessagingGateway
public interface HttpPagingGateway {

    @Gateway(requestChannel = "httpPagingGatewayChannel")
    String getPage(int page);

}

Then you get a JSON as a result, where you can convert it into some domain model or just perform a JsonPathUtils.evaluate() (based on json-path) to get the value of the last attribute to be sure that you need to call that getPage() for the page++ or not.

The page argument is going to be a payload of the message to send and that can be used as an uriVariable:

.handle(Http
        .outboundGateway("http://localhost:8080/people?page={page}")
        .httpMethod(HttpMethod.GET)
        .uriVariable("page", Message::getPayload)
        .expectedResponseType(String.class))

Of course, we can do something similar with Spring Integration, but there are going to be involved filter, router and some other components.

UPDATE

First of all I would suggest you to create a domain model (some Java Bean), let's say PersonPageResult, to represent that JSON response and this type to the expectedResponseType(PersonPageResult.class) property of the Http.outboundGateway(). The RestTemplate together with the MappingJackson2HttpMessageConverter out-of-the-box will do the trick for you to return such an object as a reply for the downstream processing.

Then, as I said before, looping would be better done from some Java code, which you could wrap to the service activator call. For this purpose you should daclare a gateway like this:

public interface HttpPagingGateway {

    PersonPageResult getPage(int page);

}

Pay attention: no annotations at all. The trick is done via IntegrationFlow:

@Bean
public IntegrationFlow httpGatewayFlow() {
    return IntegrationFlows.from(HttpPagingGateway.class)
                  .handle(Http
                       .outboundGateway("http://localhost:8080/people?page={page}")
                       .httpMethod(HttpMethod.GET)
                       .uriVariable("page", Message::getPayload)
                       .expectedResponseType(PersonPageResult.class))  
}

See IntegrationFlows.from(Class<?> aClass) JavaDocs.

Such a HttpPagingGateway can be injected into some service with hard looping logic:

int page = 1;
boolean last = false;
while(!last) {
  PersonPageResult result = this.httpPagingGateway.getPage(page++);
  last = result.getLast();
  List<Person> persons = result.getPersons();
  // Process persons
}

For processing those persons I would suggest to have separate IntegrationFlow, which may start from the gateway as well or you can just send a Message<List<Person>> to its input channel.

This way you will separate concerns about paging and processing and will have a simple loop logic in some POJO method.