Is it possible to get a list of all streams using the spring cloud dataflow java rest client?

Question

I'm using the spring cloud dataflow java rest client (https://docs.spring.io/spring-cloud-dataflow/docs/current/api/) and want to use it to retrieve all currently deployed streams.

It's easy enough to get a StreamOperations object and get a list of streams from it:

val template = DataFlowTemplate(<someUri>)
val streamOperations = template.streamOperations()
val streamDefinitionResources = streamOperations.list()

The streamDefinitionResources in the above is actually a PagedModel<StreamDefinitionResource>, that holds the first page of results using a page size of 2000.

I don't, however, see any way to iterate through all the pages to get all the streams using the java rest client (i.e. there's no paging support available via the StreamOperations or StreamDefinitionResource classes).

Is it possible to get all the streams using only the java rest client? Am I missing something?

Answer 1

Summary

The PagedModel<StreamDefinitionResource> has a getNextLink() method that you can use to manually traverse the "next" page of results.

Details

The underlying Dataflow REST API supports paging via page number and size request parameters and returns HAL responses that include _links to the next and previous pages.

For example, given 10 stream definitions this HTTP request:

GET localhost:9393/streams/definitions?page=0&size=2

returns the following response:

{
  _embedded: {
    streamDefinitionResourceList: [
      {
        name: "ticktock1",
        dslText: "time | log",
        originalDslText: "time | log",
        status: "undeployed",
        description: "",
        statusDescription: "The app or group is known to the system, but is not currently deployed",
        _links: {
          self: {
            href: "http://localhost:9393/streams/definitions/ticktock1"
          }
        }
      },
      {
        name: "ticktock2",
        dslText: "time | log",
        originalDslText: "time | log",
        status: "undeployed",
        description: "",
        statusDescription: "The app or group is known to the system, but is not currently deployed",
        _links: {
          self: {
            href: "http://localhost:9393/streams/definitions/ticktock2"
          }
        }
      }
    ]
  },
  _links: {
    first: {
      href: "http://localhost:9393/streams/definitions?page=0&size=2"
    },
    self: {
      href: "http://localhost:9393/streams/definitions?page=0&size=2"
    },
    next: {
      href: "http://localhost:9393/streams/definitions?page=1&size=2"
    },
    last: {
      href: "http://localhost:9393/streams/definitions?page=4&size=2"
    }
  },
  page: {
    size: 2,
    totalElements: 10,
    totalPages: 5,
    number: 0
  }
}

The Dataflow Java REST client exposes this HAL response in the PagedModel<StreamDefinitionResource> response which provides a getNextLink() method.

Caveat 1) However, the current implementation (as you pointed out) is hardcoded to page size of 2000. This means you would not see this behavior until you had more than 2000 stream definitions.

Caveat 2) Another point to note is that traversal of the link to the "next" page is not automatically handled and you would need to manually invoke the links URL to retrieve the next page.

Assume the StreamOperations.list accepted a page size parameter the code could look something like this:

int pageSize = 2;
PagedModel<StreamDefinitionResource> pageOfStreamDefs = streamOperations().list(pageSize);
pageOfStreamDefs.getNextLink()
    .ifPresent((link) ->  someFunctionToInvokeAndProcessNextPage(link.getHref());

More details on the REST API parameters can be found here.