How to invoke a web service after redeliveries exhausted in Apache Camel?

Question

I have failed to find an enterprise integration pattern or recipe that promotes a solution for this problem:

After the re-delivery attempts have been exhausted, I need to send a web service request back to the originating source, to notify the sender of a failed delivery.

Upon exhaustion of all re-delivery attempts, should I move the message to a dead letter queue? Then create a new consumer listening on that DL queue? Do I need a unique dead letter queue for each of my source message queues? Should I add a message header, noting the source queue, before I move it to the dead letter queue? If all messages go to a single dead letter queue, how should my consumer know where to send the web service request?

Can you point me to a book, blog post, or article? What is the prescribed approach?

I'm working in a really old version of Fuse ESB but I expect that solutions in ServiceMix to be equally applicable.

Or maybe, what I'm asking for is an anti-pattern or code-smell. Please advise.

Answer 1

If you are new to Camel and really want to get an in-depth knowledge of it, I would recommend Camel in Action, a book by Claus Ibsen. There's a second edition in the works, with 14 out of 19 chapters already done so you may also give that a shot.

If that's a bit too much, online documentation is pretty okay, you can find out the basics just fine from it. For error handling I recommend starting with the general error handling page then moving on to error handler docs and exception policy documentation.

Generally, dead letter channel is the way to go - Camel will automatically send to DLC after retries have been exhausted, you just have to define the DLC yourself. And its name implies, it's a channel and doesn't really need to be a queue - you can write to file, invoke a web-service, submit a message to a message queue or just write to logs, it's completely up to you.

// error-handler DLC, will send to HTTP endpoint when retries are exhausted
errorHandler(deadLetterChannel("http4://my.webservice.hos/path")
    .useOriginalMessage()
    .maximumRedeliveries(3)
    .redeliveryDelay(5000))

// exception-clause DLC, will send to HTTP endpoint when retries are exhausted
onException(NetworkException.class)
    .handled(true)
    .maximumRedeliveries(5)
    .backOffMultiplier(3)
    .redeliveryDelay(15000)
    .to("http4://my.webservice.hos/otherpath");

I myself have always preferred having a message queue and then consuming from there for any other recovery or reporting. I generally include failure details like exchange ID and route ID, message headers, error message and sometimes even stacktrace. The resulting message, as you can imagine, grows quite a bit but it tremendously simplifies troubleshooting and debugging, especially in environments where you have quite a number of components and services. Here's a sample DLC message from one my projects:

public class DeadLetterChannelMessage {
  private String timestamp = Times.nowInUtc().toString();
  private String exchangeId;
  private String originalMessageBody;
  private Map<String, Object> headers;
  private String fromRouteId;
  private String errorMessage;
  private String stackTrace;

  @RequiredByThirdPartyFramework("jackson")
  private DeadLetterChannelMessage() {
  }

  @SuppressWarnings("ThrowableResultOfMethodCallIgnored")
  public DeadLetterChannelMessage(Exchange e) {
    exchangeId = e.getExchangeId();
    originalMessageBody = e.getIn().getBody(String.class);
    headers = Collections.unmodifiableMap(e.getIn().getHeaders());
    fromRouteId = e.getFromRouteId();

    Optional.ofNullable(e.getProperty(Exchange.EXCEPTION_CAUGHT, Exception.class))
      .ifPresent(throwable -> {
        errorMessage = throwable.getMessage();
        stackTrace = ExceptionUtils.getStackTrace(throwable);
      });
  }

  // getters
}

When consuming from the dead letter queue, route ID can tell you where the failure originated from so you can then implement routes that are specific for handing errors coming from there:

// general DLC handling route
from("{{your.dlc.uri}}")
    .routeId(ID_REPROCESSABLE_DLC_ROUTE)
    .removeHeaders(Headers.ALL)
    .unmarshal().json(JsonLibrary.Jackson, DeadLetterChannelMessage.class)
    .toD("direct:reprocess_${body.fromRouteId}"); // error handling route

// handle errors from `myRouteId`
from("direct:reprocess_myRouteId")  
    .log("Error: ${body.errorMessage} for ${body.originalMessageBody}"); 
    // you'll probably do something better here, e.g.
    // .convertBodyTo(WebServiceErrorReport.class) // requires a converter
    // .process(e -> { //do some pre-processing, like setting headers/properties })
    // .toD("http4://web-service-uri/path"); // send to web-service


// for routes that have no DLC handling supplied
onException(DirectConsumerNotAvailableException.class)
     .handled(true)
     .useOriginalMessage()
     .removeHeaders(Headers.ALL)
     .to({{my.unreprocessable.dlc}}); // errors that cannot be recovered from