Is there a way to set the target for a task dynamically with the App Engine Java runtime?

Question

When enqueuing a background task with the App Engine Python runtime, you can specify a target for the queue which will send the task to be run on a specific service, version, or instance:

task = taskqueue.add(
    url='/update_counter',
    target='worker',
    params={'amount': amount})

Is there a way to do this in Java? The documentation mentions the target parameter but doesn't show an example of how to use it. The Queue.add method has no option for target. The TaskOptions class doesn't have anything that looks like target, either.

This question documents how to use the target, but the answer is to configure it in queue.xml. I'd like to pick the target at runtime, like Python.

Answer 1

TL;DR - There might be a way to do this based off earlier version of the documentation, but this is not described in the latest version of the documentation.

Approach based on the latest documentation

As per the latest documentation regarding Push queues (and like you already mentioned), you can configure the target module, version in the queue.xml per queue. If specified, the request for the task is sent to the specified target. As you already described, this is a static configuration and does not really answer your question (describing just for the sake of completeness).

<?xml version="1.0" encoding="UTF-8"?>
  <queue-entries>
    <queue>
      <name>queue-blue</name>
      <target>v2.task-module</target>
    </queue>
    <queue>
      <name>queue-red</name>
      <rate>1/s</rate>
    </queue>
  </queue-entries>

In the documentation for creating tasks and specifying worker service, it describes about the target chosen for the task but does not clearly describe how to specify it.

When a task is popped off its queue, the Task Queue service sends it on to a worker service. Every task has a target and a url, which determine what service and handler will ultimately perform the task.

target

The target specifies the service that will receive the HTTP request to perform the task. It is a string that specifies a service/version/instance in any one of the canonical forms. The most often-used ones are:

service
version.service
instance.version.service

The target string is prepended to the domain name of your app. There are three ways to set the target for a task:

• Explicity declare the target when you construct the task.
• Include a target directive when you define a queue in the queue.xml, as in the definition of queue-blue above. All tasks added to a queue with a target will use that target, even if a different target was assigned to the task at construction time.
• If no target is specified according to either of the previous two methods, then the task's target is the version of the service that enqueues it. Note that if you enqueue a task from the default service and version in this manner, and the default version changes before the task executes, it will run in the new default version.

url

The url selects one of the handlers in the target service, which will perform the task.

The url should match one of the handler URL patterns in the target service. The url can include query parameters if the tasks's method is GET or PULL. If no url is specified the default URL /_ah/queue/[QUEUE_NAME] is used, where [QUEUE_NAME] is the name of the task's queue.

Based off the above documentation, if you look at TaskOptions.Builder, there is no method to specify the target for the task. This could either indicate a missing documentation on how the target should be specified or just that there is no way any more to specify the target dynamically when adding the task to the queue. Look at the approach described below based on the earlier documentation.

Approach based on earlier documentation

DISCLAIMER: What I mention here looks to be based on outdated information (I've cited the sources below) and hence might not work as expected and/or break in the future.

You can use the Host header to specify the module, version and instance information which will get the request.

To specify module and version, you can do:

Queue queue = QueueFactory.getQueue("QUEUE_NAME");
    queue.add(TaskOptions.Builder
            .withUrl("/url/path")
            .param("key", "PARAM")
            .header("Host",
                    ModulesServiceFactory.getModulesService().getVersionHostname("MODULE_NAME", "VERSION")));

To specify instance you can do:

Queue queue = QueueFactory.getQueue("QUEUE_NAME");
    queue.add(TaskOptions.Builder
            .withUrl("/url/path")
            .param("key", "PARAM")
            .header("Host",
                    ModulesServiceFactory.getModulesService().getInstanceHostname("MODULE_NAME", "VERSION", "INSTANCE_NAME"));

I could not find this info in the current version of the App Engine documentation, but using the Wayback machine I found this info from an earlier version of the doc from Jan 1 2016 which describes the Push task execution. It has been discussed in a different context in this github issue as well.

Push task execution

App Engine executes push tasks by sending HTTP POST requests to your app. Specifying a programmatic asynchronous callback as an HTTP request is sometimes called a web hook. The web hook model enables efficient parallel processing.

The task's URL determines the handler for the task and the module that runs the handler.

The handler is determined by the path part of the URL (the forward-slash separated string following the hostname), which is specified by the url parameter in the TaskOptions that you include in your call to the Queue.add() method. The url must be relative and local to your application's root directory.

The module and version in which the handler runs is determined by:

• The "Host" header parameter in the TaskOptions that you include in your call to the Queue.add() method.
• The target directive in the queue.xml or queue.yaml file.

If you do not specify any of these parameters, the task will run in the same module/version in which it was enqueued, subject to these rules:

• If the default version of the app enqueues a task, the task will run on the default version. Note that if the app enqueues a task and the default version is changed before the task actually runs, the task will be executed in the new default version.

• If a non-default version enqueues a task, the task will always run on that same version.

Note: If you are using modules along with a dispatch file, a task's URL may be intercepted and re-routed to another module.

The namespace in which a push task runs is determined when the task is added to the queue. By default, a task will run in the current namespace of the process that created the task. You can override this behavior by explicitly setting the namespace before adding a task to a queue, as described on the multitenancy page.