Prioritizing Erlang nodes

Question

Assuming I have a cluster of n Erlang nodes, some of which may be on my LAN, while others may be connected using a WAN (that is, via the Internet), what are suitable mechanisms to cater for a) different bandwidth availability/behavior (for example, latency induced) and b) nodes with differing computational power (or even memory constraints for that matter)?

In other words, how do I prioritize local nodes that have lots of computational power, over those that have a high latency and may be less powerful, or how would I ideally prioritize high performance remote nodes with high transmission latencies to specifically do those processes with a relatively huge computations/transmission (that is, completed work per message ,per time unit) ratio?

I am mostly thinking in terms of basically benchmarking each node in a cluster by sending them a benchmark process to run during initialization, so that the latencies involved in messasing can be calculated, as well as the overall computation speed (that is, using a node-specific timer to determine how fast a node terminates with any task).

Probably, something like that would have to be done repeatedly, on the one hand in order to get representative data (that is, averaging data) and on the other hand it might possibly even be useful at runtime in order to be able to dynamically adjust to changing runtime conditions.

(In the same sense, one would probably want to prioritize locally running nodes over those running on other machines)

This would be meant to hopefully optimize internal job dispatch so that specific nodes handle specific jobs.

Answer 1

We've done something similar to this, on our internal LAN/WAN only (WAN being for instance San Francisco to London). The problem boiled down to a combination of these factors:

1. The overhead in simply making a remote call over a local (internal) call
2. The network latency to the node (as a function of the request/result payload)
3. The performance of the remote node
4. The compute power needed to execute the function
5. Whether batching of calls provides any performance improvement if there was a shared "static" data set.

For 1. we assumed no overhead (it was negligible compared to the others)

For 2. we actively measured it using probe messages to measure round trip time, and we collated information from actual calls made

For 3. we measured it on the node and had them broadcast that information (this changed depending on the load current active on the node)

For 4 and 5. we worked it out empirically for the given batch

Then the caller solved to get the minimum solution for a batch of calls (in our case pricing a whole bunch of derivatives) and fired them off to the nodes in batches.

We got much better utilization of our calculation "grid" using this technique but it was quite a bit of effort. We had the added advantage that the grid was only used by this environment so we had a lot more control. Adding in an internet mix (variable latency) and other users of the grid (variable performance) would only increase the complexity with possible diminishing returns...