Search code examples
erlangerlang-supervisorgen-server

How do I tell supervisor to start 1000 instances of a specific gen_server?


As part of a solving a computationally intensive task, I wish to have 1000 gen_servers doing small task and update the global database. How can I achieve this in erlang OTP? In most of the examples the supervisor supervises only a single gen_server. Can a supervisor supervise more than a thousand instances of the same gen_server?

e.g. Say I want to find maximum of a extremely long array and each gen_server instance should create work on a part of the array and update the global minimum.


Solution

  • Is it possible: yes. For example you can create a pool of 1000 processes with the following supervisor:

    -module (big_supervisor).
    
    -export([start_link/0]).
    
    -behaviour(supervisor).
    -export([init/1]).
    
    start_link() ->
        supervisor:start_link({local, ?MODULE}, ?MODULE, {}).
    
    %% @private
    init({}) ->
        Children = create_child_specs(1000),
        RestartStrategy = {one_for_one, 5, 10},
        {ok, {RestartStrategy, Children}}.
    
    create_child_specs(Number) ->
        [{{child_process,X},{child_process, start_link, []},permanent, 5000, worker,[child_process]} || X <- lists:seq(1,Number)].
    

    Is it a good architecture, I don't know. Until now I have found 2 kinds of architectures:

    • One with a limited and well identified (by role) chidren
    • One with a kind of process factory, creating dynamically as many children as needed on demand, using the simple_one_for_one strategy and the start_child/2 terminate_child/2 functions.

    Notes also that the supervisors are not mandatory if you want to spawn processes. In your explanation it seems that the processes could be created for a very limited time, in order to compute in parallel something. Two remarks in this case:

    • it is not worth to spawn more processes that the number of threads that will effectively run in parallel on your VM.
    • one exception is if the work to achieve in each process will have to wait for an external information, for example the return of an external database. In this case it may be interesting to spawn more processes, the optimal number depending on the external access limits.