Luigi not using scheduler host/port from configuration

Question

Please let me know if my understanding of Luigi is off.

I currently have luigi running on a Linux instance that I want to use as a central scheduler.

When trying to run to run Python code from another instance, I cannot get Python to point to the central scheduler. Luigi always tries to connect to http://localhost:8082 instead.

luigi.rpc.RPCError: Errors (3 attempts) when connecting to remote scheduler 'http://localhost:8082'

I have a luigi.cfg file in the project folder that contains:

[core]
default_scheduler_host=some-ip
default_scheduler_port=some-port
default_scheduler_url=http://some-ip:some-port

I have also set the LUIGI_CONFIG_PATH variable to this file.

Yet Python keeps attemping to connect to http://localhost:8082. What am I doing wrong here?

Answer 1

I ended up looking through the luigi code in my site package, and found the following code:

scheduler_host = parameter.Parameter(
    default='localhost',
    description='Hostname of machine running remote scheduler',
    config_path=dict(section='core', name='default-scheduler-host'))
scheduler_port = parameter.IntParameter(
    default=8082,
    description='Port of remote scheduler api process',
    config_path=dict(section='core', name='default-scheduler-port'))
scheduler_url = parameter.Parameter(
    default='',
    description='Full path to remote scheduler',
    config_path=dict(section='core', name='default-scheduler-url'),
)

Turns out the version of luigi I was using looks for default-scheduler-host instead of default_scheduler_port, etc.