Search code examples
djangourlcelerybroker

Unable to use '#' as broker url in celery django


app = Celery('myapp',
             broker='amqp://user:pass#1@localhost:5672//',
             backend='rpc://',
             include=['myapp.tasks'])

I get this error

ValueError: invalid literal for int() with base 10: 'pass'

This code doesn't work, I'm new to python and Django is there an escape sequence for it?

Iv tried u"", r"" ,'#' , '##' and '#' , hoping it will escape it but it doesnt.

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\source\repos\BtcApi\BtcApi\btcapienv\Scripts\celery.exe\__main__.py", line 9, in <module>
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\__main__.py", line 14, in main
    _main()
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\celery.py", line 480, in handle_argv
    return self.execute(command, argv)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\worker.py", line 221, in run_from_argv
    return self(*args, **options)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\bin\worker.py", line 255, in run
    **kwargs)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\worker\worker.py", line 99, in __init__
    self.setup_instance(**self.prepare_args(**kwargs))
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\worker\worker.py", line 120, in setup_instance
    self._conninfo = self.app.connection_for_read()
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\app\base.py", line 752, in connection_for_read
    return self._connection(url or self.conf.broker_read_url, **kwargs)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\celery\app\base.py", line 828, in _connection
    'broker_connection_timeout', connect_timeout
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\kombu\connection.py", line 181, in __init__
    url_params = parse_url(hostname)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\kombu\utils\url.py", line 34, in parse_url
    scheme, host, port, user, password, path, query = _parse_url(url)
  File "c:\users\user\source\repos\btcapi\btcapi\btcapienv\lib\site-packages\kombu\utils\url.py", line 52, in url_to_parts
    parts.port,
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\urllib\parse.py", line 167, in port
    port = int(port, 10)
ValueError: invalid literal for int() with base 10: 'pass

I know the # is a problem because obviously if I remove that character from the password it works perfectly


Solution

  • The rabbitmq documentation for the uri refers to RFC3986. They have a section about reserved characters and # is one of them.

    If data for a URI component would conflict with a reserved character's purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

    According to the spec you might replace # with %23 - or just use a password that does not include reserved characters.

    Update

    You are right, but the same RFC is followed there, too. Celery (or kombu) uses urllib and urllib tells they want to match the RFC, too.

    It crashes in this line and there your password gets interpreted as a port. Seems like the # inside of the password enforces the library to interpret the password as the port (which fails, as it is not castable to int). Note that domain and port are separated by the same character : as username and password.

    The following illustrates whats going on. Note that everything after # is interpreted as the fragment of your url.

    >>> from urllib.parse import urlparse
    >>> url = 'amqp://user:pass#1@localhost:5672//'
    >>> urlparse(url)
    ParseResult(scheme='amqp', netloc='user:pass', path='', params='', query='', fragment='1@localhost:5672//')
    

    See what happens when we remove the #

    >>> url = 'amqp://user:pass%231@localhost:5672//'
    >>> urlparse(url)
    ParseResult(scheme='amqp', netloc='user:pass%231@localhost:5672', path='//', params='', query='', fragment='')
    >>> urlparse(url).port
    5672
    >>> urlparse(url).password
    'pass%231'
    

    The url can be parsed correctly - but I guess the password is wrong now. Sadly I can not find any sources that describe how to escape something in the password of an URI. But to be honest - this sounds weired. Escape characters in a password? I would recommend to just pick a password without # as this confuses pythons URL parser and most likely other implementations, too.