Search code examples
pythonpython-3.xlinuxpython-3.6redhat

tempfile.TemporaryDirectory contextmanager does not use /tmp folder

I use the tempfile.TemporaryDirectory class as a contextmanager. It should use the /tmp folder in default. I have tried to use the default value and also tried to force to use the /tmp folder. It creates the temp folder to where the caller script is.

Set-up:

  • Red Hat 7 Linux
  • Python 3.6.6
  • The script is called from Jenkins

Code:

import tempfile
import os

print(os.path.dirname(__file__))
TMP_DIR_PREFIX = "my_test_"
with tempfile.TemporaryDirectory(prefix=TMP_DIR_PREFIX, dir="/tmp") as tmp_dir:
       print(tmp_dir)

Output:

>>> python3 /home/my_home/test.py 
home/my_home
home/my_home/my_test_m1vljq2h

My questions:

  • Does anybody have any idea how I can solve it?

  • How is it possible? I have checked the implementation of the TemporaryDirectory class but I couldn't see any reasons. I would be very happy if somebody would be able to explain the reason of it.

I have already read the related official documentation as well as the implementation of tempfile module but I didn't find any related code part which can cause this type of issue.

NOTE: If it is possible I don't want to inherit and change many elements of this module but I am open for ideas.

EDIT:

Complete traceback (From Jenkins running):

File "/my_path/python/3.6.0/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/my_path/python/3.6.0/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/my_home/copy_to_location.py", line 143, in upload_files
    with tempfile.TemporaryDirectory(prefix=TMP_DIR_PREFIX, dir="/tmp") as tmp_dir:
  File "/my_path/python/3.6.0/lib/python3.6/tempfile.py", line 790, in __init__
    self.name = mkdtemp(suffix, prefix, dir)
  File "/my_path/python/3.6.0/lib/python3.6/tempfile.py", line 368, in mkdtemp
    _os.mkdir(file, 0o700)
PermissionError: [Errno 13] Permission denied: '/home/my_home/my_test_q1pldmf2'

EDIT2:

I am not able to reproduce it in local. This issue happens only in Jenkins constantly!

EDIT3:

Added line: 

logging.info(tempfile._sanitize_params("my_test_", None, None))

Output in Jenkins:

2019-11-08 15:09:26 [Thread-1] [INFO] ('my_test_', '', '/tmp', <class 'str'>)

Changed line:

logging.info(tempfile._sanitize_params("my_test_", None, "/tmp"))

Added in Jenkins:

2019-11-08 15:13:46 [Thread-1] [INFO] ('my_test_', '', '/tmp', <class 'str'>)
Solution

  • The tempfile.TemporaryDirectory() object uses tempfile.mkdtemp() to create the temporary directory from the arguments passed in. This, in turn, will use tempfile.gettempdir() if you don't give it a dir argument.

    If you are passing in a dir='/tmp' and still don't see the directory created in /tmp, then there are two possibilities:

    • your prefix value is not what you think it is, but rather starts with a /
    • the tempfile module on your system has been altered and is no longer behaving the same way as the standard library version distributed with Python 3.6.0. The changes could have been made on disk, or by other Python code that changed the behaviour dynamically.

    The normal behaviour is for the mkdtemp() function to call an internal function named _sanitize_params(), which returns dir unchanged if it is set, and the value of gettempdir() otherwise:

    >>> import tempfile
>>> tempfile._sanitize_params('my_test_', None, '/tmp')
('my_test_', '', '/tmp', <class 'str'>)
>>> tempfile._sanitize_params('my_test_', None, None)
('my_test_', '', '/tmp', <class 'str'>)
>>> tempfile.gettempdir()
'/tmp'

    mkdtemp() then uses the results of that call (returning updated prefix, suffix, dir and either the bytes or str type) together with random strings, to create a new directory for you.

    This leads to the possibility is that you haven't ruled out, properly, that the prefix value is indeed what you think it is. The mkdtemp() function uses:

    os.path.join(dir, prefix + name + suffix)

    to join the dir path with the concatenation of your prefix, the candidate name (random value) and the suffix (empty string in your case). But note that the os.path.join() function will discard any path elements that come before an argument that starts with a / slash:

    >>> import os.path
>>> os.path.join("/foo", "bar")
'/foo/bar'
>>> os.path.join("/foo", "/bar")
'/bar'

    So the behaviour you see could also be explained by your prefix starting with a slash, so:

    TMP_DIR_PREFIX = "/home/my_home/my_test_"

    would immediately produce the same result:

    >>> TMP_DIR_PREFIX = "/home/my_home/my_test_"
>>> tempfile.mkdtemp(prefix=TMP_DIR_PREFIX, dir="/tmp")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../lib/python3.6/tempfile.py", line 368, in mkdtemp
    _os.mkdir(file, 0o700)
PermissionError: [Errno 13] Permission denied: '/home/my_home/my_test_v4cqpamm'

    This was reported previously to the Python project as issue #35278.

    You can trivially include two tests in your Jenkins job to rule out these options. Make sure you log the TMP_DIR_PREFIX value, as well as what tempfile._sanitize_params(TMP_DIR_PREFIX, None, '/tmp') returns.

    If either doesn't produce the expected output on your system, then you know you need to focus on looking for; either the tempfile module behaviour has changed, or your assumption that TMP_DIR_PREFIX has the value that it does is incorrect.

    You can check if the local copy differs from the publish version with the following shell command:

    $ diff -u \
> <(curl -s https://raw.githubusercontent.com/python/cpython/v3.6.0/Lib/tempfile.py) \
> /my_path/python/3.6.0/lib/python3.6/tempfile.py

    or you could calculate a checksum for the file:

    import hashlib
with open(tempfile.__file__, 'rb') as file_to_hash:
    tempfile_checksum = hashlib.sha1(file_to_hash.read()).hexdigest()

    and compare that checksum value with the one for the published file:

    $ curl -s https://raw.githubusercontent.com/python/cpython/v3.6.0/Lib/tempfile.py | \
> sha1sum
38ad01ccc5972e193e1b96a1de8b7ba1bd8d289d  -

    If that doesn't turn up anything, you'll either have step through the calls with a debugger or look at the __module__ attributes of the functions involved. E.g. if _sanitize_params() was dynamically altered (monkey patched) then tempfile._sanitize_params.__module__ will not be set to 'tempfile', for example. However, note that your traceback already shows that both TemporaryDirectory.__init__ and mkdtemp are taken from the correct file and that the line numbers for the two lines visible match those in the published source.