subprocess.Popen simple code does not allow me to perform a cd (change directory)

I'm trying to use a Python script to change directory, but I am getting an error.

The python code:

import subprocess
p = subprocess.Popen(['cd', '~'], stdout=subprocess.PIPE)
output = p.communicate()
print output

I get this error:

File "test_sub.py", line 2, in <module>
p = subprocess.Popen(['cd', '~'], stdout=subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

What does the error mean, what am I doing wrong, and how do I change directory in a python subprocess?

Solution

>>> Popen('cd ~', shell=True, stdout=PIPE).communicate()
(b'', None)

Without shell=True (which, runs the command in shell, on POSIX that defaults to /bin/sh)

>>> Popen(['cd', '~'], stdout=PIPE).communicate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/subprocess.py", line 858, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1456, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'cd'
>>>

You can't change directory unless you do it via:

import os
os.chdir(os.path.abspath(os.path.expanduser('~')))

So the problem isn't that the path ~ doesn't exist, but rather cd doesn't exist as an option unless your command is run in a shell that supports it. Passing directly to an actual shell makes cd work. But note that shell=True is a risk, never use it unless you need to..
So use os.chdir instead.

A working scenario:

import os, subprocess
os.chdir(os.path.abspath('/tmp/'))
print(subprocess.Popen(['ls', '-lah'], stdout=subprocess.PIPE).communicate()[0].decode('utf-8'))

Resulting in:

[torxed@archie ~]$ python
Python 3.4.1 (default, May 19 2014, 17:23:49) 

>>> import os, subprocess
>>> os.chdir(os.path.abspath('/tmp/'))
>>> print(subprocess.Popen(['ls', '-lah'], stdout=subprocess.PIPE).communicate()[0].decode('utf-8'))

total 12K
drwxrwxrwt  9 root   root   220 Jun 11 12:08 .
drwxr-xr-x 19 root   root  4.0K May 28 08:03 ..
drwxrwxrwt  2 root   root    40 Jun 11 09:30 .font-unix
drwx------  2 torxed users   60 Jun 11 09:33 gpg-LBLcdd
drwxrwxrwt  2 root   root    40 Jun 11 09:30 .ICE-unix
drwx------  2 torxed users   80 Jun 11 09:34 .org.chromium.Chromium.LEqfXB
-rw-------  1 torxed users  153 Jun 11 09:34 serverauth.EHWB0LqCv6
drwxrwxrwt  2 root   root    40 Jun 11 09:30 .Test-unix
-r--r--r--  1 root   users   11 Jun 11 09:34 .X0-lock
drwxrwxrwt  2 root   root    60 Jun 11 09:34 .X11-unix
drwxrwxrwt  2 root   root    40 Jun 11 09:30 .XIM-unix

>>>

Note that i started the shell in ~ and via os.chdir changed it to tmp and actually got my tmp directory content.

Explanation of shells and commands:

A shell-command is something that's built into the shell while a regular old command is something you'll find under /bin, for instance:

[torxed@archie ~]$ ls /bin
2to3            2to3-2.7
7z              7za
...

Where 7z is a command i can actually execute:

>>> from subprocess import *
>>> Popen(['7z'], stdout=PIPE).communicate()

(b'\n7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18\np7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)\n

While for instance cd is a built in shell command, something that you will not find under /bin but works anyway in most "terminals" (using a shell) because it's (as mentioned), built into the shell you normally see.

But because Python will by default not execute the command in a shell you more or less have to rely on using os.chdir(...) or wrap your command in /bin/sh -c "cd ..." or something similar.