Lessons learned while using Python’s subprocess module

I was recently poking around the idea of Python’s subprocess module to shell out and call a few commands. It didn’t pan out. But I learned some things along the way that might help someone else one day.

Wait, my last command isn’t done yet!

At one point I had a few git commands strung together, like so:

process = subprocess.Popen(['git', 'clone', 'some-repo'])
...

They were working fine when I was debugging locally, but when testing it out, the script would fail as the directory didn’t exist yet. But when I looked into it, the directory did exist. As it turns out, Python doesn’t wait for the first subprocess command to exit before going on to the next line of code to execute.

What you want to do it remember to call .communicate() on the returned process. This will ensure that the next command isn’t run until the initial process completes. It should look like this:

process = subprocess.Popen(['git', 'clone', 'some-repo'],
                    stdout=subprocess.PIPE,
                    stderr=subprocess.PIPE)
stdout, stderr = process.communicate()

Trying to change directory with subprocess is pointless

At one point I was trying to change a directory and calling it like so:

process = subprocess.Popen(['cd', 'some-dir'],
                    stdout=subprocess.PIPE,
                    stderr=subprocess.PIPE)
stdout, stderr = process.communicate()

After scratching my head as to why the next command failed and adding ls’s before and after I finally resorted to doing a quick Google search, and StackOverflow had the answer.

The gist of it is that no process can change another process’s working directory. Instead you want to use os.chdir

os.chdir('some-dir')

Summary

That’s it! Hopefully you don’t have to resort to calling subprocess in your Python work, but if you do, I hope my lesson’s learned avoid you some pain.