Python subprocess.call() cannot find shell script file location in docker image

Question

I have a Python (3.9) script that exists in the following file structure:

L usr
  L src
    L project_root
      L folder_a
        L folder_b
          L foo.py
          L bar.sh

foo.py contains this line of code, which calls a subprocess bar.sh and passes some data to it:

subprocess.call(['/usr/src/folder_a/folder_b/bar.sh', f'{some_data}'])

This subprocess call finds the shell script just fine when run locally.

However, when I run this same code in a docker container, I get the following exception:

Exception: [Errno 2] No such file or directory: '/usr/src/folder_a/folder_b/bar.sh'

Troubleshooting steps I have tried:

1. I confirmed that the path is correct, both spelling and location
2. The docker container's top-level is the project_root folder
3. I have tried both the relative and exact paths, neither work
4. I have SSH'd into the container and checked the files, the shell script is present in the exact directory that I provided.
5. I have tried using os.path.abspath() to generate the absolute path to the shell script, but it was still not found.
6. I have checked os.cwd to confirm that the current working directory is usr/src
7. I have used Path(__file__).parent('./bar.sh') to find the absolute path to the shell script, which just gave me the string /usr/src/folder_a/folder_b/bar.sh, the same as I've been using.

Any ideas?

Answer 1

After looking into the documentation for the subprocess module, it looks like there are a couple things wrong with my original approach.

1. The documentation states that for Python 3.5 or higher, use subprocess.run() and not subprocess.call(), as the call function is only still there for backwards compatibility:

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

The run() function was added in Python 3.5; if you need to retain compatibility with older versions, see the Older high-level API section.

2. The key is to pass the value True to kwarg shell:

subprocess.run(['/usr/src/folder_a/folder_b/bar.sh', f'{some_data}'], shell=True)

The shell kwarg is, by default, set to False. When trying to execute a shell script (or possibly any script that is a file rather than a command line tool) you can to set the shell kwarg to True to have the list of arguments passed as a sting to run through shell. For some reason, not doing this will result in a FileNotFoundError, which in my opinion is a very confusing and misleading error output...

One thing to also note is that the subprocess will run asynchronously, so there is some error trapping to be done. I found some information about checking if a subprocess is still running in another StackOverflow question.