import - Why python finds module instead of package if they have the same name?

Question

Ask a Question

Welcome To Ask or Share your Answers For Others

import - Why python finds module instead of package if they have the same name?

asked Oct 24, 2021 in Technique[技术] by 深蓝 (71.8m points)

Here is my directory structure:

/home/dmugtasimov/tmp/name-res
    root
        tests
            __init__.py
            test_1.py
        __init__.py
        classes.py
        extra.py
        root.py

File contents: root/tests/_init_.py

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__),
                                             '../..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)
# These imports are required for unittest to find test modules in package properly
from root.tests import test_1

root/tests/test_1.py

import unittest
from root.classes import Class1
class Tests(unittest.TestCase):
    pass

root/_init_.py - empty
root/classes.py

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)

print 'sys.path:', sys.path
print 'BEFORE: import root.extra'
import root.extra
print 'AFTER: import root.extra'

class Class1(object):
    pass

class Class2(object):
    pass

root/extra.py

class Class3(object):
    pass

root/root.py

import os
import sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    sys.path.insert(0, ROOT_DIRECTORY)
from classes import Class2

I get following output:

$ python -m unittest tests.test_1
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
 File "/usr/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
 File "/usr/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
 File "/usr/lib/python2.7/unittest/main.py", line 149, in parseArgs
    self.createTests()
 File "/usr/lib/python2.7/unittest/main.py", line 158, in createTests
    self.module)
 File "/usr/lib/python2.7/unittest/loader.py", line 128, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
 File "/usr/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
 File "tests/__init__.py", line 9, in <module>
    from root.tests import test_1
 File "/home/dmugtasimov/tmp/name-res/root/tests/__init__.py", line 9, in <module>
    from root.tests import test_1
 File "/home/dmugtasimov/tmp/name-res/root/tests/test_1.py", line 3, in <module>
    from root.classes import Class1
 File "/home/dmugtasimov/tmp/name-res/root/classes.py", line 9, in <module>
    import root.extra
 File "/home/dmugtasimov/tmp/name-res/root/root.py", line 6, in <module>
    from classes import Class2
ImportError: cannot import name Class2

It turns out the problem is the order used by python interpreter to search for package or module:

$ python -vv -m unittest tests.test_1
…skipped...
import root.classes # precompiled from /home/dmugtasimov/tmp/name-res/root/classes.pyc
sys.path: ['/home/dmugtasimov/tmp/name-res', '', '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE: import root.extra
# trying /home/dmugtasimov/tmp/name-res/root/root.so
# trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so
# trying /home/dmugtasimov/tmp/name-res/root/root.py
# /home/dmugtasimov/tmp/name-res/root/root.pyc matches /home/dmugtasimov/tmp/name-res/root/root.py
…skipped...

According to python documentation http://docs.python.org/2/tutorial/modules.html#the-module-search-path: “When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path.”

This means that python should look at sys.path index 0 entry, get path '/home/dmugtasimov/tmp/name-res' and find package named root and then search module named extra in this package. But instead it searches in /home/dmugtasimov/tmp/name-res/root/ directory for module root and then tries to find something named extra in it. What does it happen? Does not it contradict official documentation? Or are rules for searching packages different than for modules? If so, are these rules covered somewhere in documentation?

UPDATE

I put it here for better formatting.
For futher investigation do the following:

Remove root.pyc
Rename root.py to root2.py
Run python -vv -m unittest tests.test_1

# trying /home/dmugtasimov/tmp/name-res/root/root.so
# trying /home/dmugtasimov/tmp/name-res/root/rootmodule.so
# trying /home/dmugtasimov/tmp/name-res/root/root.py
# trying /home/dmugtasimov/tmp/name-res/root/root.pyc
# trying /home/dmugtasimov/tmp/name-res/root/extra.so
# trying /home/dmugtasimov/tmp/name-res/root/extramodule.so
# trying /home/dmugtasimov/tmp/name-res/root/extra.py

It appears python disregard sys.path only for initial 4 tries.

UPDATE 2

Simplified version:

/home/dmugtasimov/tmp/name-res3/xyz
    __init__.py
    a.py
    b.py
    t.py
    xyz.py

Files init.py, b.py and xyz.py are empty
File a.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in a.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in a.py'

print 'sys.path:', sys.path
print 'BEFORE import xyz.b'
import xyz.b
print 'AFTER import xyz.b'

File t.py:

import os, sys
ROOT_DIRECTORY = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
if not sys.path or ROOT_DIRECTORY not in sys.path:
    print 'sys.path is modified in t.py'
    sys.path.insert(0, ROOT_DIRECTORY)
else:
    print 'sys.path is NOT modified in t.py'

import xyz.a

Run:

python a.py

Output:

sys.path is modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
AFTER import xyz.b

Run:

python -vv a.py

Output:

import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
# /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
import xyz # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/bmodule.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/b.py
# /home/dmugtasimov/tmp/name-res3/xyz/b.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/b.py
import xyz.b # precompiled from /home/dmugtasimov/tmp/name-res3/xyz/b.pyc

Run:

python t.py

Output:

sys.path is modified in t.py
sys.path is NOT modified in a.py
sys.path: ['/home/dmugtasimov/tmp/name-res3', '/home/dmugtasimov/tmp/name-res3/xyz',
 '/usr/local/lib/python2.7/dist-packages/tornado-2.3-py2.7.egg',
 '/home/dmugtasimov/tmp/name-res3/xyz', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg-info',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PIL',
 '/usr/lib/python2.7/dist-packages/gst-0.10',
 '/usr/lib/python2.7/dist-packages/gtk-2.0',
 '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']
BEFORE import xyz.b
Traceback (most recent call last):
  File "t.py", line 9, in <module>
    import xyz.a
  File "/home/dmugtasimov/tmp/name-res3/xyz/a.py", line 11, in <module>
    import xyz.b
ImportError: No module named b

Run:

python -vv t.py

Output:

import xyz # directory /home/dmugtasimov/tmp/name-res3/xyz
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__module.so
# trying /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
# /home/dmugtasimov/tmp/name-res3/xyz/__init__.pyc matches /home/dmugtasimov/tmp/name-res3/xyz/__init__.py
import

与恶龙缠斗过久,自身亦成为恶龙；凝视深渊过久,深渊将回以凝视…

490 views

1 Answer

深蓝 · Answer 1 · 2021-10-23T19:22:53+0000

Do not modify sys.path it leads to the issues when the same module is available under different names. See Traps for the Unwary.

Use absolute or explicit relative imports instead in the code and run your scripts from the project directory. Run the tests using the full name:

$ python -munittest root.tests.test_1

Some packages do modify sys.path internally e.g., see how twisted uses _preamble.py or pypy's autopath.py. You could decide whether their downsides (introduction of obscure import problems) are worth the convience (more ways to run your scripts are allowed). Avoid modifying sys.path in a code that is used as a library i.e., limit it to test modules and your command-line scripts.

Categories

import - Why python finds module instead of package if they have the same name?

UPDATE

UPDATE 2

Please log in or register to add a comment.

Please log in or register to answer this question.

1 Answer

Please log in or register to add a comment.

Just Browsing Browsing

Most popular tags