Tool for pinpointing circular imports in Python/Django?

Question

I have a Django app and somewhere in it is a recursive import that is causing problems. Because of the size of the app I'm having a problem pinpointing the cause of the circular import.

I know that the answer is "just don't write circular imports" but the problem is I'm having a hard time figuring out where the circular import is coming from, so ideally a tool that traced the import back to its origin would be ideal.

Does such a tool exist? Barring that, I feel like I am doing everything I can to avoid circular import problems -- moving imports to the bottom of the page if possible, moving them inside of functions rather than having them at the top, etc. but still running into problems. I'm wondering if there are any tips or tricks for avoiding them altogether.

To elaborate a bit...

In Django specifically when it encounters a circular import, sometimes it throws an error but sometimes it passes through silently but results in a situation where certain models or fields just aren't there. Frustratingly, this often happens in one context (say, the WSGI server) and not in another (the shell). So testing in the shell something like this will work:

Foo.objects.filter(bar__name='Test')

but in the web throws the error:

FieldError: Cannot resolve keyword 'bar__name' into field. Choices are: ...

With several fields conspicuously missing.

So it can't be a straightforward problem with the code since it does work in the shell but not via the website.

Some tool that figured out just what was going on would be great. ImportError is maybe the least helpful exception message ever.

question from:https://stackoverflow.com/questions/9098787/tool-for-pinpointing-circular-imports-in-python-django

Answer 1

The cause of the import error is easily found, in the backtrace of the ImportError exception.

When you look in the backtrace, you'll see that the module has been imported before. One of it's imports imported something else, executed main code, and now imports that first module. Since the first module was not fully initialized (it was still stuck at it's import code), you now get errors of symbols not found. Which makes sense, because the main code of the module didn't reach that point yet.

Common causes in Django are:

1. Importing a subpackage from a totally different module,

   e.g. from mymodule.admin.utils import ...

   This will load admin/__init__.py first, which likely imports a while load of other packages (e.g. models, admin views). The admin view gets initialized with admin.site.register(..) so the constructor could start importing more stuff. At some point that might hit your module issuing the first statement.

   I had such statement in my middleware, you can guess where that ended me up with. ;)

2. Mixing form fields, widgets and models.

   Because the model can provide a "formfield", you start importing forms. It has a widget. That widget has some constants from .. er... the model. And now you have a loop. Better import that form field class inside the def formfield() body instead of the global module scope.

3. A managers.py that refers to constants of models.py

   After all, the model needs the manager first. The manager can't start importing models.py because it was still initializing. See below, because this is the most simple situation.

4. Using ugettext() instead of ugettext_lazy.

   When you use ugettext(), the translation system needs to initialize. It runs a scan over all packages in INSTALLED_APPS, looking for a locale.XY.formats package. When your app was just initializing itself, it now gets imported again by the global module scan.

   Similar things happen with a scan for plugins, search_indexes by haystack, and other similar mechanisms.

5. Putting way too much in __init__.py.

   This is a combination of points 1 and 4, it stresses the import system because an import of a subpackage will first initialize all parent packages. In effect, a lot of code is running for a simple import and that increases the changes of having to import something from the wrong place.

The solution isn't that hard either. Once you have an idea of what is causing the loop, you remove that import statement out of the global imports (on top of the file) and place it inside a function that uses the symbol. For example:

# models.py:
from django.db import models
from mycms.managers import PageManager

class Page(models.Model)
    PUBLISHED = 1

    objects = PageManager()

    # ....


# managers.py:
from django.db import models

class PageManager(models.Manager):
    def published(self):
        from mycms.models import Page   # Import here to prevent circular imports
        return self.filter(status=Page.PUBLISHED)

In this case, you can see models.py really needs to import managers.py; without it, it can't do the static initialisation of PageManager. The other way around is not so critical. The Page model could easily be imported inside a function instead of globally.

The same applies to any other situation of import errors. The loop may include a few more packages however.