Python import: Advanced Techniques and Tips

Modules

The Python.org glossary defines module as follows:

An object that serves as an organizational unit of Python code. Modules have a namespace containing arbitrary Python objects. Modules are loaded into Python by the process of importing. (Source)

In practice, a module usually corresponds to one .py file containing Python code.

The true power of modules is that they can be imported and reused in other code. Consider the following example:

>>> import math
>>> math.pi
3.141592653589793

In the first line, import math, you import the code in the math module and make it available to use. In the second line, you access the pi variable within the math module. math is part of Python’s standard library, which means that it’s always available to import when you’re running Python.

Note that you write math.pi and not just simply pi. In addition to being a module, math acts as a namespace that keeps all the attributes of the module together. Namespaces are useful for keeping your code readable and organized. In the words of Tim Peters:

Namespaces are one honking great idea—let’s do more of those! (Source)

You can list the contents of a namespace with dir():

>>> import math
>>> dir()
['__annotations__', '__builtins__', ..., 'math']

>>> dir(math)
['__doc__', ..., 'nan', 'pi', 'pow', ...]

Using dir() without any argument shows what’s in the global namespace. To see the contents of the math namespace, you use dir(math).

You’ve already seen the most straightforward use of import. However, there are other ways to use it that allow you to import specific parts of a module and to rename the module as you import it.

The following code imports only the pi variable from the math module:

>>> from math import pi
>>> pi
3.141592653589793

>>> math.pi
NameError: name 'math' is not defined

Note that this places pi in the global namespace and not within a math namespace.

You can also rename modules and attributes as they’re imported:

>>> import math as m
>>> m.pi
3.141592653589793

>>> from math import pi as PI
>>> PI
3.141592653589793

For more details about the syntax for importing modules, check out Python Modules and Packages – An Introduction.

Packages

You can use a package to further organize your modules. The Python.org glossary defines package as follows:

A Python module which can contain submodules or recursively, subpackages. Technically, a package is a Python module with an __path__ attribute. (Source)

Note that a package is still a module. As a user, you usually don’t need to worry about whether you’re importing a module or a package.

In practice, a package typically corresponds to a file directory containing Python files and other directories. To create a Python package yourself, you create a directory and a file named __init__.py inside it. The __init__.py file contains the contents of the package when it’s treated as a module. It can be left empty.

In general, submodules and subpackages aren’t imported when you import a package. However, you can use __init__.py to include any or all submodules and subpackages if you want. To show a few examples of this behavior, you’ll create a package for saying Hello world in a few different languages. The package will consist of the following directories and files:

world/
│
├── africa/
│   ├── __init__.py
│   └── zimbabwe.py
│
├── europe/
│   ├── __init__.py
│   ├── greece.py
│   ├── norway.py
│   └── spain.py
│
└── __init__.py

Each country file prints out a greeting, while the __init__.py files selectively import some of the subpackages and submodules. The exact contents of the files are as follows:

# world/africa/__init__.py  (Empty file)

# world/africa/zimbabwe.py
print("Shona: Mhoroyi vhanu vese")
print("Ndebele: Sabona mhlaba")

# world/europe/__init__.py
from . import greece
from . import norway

# world/europe/greece.py
print("Greek: Γειά σας Κόσμε")

# world/europe/norway.py
print("Norwegian: Hei verden")

# world/europe/spain.py
print("Castellano: Hola mundo")

# world/__init__.py
from . import africa

Note that world/__init__.py imports only africa and not europe. Similarly, world/africa/__init__.py doesn’t import anything, while world/europe/__init__.py imports greece and norway but not spain. Each country module will print a greeting when it’s imported.

Let’s play with the world package at the interactive prompt to get a better understanding of how the subpackages and submodules behave:

>>> import world
>>> world
<module 'world' from 'world/__init__.py'>

>>> # The africa subpackage has been automatically imported
>>> world.africa
<module 'world.africa' from 'world/africa/__init__.py'>

>>> # The europe subpackage has not been imported
>>> world.europe
AttributeError: module 'world' has no attribute 'europe'

When europe is imported, the europe.greece and europe.norway modules are imported as well. You can see this because the country modules print a greeting when they’re imported:

>>> # Import europe explicitly
>>> from world import europe
Greek: Γειά σας Κόσμε
Norwegian: Hei verden

>>> # The greece submodule has been automatically imported
>>> europe.greece
<module 'world.europe.greece' from 'world/europe/greece.py'>

>>> # Because world is imported, europe is also found in the world namespace
>>> world.europe.norway
<module 'world.europe.norway' from 'world/europe/norway.py'>

>>> # The spain submodule has not been imported
>>> europe.spain
AttributeError: module 'world.europe' has no attribute 'spain'

>>> # Import spain explicitly inside the world namespace
>>> import world.europe.spain
Castellano: Hola mundo

>>> # Note that spain is also available directly inside the europe namespace
>>> europe.spain
<module 'world.europe.spain' from 'world/europe/spain.py'>

>>> # Importing norway doesn't do the import again (no output), but adds
>>> # norway to the global namespace
>>> from world.europe import norway
>>> norway
<module 'world.europe.norway' from 'world/europe/norway.py'>

The world/africa/__init__.py file is empty. This means that importing the world.africa package creates the namespace but has no other effect:

>>> # Even though africa has been imported, zimbabwe has not
>>> world.africa.zimbabwe
AttributeError: module 'world.africa' has no attribute 'zimbabwe'

>>> # Import zimbabwe explicitly into the global namespace
>>> from world.africa import zimbabwe
Shona: Mhoroyi vhanu vese
Ndebele: Sabona mhlaba

>>> # The zimbabwe submodule is now available
>>> zimbabwe
<module 'world.africa.zimbabwe' from 'world/africa/zimbabwe.py'>

>>> # Note that zimbabwe can also be reached through the africa subpackage
>>> world.africa.zimbabwe
<module 'world.africa.zimbabwe' from 'world/africa/zimbabwe.py'>

Remember, importing a module both loads the contents and creates a namespace containing the contents. The last few examples show that it’s possible for the same module to be part of different namespaces.

>>> import math
>>> math.__dict__["pi"]
3.141592653589793

It’s fairly common to import subpackages and submodules in an __init__.py file to make them more readily available to your users. You can see one example of this in the popular requests package.

Absolute and Relative Imports

Recall the source code of world/__init__.py in the earlier example:

from . import africa

You’ve already seen from...import statements such as from math import pi, but what does the dot (.) in from . import africa mean?

The dot refers to the current package, and the statement is an example of a relative import. You can read it as “From the current package, import the subpackage africa.”

There’s an equivalent absolute import statement in which you explicitly name the current package:

from world import africa

In fact, all imports in world could have been done explicitly with similar absolute imports.

Relative imports must be in the form from...import, and the location you’re importing from must start with a dot.

The PEP 8 style guide recommends using absolute imports in general. However, relative imports are an alternative for organizing package hierarchies. For more information, see Absolute vs Relative Imports in Python.

Python’s Import Path

How does Python find the modules and packages it imports? You’ll see more details about the mechanics of the Python import system later. For now, just know that Python looks for modules and packages in its import path. This is a list of locations that are searched for modules to import.

You can inspect Python’s import path by printing sys.path. Broadly speaking, this list will contain three different kinds of locations:

1. The directory of the current script (or the current directory if there’s no script, such as when Python is running interactively)
2. The contents of the PYTHONPATH environment variable
3. Other, installation-dependent directories

Typically, Python will start at the beginning of the list of locations and look for a given module in each location until the first match. Since the script directory or the current directory is always first in this list, you can make sure that your scripts find your self-made modules and packages by organizing your directories and being careful about which directory you run Python from.

However, you should also be careful that you don’t create modules that shadow, or hide, other important modules. As an example, say that you define the following math module:

# math.py

def double(number):
    return 2 * number

Using this module works as expected:

>>> import math
>>> math.double(3.14)
6.28

But this module also shadows the math module that’s included in the standard library. Unfortunately, that means our earlier example of looking up the value of π no longer works:

>>> import math
>>> math.pi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'math' has no attribute 'pi'

>>> math
<module 'math' from 'math.py'>

The problem is that Python now searches your new math module for pi instead of searching the math module in the standard library.

To avoid these kinds of issues, you should be careful with the names of your modules and packages. In particular, your top-level module and package names should be unique. If math is defined as a submodule within a package, then it won’t shadow the built-in module.

Example: Structure Your Imports

While it’s possible to organize your imports by using the current directory as well as by manipulating PYTHONPATH and even sys.path, the process is often unruly and prone to errors. To see a typical example, consider the following application:

structure/
│
├── files.py
└── structure.py

The app will re-create a given file structure by creating directories and empty files. The structure.py file contains the main script, and files.py is a library module with a few functions for dealing with files. The following is an example of output from the app, in this case by running it in the structure directory:

$ python structure.py .
Create file: /home/gahjelle/structure/001/structure.py
Create file: /home/gahjelle/structure/001/files.py
Create file: /home/gahjelle/structure/001/__pycache__/files.cpython-38.pyc

The two source code files as well as the automatically created .pyc file are re-created inside a new directory named 001.

Now take a look at the source code. The main functionality of the app is defined in structure.py:

 1# structure/structure.py
 2
 3# Standard library imports
 4import pathlib
 5import sys
 6
 7# Local imports
 8import files
 9
10def main():
11    # Read path from command line
12    try:
13        root = pathlib.Path(sys.argv[1]).resolve()
14    except IndexError:
15        print("Need one argument: the root of the original file tree")
16        raise SystemExit()
17
18    # Re-create the file structure
19    new_root = files.unique_path(pathlib.Path.cwd(), "{:03d}")
20    for path in root.rglob("*"):
21        if path.is_file() and new_root not in path.parents:
22            rel_path = path.relative_to(root)
23            files.add_empty_file(new_root / rel_path)
24
25if __name__ == "__main__":
26    main()

In lines 12 to 16, you read a root path from the command line. In the above example you use a dot, which means the current directory. This path will be used as the root of the file hierarchy that you’ll re-create.