43  Modules and packages

Using code in other files lets you stand on the shoulders of giants.

import sandbox_widget

Python code can be shared between files. That is handy if you want to split a large program you are writing across multiple files. A file you import code from is called a module. It is easier to show than to explain. Put two files in the same folder that this notebook is in. Give each file the name and content as shown below:

seqtools.py

def gc_content(dna):
    "Fraction of the bases that are G or C."
    gc = dna.count('G') + dna.count('C')
    return gc / len(dna)

analyze.py

import seqtools

gc = seqtools.gc_content('ATGC')
print(gc)

In the terminal, navigate to the folder and run the code in second one, just like you usually do:

Terminal

python analyze.py
0.5

The import statement in analyze.py imported the content of seqtools.py as a module, making it available as the name seqtools. seqtools.gc_content('ATGC') used the . operator to access and call the gc_content function defined in that module (file). The . operator can be used to access things inside a module just like we use it to access methods and attributes inside an object: “Hey module, give me your gc_content!”

A module is just a file with Python code in it. It does not mean that all files with Python code are imported from other files, but when they are we call them modules.

If the code for your program is distributed across many files, several of the files may need the gc_content function. Rather than duplicating the definition of gc_content in each file, each file that needs it can just import it from the definition in seqtools.py.

seqtools.py is a module whose name is seqtools, the file name without the .py. Because that name becomes a variable in the program that imports it, the file name has to be one that you can legally give a variable. I.e., no dashes, no spaces, and it cannot start with a digit. seq_tools.py is fine. seq-tools.py will not work and neither will best seq tool file ever.py.

When python runs the import statement, the following happens in this order:

1. It finds the file. It looks in the folder you are running from, and then in the places where installed libraries live, and it takes the first seqtools it finds.
2. It runs the file exactly as if you had typed python seqtools.py yourself. Each def defines a function, and each assignment at the left margin runs and creates a variable.
3. It binds a name. When the file run, all definitions are packed into a single module object named after the imported file.

So import seqtools is really an assignment statement. It assigns the content of seqtools.py to the variable seqtools. The type of value is module, so you can add it to the list of Python types you know by now. See for yourself by running the cell below.

%%sandbox
import seqtools

print(type(seqtools))
print(seqtools)

The cell imported seqtools, this time into this notebook, and show you that the type of module is <class 'module'>. Printing the module object also shows you which file it came from.

Exercise 43-1

SOLO

The middle step (nr 2.) that runs the file often surprises people. Change the content of seqtools.py by adding a print statement at the left margin of the file:

seqtools.py

def gc_content(dna):
    "Fraction of the bases that are G or C."
    gc = dna.count('G') + dna.count('C')
    return gc / len(dna)

print("seqtools is being imported")    

Then run python analyze.py again and the “seqtools is being imported” sentence appears in the terminal. There is noting in analyse.py that asks for it to be printed. It happens anyway because importing a module executes its contents. Anything sitting at the left margin of a module runs the moment somebody imports it, whether they wanted it to or not.

Exercise 43-2

SOLO

Put print("analyse is starting") as the very first line of analyse.py, and then write import seqtools twice, on two lines in a row. Decide what you think the program prints, and in what order, and how many times, before you run it. Then run it. One part of the result is going to be different from what you predicted, and the explanation is that Python keeps a note of every module it has already imported and refuses to run the same file twice.

Important

Because importing a module runs it, a file used as a module should contain definitions of functions, classes, and constants, but not expensive actions. Putting a for loop at the left margin that loops over all bases in the human genome will make it do just that every time you import it, making the import statement take five minutes rather than milliseconds.

Three variants of the import statement

%%sandbox
import seqtools
print(seqtools.gc_content('ATGC'))

The first one above should be your default because it most explicitly shows where the function comes from when you call it.

%%sandbox
import seqtools as st
print(st.gc_content('ATGC'))

The is the same thing, but assigning the imported module to a shorter name like st. Use only if the module name is really long or when the community has agreed on an abbreviation, the way everyone writes import pandas as pd.

%%sandbox
from seqtools import gc_content
print(gc_content('ATGC'))

This last version reaches into the module and copies one name out of it, so you can use it without saying where it came from. However, a file full of bare function names makes it difficult to see which one you wrote and which ones were imported. Use mainly to import a handful of recognizable classes like from Bio.Seq import Seq.

(There is actually a fourth form, from seqtools import *, which empties the entire module into your program. Do not use that, as you may unwittingly import definitions that replace your own.)

what you write	what you get
import seqtools	the name seqtools
import seqtools as st	the name st
from seqtools import gc_content	the name gc_content

Every one of them runs the whole file. They differ only in what name is left behind.

Modules in Python’s Standard Library

The Python installation comes with bunch of modules all part of the standard library. Check if something is already available in the standard library before you start writing your own. One example is the math module:

%%sandbox
import math

print(math.sqrt(16))
print(math.log(100, 10))
print(math.floor(3.7))

Important

Do not call one of your own files math.py.

Import looks in your current folder first. So if a math.py is sitting next to your program, that is the math that gets imported. I.e., yours not the one from the standard library. That also goes for random, string, csv and many nore. So give your files specific names like seqtools.py or orffinder.py that are unlikely to clash with builtin modules.

A package is a folder of modules

Once a library grows past what fits comfortably in one file, it becomes a folder of files. A folder of modules is called a package, and the dots in an import statement are steps down into it:

from Bio.Seq import Seq

Read it left to right. Go into the package Bio. Inside it, find the module Seq. Out of that module, take the name Seq, which here is a class. The fact that the module and the class have the same name is unfortunate but very common.

As you will soon learn, pandas is a package and so is Bio. You installed both at the beginning of the course. Both are folders full of ordinary .py files that somebody else wrote, sitting somewhere on your disk, containing functions and classes defined with exactly the def and class statements you already know.

When you use pixi, conda, pip, or uv to install a package, it downloads that folder and puts it somewhere import already knows to look.

Two errors worth telling apart

ModuleNotFoundError: No module named 'Bio'

Python could not find the module you tried to import, either because it is not installed or because you are running from the wrong folder.

ImportError: cannot import name 'reverse_translate' from 'Bio.Seq'

Python found the module, but the name you asked for is not in it.

Exercise 43-3

AI: Drafter

Ask the assistant for a function from the math module that you suspect does not exist — something plausible, like a function that returns the greatest common divisor of a list, or one that rounds to a given number of significant figures. Ask it for the exact import statement.

Then run the import. Whatever happens, you learn something: either the function exists and you have found a useful one, or you get an ImportError naming precisely what was invented. Write both the claim and the outcome in your logbook.

Splitting a project of your own

Splitting code across files is a way to provide order and structure, in much the same way you do when you isolate code into functions. Take the ORF project. By the end it holds a codon table, a translation function, several ORF-finding functions and something that runs the whole analysis. Three groups, so three files:

translation.py

CODON_MAP = {'TTT': 'F', 'TTC': 'F'}   # rest of the table here...

def translate_orf(orf):
    # rest of code here...

orffinder.py

def find_start_positions(seq):
    # rest of code here...

def find_orfs(seq):
    # rest of code here...

main.py

import translation
import orffinder

def report(genome):
    for start, end in orffinder.find_orfs(genome):
        print(translation.translate_orf(genome[start:end]))

if __name__ == '__main__':
    report('ATGAAATAGAAATGAAATAGTAA')

The files go in the same folder. Each module holds definitions, not actions. main imports the other two, and neither of them imports main. If two modules import each other, Python has to start running one of them before the other has finished, and the resulting error is as confusing as it sounds.

Exercise 43-4

AI: Drafter

Take a program you have already written that has at least two functions in it. Decide yourself, on paper, which functions belong together and what the two files should be called — that decision is the part worth doing, and it is the part the assistant cannot do for you, because it does not know what your program is for.

Then ask the assistant to perform the split for you, telling it exactly which functions go where. Run the result. If it does not run, read the error before asking for a fix: nine times out of ten it is a missing import, and being able to say that out loud before anyone tells you is the whole point of this chapter.