49 Code in other files

Every useful program is spread across many files, and import is the one statement that puts them back together.

import sandbox_widget

Everything you have written so far has lived in a single file. That was fine while a program was forty lines long, and it stops being fine somewhere around two hundred. But there is a bigger point hiding behind the practical one. You have been calling print and len since week one without ever asking where they live. You have been running tests written by somebody else against code written by you, in two different files, and somehow the tests could see your functions. Every one of those things happened because of a single statement, which you have typed a few times already without being told what it does.

This chapter is about what import actually does. It is a short chapter, because the answer is short. It is also the chapter that quietly unlocks the second half of the course, because from here on you will be using code that other people wrote, and you should know what is happening when you do.

A module is a file

Make a folder. Put two files in it. Type them both; do not copy them.

seqtools.py

def gc_content(dna):
    "Fraction of the bases that are G or C."
    gc = dna.count('G') + dna.count('C')
    return gc / len(dna)

analyse.py

import seqtools

print(seqtools.gc_content('ATGC'))

Now run the second one from the terminal, the way you have been running programs all term:

Terminal

$ python analyse.py
0.5

A function defined in one file was called from another one. That is the whole feature, and you have just used it.

What you wrote is called a module. A module is a file with Python in it. That is the entire definition, and there is nothing else to it. seqtools.py is a module whose name is seqtools — the file name without the .py. Because that name becomes a variable in the program that imports it, it has to be a name you could legally give a variable: no dashes, no spaces, and it cannot start with a digit. seq_tools.py is fine. seq-tools.py will not import, ever, and the error will not be helpful about why.

What `import` actually does

Three things, in this order.

First it finds the file. It looks in the folder you are running from, and then in the places where installed libraries live, and it takes the first seqtools it finds.

Second it runs the file. Top to bottom, every line, exactly as if you had typed python seqtools.py yourself. Each def runs and creates a function object; each assignment at the left margin runs and creates a variable.

Third it binds a name. When the file has finished running, everything it created is packed into a single object, and that object is assigned to the variable seqtools in your program.

The third one is the one that makes the rest make sense: import seqtools is an assignment statement. Afterwards, seqtools is an ordinary variable holding an ordinary object, and you can look at it. Run the code below:

%%sandbox
import seqtools

print(type(seqtools))
print(seqtools)

It reports <class 'module'> and then tells you which file it came from. So seqtools.gc_content is not new syntax. It is the dot from Chapter 20, doing what it always does: “Hey module, give me your gc_content!” A module is an object whose attributes are the things defined in its file.

what you write	what you get
`import seqtools`	the name `seqtools`
`import seqtools as st`	the name `st`
`from seqtools import gc_content`	the name `gc_content`

Every one of them runs the whole file. They differ only in what name is left behind.

The middle step, the one where the file runs, is the one that surprises people. Try it. Put a line at the left margin of the module, above the def:

seqtools.py

print("seqtools is being imported")

def gc_content(dna):
    "Fraction of the bases that are G or C."
    gc = dna.count('G') + dna.count('C')
    return gc / len(dna)

Run analyse.py again and that sentence appears, without anything in analyse.py asking for it. It happened because importing a module executes it. Anything sitting at the left margin of a module runs the moment somebody imports it, whether they wanted it to or not.

Exercise 49-1

SOLO

Put print("analyse is starting") as the very first line of analyse.py, and then write import seqtools twice, on two lines in a row. Decide what you think the program prints, and in what order, and how many times, before you run it. Then run it. One part of the result is going to be different from what you predicted, and the explanation is that Python keeps a note of every module it has already imported and refuses to run the same file twice.

Important

Because importing a module runs it, a module should contain definitions, not actions. Functions, classes, and constants: yes. A loop over your whole genome that takes five minutes: no, not at the left margin, because it will happen to anybody who imports your file for any reason at all.

When a file needs to be both — a module other files can import and a program you can run directly — the fix is one incantation you may as well learn as a recipe:

analyse.py

import seqtools

def main():
    print(seqtools.gc_content('ATGC'))

if __name__ == '__main__':
    main()

Python sets the variable __name__ inside every file it runs. If you ran the file yourself, it is set to '__main__'; if somebody imported the file, it is set to the module’s name instead. So the body of that if runs when you run the file and stays quiet when somebody imports it. That is all it is: a normal if-statement, asking a normal question about a variable somebody else set for you.

The three ways to write it

%%sandbox
import seqtools
print(seqtools.gc_content('ATGC'))

%%sandbox
import seqtools as st
print(st.gc_content('ATGC'))

%%sandbox
from seqtools import gc_content
print(gc_content('ATGC'))

All three run the whole file. They differ only in the name you are left holding.

The first is the honest one, and it should be your default. Six months from now, seqtools.gc_content(dna) still tells the reader where that function came from, and the reader might be you.

The second is the same thing with a shorter name, and you use it when the community has agreed on an abbreviation, the way everyone writes import pandas as pd. Do not invent your own abbreviations for well-known libraries; you will only confuse people who already know the conventional one.

The third reaches into the module and copies one name out of it, so you can use it without saying where it came from. That is convenient, and the convenience is exactly the cost: a file full of bare function names gives a reader no way to tell which ones you wrote and which ones arrived from somewhere else. Use it sparingly, for a handful of names you use constantly, the way from Bio.Seq import Seq is written by everybody.

There is a fourth form, from seqtools import *, which empties the entire module into your program. It looks like a shortcut and is a trap: any name in the module that happens to match a name of yours silently replaces it, and you find out much later, in the form of a function that does something you did not write. Do not use it.

You have been importing your own code all term

Open the test file from any of the projects. You will find functions that look like this:

@requires("find_orfs")
def test_find_orfs(module):
    assert module.find_orfs("ATGAAATAGAAATGAAATAGTAA") == [[0, 6], [11, 17]]

There is that dot again, and now you can read the line properly. The test suite imports your project file as a module, and hands the resulting module object to each test as the argument module. Then module.find_orfs is the test asking your module for the function of that name, exactly the way analyse.py asked seqtools for gc_content.

That also explains a message you have seen. When the test report says a function is not written yet, nothing clever is going on: the tests looked up a name in your module, the name was not there, and the report says so. Your file has been a module all along. Nobody mentioned it, because until now you did not need to know.

Modules that came with Python

A large pile of modules arrived on your computer along with Python itself, and you can import any of them without installing anything. This is called the standard library, and it is one of the reasons people like Python.

%%sandbox
import math

print(math.sqrt(16))
print(math.log(100, 10))
print(math.floor(3.7))

random gives you random choices and shuffles, statistics gives you means and medians, os and pathlib deal with files and folders, and csv reads the comma-separated tables from Chapter 60 without pandas being involved at all. You do not need to learn them; you need to know they exist, so that you check before writing something that already exists.

Important

Do not call one of your own files math.py. Or random.py, or csv.py, or string.py.

You now know exactly why, which is the point of putting this warning here rather than earlier. Import looks in your folder first. If there is a math.py sitting next to your program, that is the math that gets imported, yours, and the real one becomes unreachable — and the error you get will be about sqrt not existing, which points at the wrong file entirely. It is a genuinely nasty afternoon. Give your files names that are about your problem: seqtools.py, orffinder.py, translation.py.

A package is a folder of modules

Once a library grows past what fits comfortably in one file, it becomes a folder of files. A folder of modules is called a package, and the dots in an import statement are steps down into it:

from Bio.Seq import Seq

Read it left to right. Go into the package Bio. Inside it, find the module Seq. Out of that module, take the name Seq, which here is a class. The fact that the module and the class have the same name is unfortunate and very common; the dots tell you which is which.

That is all a package is. pandas is a package. Bio is a package. Both are folders full of ordinary .py files that somebody else wrote, sitting somewhere on your disk, containing functions and classes defined with exactly the def and class statements you already know.

Which also demystifies installing. When pixi, conda or pip installs a package, it downloads that folder and puts it somewhere import already knows to look. There is no other magic step. It is copying files into a place on a list.

Two errors worth telling apart

ModuleNotFoundError: No module named 'Bio'

Import went looking for a file and found nothing. Not installed, or misspelled, or you are running from the wrong folder.

ImportError: cannot import name 'reverse_translate' from 'Bio.Seq'

Import found the module. The name you asked for is not in it.

Those two errors are worth being able to tell apart at a glance, and the second one is about to become important. An assistant that has read a great deal of code will happily offer you from Bio.Seq import reverse_translate, because that is the kind of thing a sequence library plausibly contains, and plausible is what these models are good at. The module exists, so the import gets that far; the name does not, so it stops there. That ImportError is the machine telling you the assistant made something up. Chapter 46 is about learning to catch it a step earlier, by reading what the library actually promises.

Exercise 49-2

AI: Drafter

Ask the assistant for a function from the math module that you suspect does not exist — something plausible, like a function that returns the greatest common divisor of a list, or one that rounds to a given number of significant figures. Ask it for the exact import statement.

Then run the import. Whatever happens, you learn something: either the function exists and you have found a useful one, or you get an ImportError naming precisely what was invented. Write both the claim and the outcome in your logbook.

Splitting a project of your own

The same feature that lets you use other people’s code lets you organise your own, and it is the same reason you started writing functions in the first place. A function gives a name to a piece of a program. A module gives a name to a group of functions.

Take the ORF project. By the end it holds a codon table, a translation function, several ORF-finding functions and something that runs the whole analysis. Three groups, so three files:

translation.py

CODON_MAP = {'TTT': 'F', 'TTC': 'F'}   # rest of the table here...

def translate_orf(orf):
    # rest of code here...

orffinder.py

def find_start_positions(seq):
    # rest of code here...

def find_orfs(seq):
    # rest of code here...

main.py

import translation
import orffinder

def report(genome):
    for start, end in orffinder.find_orfs(genome):
        print(translation.translate_orf(genome[start:end]))

if __name__ == '__main__':
    report('ATGAAATAGAAATGAAATAGTAA')

Three rules and you have everything you need. The files go in the same folder. Each module holds definitions, not actions. And the arrows point one way: main imports the other two, and neither of them imports main. If two modules import each other, Python has to start running one of them before the other has finished, and the resulting error is as confusing as it sounds. Keep the arrows pointing downhill and it cannot happen.

What you get for the trouble is what you got from functions, one level up. A name for each part. A file you can read without holding the other two in your head. And a piece you can test on its own, which is going to matter more and more from here.

Exercise 49-3

AI: Drafter

Take a program you have already written that has at least two functions in it. Decide yourself, on paper, which functions belong together and what the two files should be called — that decision is the part worth doing, and it is the part the assistant cannot do for you, because it does not know what your program is for.

Then ask the assistant to perform the split for you, telling it exactly which functions go where. Run the result. If it does not run, read the error before asking for a fix: nine times out of ten it is a missing import, and being able to say that out loud before anyone tells you is the whole point of this chapter.

What comes next

You now know that the code you import is code, in files, on your disk, no different in kind from the files you write. It was written with the same def statements, by people who were also once confused about import.

That matters because of what happens next. In Chapter 46 you meet BioPython, a package containing careful, tested versions of several functions you spent a week writing by hand. Handing that work over is not a defeat; doing it once yourself is what makes you able to judge somebody else’s version. And the question that chapter asks — how do I know this library does what it claims? — is the same question you will be asking about every piece of code an assistant hands you for the rest of the term.

Exercise 49-4