12  Notebooks

Scripts versus notebooks

You have already written your first program in VScode. You saved the code to a file with a .py suffix and then used the terminal to have Python run your code. That is one way of running Python, and it is the way most Python code in the world gets run. This chapter shows you the other way, which better fits the purpose of this course. the one you are looking at right now, because this page is itself a running Python program. The two ways are not the same, and the places where they differ are exactly the places where beginners get confused. So let us take them apart. Your hello.py is a script. It is a plain text file sitting on the disk with some Python code written in it. To run it, you go to the terminal (activate the pixi environment with pixi shell) and type:

Terminal

python hello.py

The program called python is the interpreter. It opens your file, reads the first line, does what it says, reads the next line, does what that one says, and carries on until there are no more lines. Then it stops and exits. So the order the lines appear in the file is the order in which they run. When the Python is finished running your code, everything it was holding is thrown away, because it was all in memory, and memory is emptied the moment a program ends. Because Python starts from a blank slate every time it runs a script, you are guaranteed to get the same result every time it is run, whether on your computer or someone elses.

Notebooks in VScode

The notes you are now reading are also notebooks you can run. Reading this page in a browser will teach you nothing, because this chapter is about clicking things. The week1 notebooks are already in the course folder, in the week1 subfolder. Open the course folder in vscode and expand the week1 folder to show its contents. Then double-click the notebooks.ipynb file to open it and work through it there. Everything below assumes you are looking at the real notebook with a cursor in it.

NoteDownloading notebooks during the course.

Every other chapter you fetch when we get to it, so that the notebook you work in is always the current version of the chapter you are reading. There is a Download notebook button in the margin of every chapter on the website so you can download the notebook and put it in the course folder.

I also made you an easier way to get the notebooks. Say you want the notebook for the “Iteration” chapter, all you need to do is navigate to the week1 folder in your course folder and run im get iteration:

Terminal

im get iteration

To see what you can ask for other than iteration, run it without a name:

Terminal

im get

If you already have a notebook of that name, yours is left exactly as it is and the fresh copy arrives beside it as iteration-2.ipynb. Nothing you have written is ever overwritten, so it is always safe to ask for a clean copy.

Inside the notebook

Open the instructing-machines course folder in VScode (“Open folder”) and double-click the iteration.ipynb in the “Explorer” panel in the left side panel. Now look at where you are. This page is a Jupyter notebook, a file whose name ends in .ipynb, not .py. Instead of a single monolithic chunk of code in a script, a notebook is made up of individual cells. Some cells are markdown cells with text, like this one. Other cells are Python cells with code and can be run one at a time in any order.

A notebook does not run by itself. It needs a Python to run on, and VS Code calls that Python the kernel. Look at the top right corner of the notebook. If it says Select Kernel, click it. In the dropdown it should be the only one listed under Jupyter Kernel.... If there is no Jupyter Kernel..., pick the one recommended under Python Environments.... As a last resort, you can type this in .pixi/envs/default/bin/python. Do not pick “+ Create Python Environment”, even though VS Code may offer it first. You already have the environment you need. The other entries in that list are Pythons that came with your machine, or with something else you once installed. None of them has the course widgets. A notebook running on the wrong Python looks perfectly happy right up until an exercise shows up as blank space where a widget should be.

You only do this once for each notebook. From here on VS Code remembers the choice for this folder and makes it for you every time you open a notebook here.

Now run the cell below. Click it and press Shift-Enter. You can also click the small “Play” icon that appears if you hover over the right margin of the cell.

dna = "ATGTGCAT"
len(dna)

When you run a code cell two things happen. A small number appeared in brackets to the left of the cell, and the length of the string appeared underneath it even though you never wrote print. Both of these are important as you will see in a bit.

Now run the next cell.

dna.count("A")

That cell never mentions what dna is, and yet it worked. The reason is that there is a Python process that has been running the entire time, and still is. It is called the python kernel. When you run a cell, the code in that cell is handed to that same persistent python process, which runs the code then waits for whatever you hand it next. The first cell put a string into that process and gave it the name dna. The second cell asked the same process about a name it already knew.

So a notebook is not a file that the interpreter reads from top to bottom. It is a pile of small pieces of code that you hand, one piece at a time, to a session that remembers everything you have handed it so far.

The number in brackets is the execution count, and it tells you the order in which you handed things over. The first cell you run in a session gets a 1, the next gets a 2, and so on. Run the same cell again and its number goes up. This is the notebook quietly telling you something important, which is that the kernel remembers the order you ran cells in, not the order the cells sit on the page. You can run the bottom cell first, then the top one, then the bottom one again, and the kernel will happily oblige. You can even delete a cell after running it, and the names it defined will still be sitting in the kernel, alive and usable, with nothing on the page left to explain where they came from.

That is the notebook’s great convenience and its great trap. It is a convenience because you can try one small thing, look at the answer, and try the next small thing, without rerunning your whole program each time. It is a trap because what you see on the page can stop matching what the kernel actually holds.

The other loose end was the length appearing without print. When the last line of a cell is a bare expression, something that reduces to a value, the notebook displays that value for you. Only the last line, and only if it is an expression. An assignment like dna = "ATG" produces no value, so nothing is shown. This is a courtesy of the notebook, not a rule of Python. Try to run the cell below:

dna

From the top menu, choose View -> Command Palette, type “jupyter variables” and select “Jupyter: Open Jupyter Variables View”. This opens a panel showing the defined variables and their current values.

Exercise 12-1

SOLO

Decide what will happen before you touch anything. Scroll back up to the cell that defines dna, change "ATGTGCAT"' the string to "GGGGG", and run only that cell. Then run the cell that counts As again. What number do you expect, and what do you get? Now change "GGGGG" back to "ATGTGCAT"' without rerunning that cell. Now what happens if you run the cell that counts As? What do you get, and why?

Exercise 12-2

SOLO

Find the command in your editor that restarts the kernel and runs all the cells from the top. In VScode it sits at the top of the notebook. Predict what will happen to the execution counts before you use it, then use it and check.

Two kinds of cells

A notebook is a stack of cells, and there are two kinds.

Text cells hold prose, written in a simple formatting language called Markdown. The paragraph you are reading is one of them. When a text cell is finished it is rendered, meaning you see the formatted result rather than the raw text you typed. Double-click one and it turns back into editable text; press Shift and Enter together and it renders again.

Code cells hold Python. They have a run button on the left and a slot on the left of that where a number appears after they have been run. Everything in a code cell is Python, which means a line of prose typed into a code cell by mistake will produce a SyntaxError, and a line of Python typed into a text cell by mistake will simply sit there looking like a sentence and never run. Both mistakes are common and both are worth recognizing on sight.

If you want Python cell to be a markdown cell or vice-versa, you just click the bottom right corner of the cell. Notice also the controls that appear if you hover over the bottom edge of a cell.

Exercise 12-3

SOLO

Double-click this paragraph. You should see the raw Markdown, with the exercise heading above it starting with four hash marks. Add a sentence of your own at the end. Press Shift and Enter to render it back. Now double-click it again and remove your sentence. You have just done the two things you will do to every text cell for the rest of the term.

Restarting the kernel

Because the kernel keeps everything you have run, it will happily keep definitions you have since changed or deleted from the page. E.g. a value from an experiment you abandoned twenty minutes ago. Code that depends on such updated cells may look fine, but they won’t work the next time you open and run the notebook.

Restarting the kernel throws all of it away and gives you a fresh interpreter with nothing in it. Your cells and their text are untouched; only the memory is emptied. The Restart button is in the toolbar along the top of the notebook, next to a Clear Outputs button that wipes the displayed results without restarting anything, and next to Restart and Run All, which restarts and then runs every cell down the page in order.

Restart and Run All is the one that matters. It is the only way to find out whether your notebook does what it appears to do, because it is the only way to run it the way a stranger would. Anything that survives that has been proved to work in page order. Anything that fails was depending on something that is not written down.

Exercise 12-4

SOLO

Run the cell below, so that virus exists in the kernel.

virus = "HIV"
virus

Now press Restart. Then run the cell below, and only that cell. Predict first what will happen, and be specific about the error you expect.

virus.lower()

You should get a NameError saying that virus is not defined, because the restart emptied the kernel and the cell that creates virus has not been run since. Now use Restart and Run All, and watch the whole page run from the top with the counts numbering themselves neatly downward. Both cells work again, because this time they ran in the order they are written.

If a notebook of yours is behaving strangely and you cannot see why, restart and run all before you do anything else. Most of the time the strangeness is stale memory, and most of the rest of the time the restart turns a mystery into an error message you can read.

Running a code cell

There are three ways to run the cell your cursor is in, and the difference between them is only where you end up afterward.

Shift and Enter together runs the cell and moves the cursor to the next one, which is what you want when you are working down a page. Control and Enter together, or Command and Enter on a Mac, runs the cell and leaves the cursor where it is, which is what you want when you are editing one cell repeatedly until it works. Alt and Enter together, or Option and Enter on a Mac, runs the cell and makes a new empty one below it, which is what you want when you are building something up a piece at a time.

Run the cell below with Shift and Enter.

dna = "ATGGTGCATCTGACTCCTGAGGAGAAG"
len(dna)

Two things happened. A number appeared in the brackets to the left of the cell, and the value 27 appeared underneath it.

The number is the execution count. It says which run this was, counting every code cell you have run since the kernel started. The first cell you run gets 1, the next gets 2, and so on. If you run the same cell three times its number changes three times, because the count records runs and not cells.

The value underneath appeared because the last line of the cell was an expression sitting on its own. A notebook shows you the value of the final expression as a courtesy, without you asking. This is a notebook habit rather than a Python rule: put len(dna) on its own line in a script and nothing at all appears on your screen.

Exercise 12-5

SOLO

Run the cell above two more times and watch the number in the brackets. Predict what it will say before each run. Then add a second line dna.count("A") below the len(dna) line, run it once more, and see which of the two values gets displayed. Work out the rule from what you observe.

Order on the page and order in time

This is the part of notebooks that trips up beginners so it it is worth getting straight right away. As opposed to a a script that always runs from top to bottom, A notebook runs in whatever order you ran its cells. The cells are laid out down the page, but the kernel remembers them in the order you ran them, and those two orders have nothing to do with each other. A notebook where you have been jumping around can show a page that reads perfectly and produces results that nobody could reproduce, including you, because the state that produced them came from a sequence of clicks that is not written down anywhere. The execution counts are the only record you have of that sequence. When you look the notebook and the counts down the page read 4, 1, 9, 2, you see the order in which cells were run.

codon_count = 64
codon_count

Exercise 12-6

SOLO

Click into the dna cell further up the page and run it again. Then come back down here and run the codon_count cell. Look at the execution counts on the two cells and confirm that the higher number is now on the cell that appears earlier on the page. That is a notebook telling you the truth about itself, and it is the only warning you will get.

Restart and run all

The exercise you just did is a useful habit around notebooks. Restarting the kernel throws away the running process and starts a fresh one that knows nothing. Running all the cells afterward feeds them back in, from the top, in page order. In other words, restarting and running all cells in order makes your notebook behave like a script.

A notebook should also be able to run its cells in order, so always do this before you believe your notebook works, and certainly before you hand one to anybody else. A notebook that only works because of something you ran a cell twenty minutes ago that was then deleted will fail for the next person who opens it, and that next person is often you, next week.

The sandbox

The python kernel’s long memory is a gift when you are exploring and but can get in your way if you want to do independent exercises in a single notebook. Suppose an exercise asks you to run a piece of code that should fail, so that you can read the error message. If you happen to have defined that name in some other cell half an hour ago, the code will not fail at all, and the lesson evaporates. Or suppose you write a small piece of code, run it, see it work, and conclude that it is correct. It may be correct. It may also be leaning on a name that a cell somewhere above quietly left lying around, in which case it will collapse the moment it is moved anywhere else. What we want, for a lot of the exercises in this book, is a cell that behaves like a script: one that starts from nothing, contains everything it needs, and leaves nothing behind. That is what the sandbox is for.

Run this cell once. It brings in the tool and registers the magic with IPython.

import sandbox_widget

Now run this one.

%%sandbox
print("Hello from a fresh interpreter")

The %%sandbox on the first line is not really Python code, which is why it is called a “cell magic”. Think of it as magic that makes a python cell do something slightly different that it otherwise would. In this course you will also meet %%steps, %%puzzle, %%codelens.

The %%sandbox magic turns a cell into sandbox that from nothing and where you can do anything you like without affecting the rest of the notebook. In practice, it makes the cell work just like a little script. Predict what the next cell will do before you run it. Remember that dna has been sitting in the kernel since the top of the chapter, and that the cell below asks for it by name. This cell is meant to fail.

%%sandbox
print("The sandbox is about to look for dna.")
print(dna)

It printed the first line and then raised a NameError on the second, and both the printed line and the full traceback landed in the box. The reason it fails is that %%sandbox runs the cell in a brand new Python interpreter, a separate process started for that one cell only and thrown away the moment it finishes. Nothing that the notebook knows about is visible inside it. The kernel has a dna, and sandbox cell has never heard of it because it starts from nothing with a new Python.

The barrier runs both ways. Run the next two cells in order.

%%sandbox
codon_count = 64
codon_count

codon_count

The sandbox cell defined codon_count, but “whatever happens in the sandbox, stays in the sandbox” (could also have named it %%Vegas). So a %%sandbox cell is a tiny script that happens to live inside a notebook. It runs from nothing, top to bottom, and it has to carry everything it needs. Anything it defines dies with it, and any mistake it makes is caught and shown to you rather than left to poison the rest of the session. That last part is why so many exercises in this book use it. A cell that is supposed to break can be relied upon to break, every time, for everybody, no matter what you ran before it.

Exercise 12-7

SOLO

The two cells below have the same last line. Decide what each one will show before you run either of them, and write your two predictions down. Then run them from the top.

message = "ATG"
print(message * 3)

%%sandbox
print(message * 3)

Exercise 12-8

SOLO

Write a new cell that starts with %%sandbox and that prints ATGATGATG successfully. You may not change any cell above it. There is only one way to do this, and working out what it is tells you that you have understood the section.

Exercise 12-9

AI: Explainer

Copy the traceback from the failing sandbox cell above, paste it to the assistant, and ask it to explain what went wrong and why. Read its answer carefully. It has not seen this chapter and does not know that the cell was run in a separate process, so watch for the moment it explains the error confidently in terms it has invented. Write down in your logbook what it got right, what it got wrong, and how you knew.

Adding, moving, copying and deleting cells

Hover in the space between two cells and a small toolbar appears offering a new code cell or a new text cell at that spot. That is the easiest way to add one, and it is the way you will use most of the time.

Everything else is faster from the keyboard, and to use the keyboard you need to know about the two modes a cell can be in. When your cursor is inside a cell, typing changes its contents, and that is edit mode. Press Escape and the cursor leaves the cell; the cell is still selected, shown by a bar down its left side, but now the keys are commands rather than text. That is command mode. Press Enter to go back into the cell.

In command mode, A makes a new cell above the selected one and B makes one below. M turns the selected cell into a text cell and Y turns it back into a code cell. C copies the selected cell, X cuts it, and V pastes what you copied below the selection. Pressing D twice, quickly, deletes the selected cell, and Z undoes that, which is a good pair to know in that order.

To move a cell up or down, pick the cell up by its left edge and drag it. Every one of these commands also lives in the Command Palette, which you open with Control and Shift and P together, or Command and Shift and P on a Mac. Typing part of a command name there shows you the command along with whatever keyboard shortcut your copy of VScode has assigned to it, which is the reliable way to answer any shortcut question rather than trusting a list in a book.

Exercise 12-10

SOLO

Do all of this without touching the mouse after the first click. Click on this cell and press Escape to get into command mode. Press B to make a new cell below, press Enter to get into it, and type print("first"). Press Escape, then B again, then Enter, and type print("second"). Run both. Now select the print("first") cell and move it below the other one using the arrows in its toolbar, and run both again in page order to confirm the output changed. Finally select each of your two cells in turn and press D twice to remove them, leaving the page as you found it. If you delete one cell too many, press Z.

What a notebook is on disk

The .ipynb file holds your cells, in page order, along with whatever output each cell produced the last time it ran. That is why you can send someone a notebook and they see your results before running anything, and it is also why the file changes every time you run a cell even if you typed nothing. It is worth knowing when you start putting notebooks into a repository and wonder why a file you only looked at is showing up as modified.

A notebook is a good place to work something out, with prose and code and results interleaved on one page. A .py file is a better place for code that other code depends on, because it starts from nothing every time and therefore says exactly what it needs.

Exercise 12-11

SOLO

Make a new notebook of your own, from the “Command Palette” (Cmd-Shift-P / Ctrl-Shift-P) or from File and New File. Attach the course kernel to it if that does not happen automatically. Give it a text cell with a heading and a sentence saying what it is for, then a code cell with print("hello world") in it.