43 AI: Tests are the contract
You already know how to write a test. In the previous part of the course you learned to make a claim about a function in a form the machine can check, by writing a statement like assert gc_content(‘GGCC’) == 1.0 and letting Python confirm or deny it. This week that skill stops being a way to check your own code and becomes the single most important tool you have for working with an assistant. The idea to hold on to is short: a test is a specification you can execute, and a specification is a contract. Once you see tests that way, the whole business of getting code out of a machine you cannot fully trust becomes manageable, because the test, and not the assistant and not even you, becomes the thing that decides whether the code is right.
Start with the word specification, because it is doing quiet work. Before you can judge whether a piece of code is correct, you have to have decided what correct would mean, and that decision is separate from, and comes before, any code. If you ask an assistant for a function that finds genes, the request has no truth value at all. There is nothing to be right or wrong about, because you have not said what output the function should produce for what input. A specification is exactly that missing thing: a statement of what the result must be for given inputs. When it is written in prose, as a comment or a docstring, it is a promise about what the code will do. When it is written as a test, it is a promise the machine will enforce. That difference is everything. A comment that says the function returns the reverse complement is a promise the code can quietly break, and you saw last week how casually generated code breaks exactly such promises. A test that says assert reverse_complement(‘ATGC’) == ‘GCAT’ is a promise that cannot be quietly broken, because the moment the code fails to keep it the machine says so.
This gives you a discipline, and the discipline is mostly about order. The natural way to work with an assistant is to ask for the code, receive it, glance at it, and hope. That order puts hope where proof should be. The disciplined order is the reverse: first you decide the specification by choosing concrete examples, then you write those examples as tests, and only then do you ask the assistant for the code, and finally you run the tests to see whether the code it produced actually satisfies the contract you wrote. Writing the tests before you see the assistant’s code matters for a reason that is easy to underestimate. If you write the tests afterward, having already seen the plausible looking function the assistant produced, you will, without meaning to, write tests that the function happens to pass. The code will anchor your idea of what correct means, and you will end up checking that the code does what the code does, which proves nothing. Writing the tests first keeps the decision about what correct means where it belongs, with you, made before the assistant had any chance to bias it.
Turning a vague request into a specification is done by choosing examples and working out their answers by hand. Suppose you want the fraction of bases in a strand that are G or C, the GC content. You do not begin by prompting; you begin by writing down that gc_content(‘GGCC’) should be 1.0, that gc_content(‘ATAT’) should be 0.0, and that gc_content(‘GGAT’) should be 0.5, each of which you can compute yourself with certainty. Each such example is one point of the specification, and each becomes one line of your test. The strength of the contract you are writing is exactly the strength of the examples you choose. If every example is gentle, the contract is weak, and a wrong function can satisfy it while still being wrong, which is the happy path trap from last week wearing a different hat. A strong contract includes the awkward inputs on purpose: the empty string, a single base, lowercase letters, a character that should not appear. The nastier your examples, the stronger the promise the machine will hold the code to.
There is a further reason to write the tests first, which is that the act of writing them forces you to make decisions you would otherwise leave to the assistant to guess. The interesting decisions in a specification almost always live in the awkward cases, and the awkward cases are exactly the ones a fluent, confident assistant will silently decide for you, in whatever way its training happened to lean. Consider a function is_gc_rich that is supposed to say whether a strand is GC rich. What should it return for a strand that is exactly half G and C, like GGAT, whose GC content is exactly 0.5? Is a strand that is exactly half GC rich, or not? There is no universal answer; it is a decision, and it is your decision to make. If you write the test first, you are forced to decide it, and you write down assert is_gc_rich(‘GGAT’) == False, or True, deliberately. If you do not, the assistant will pick one silently, using a greater than where you might have wanted a greater than or equal, and you will have no way of knowing whether its choice matches your intention, because you never formed an intention. The same thing happens with a function that returns the most common base in a strand. What should it return when two bases are tied, as G and C are in the strand GC? Run this and watch:
def most_common_base(dna):
return max(set(dna), key=dna.count)
print(most_common_base('GC'))Exercise 43-1
Before you run that, write down which base you expect to come back, and the reason you expect it. Then run it. Now the real question, which is not about the answer at all: could you have predicted it from the code alone, without knowing how set happens to order its elements and how max happens to break a tie? If the answer is no, then the behaviour of this function on a tie is not something you decided; it is something that fell out. Write the single assert you would have to add to take that decision back, and notice that writing it forces you to make the decision you had been avoiding.
The answer you get back is essentially arbitrary, an accident of how max happens to break the tie over a set whose order is not something you should rely on. The point is not which base it returns. The point is that the tie is a real decision about what the function should mean, the assistant will make that decision for you and never mention it, and the only way to take the decision back is to write a test that states what you want to happen on a tie before any code exists. Writing tests first is, in this sense, not really about testing at all. It is about doing the thinking, in advance, about the cases where thinking is actually required.
This is why it is worth calling a test a contract rather than merely a check. A contract says: I do not care how you do the job, but the result must satisfy these conditions. That is the relationship you want with an assistant that you cannot fully trust and whose inner workings you cannot inspect. You own the contract, you write it in a form the machine enforces, the assistant supplies an implementation, and the contract, not your hope and not the assistant’s confidence, decides whether the implementation is acceptable. A useful consequence of this arrangement is that the implementation becomes replaceable. Because the contract is separate from any particular code, you can throw away the assistant’s version and write your own, or accept a faster version later, or fix a bug and try again, and the same contract keeps guarding you the whole time. The tests outlive any one attempt at the code, which is exactly what you want when the code is coming from a source that gets things wrong.
You have already been on the receiving end of this idea without writing the contracts yourself. When you ran the check widget on a project, the tests had been written for you, and every tick and cross you saw was a clause of a contract being enforced against your code. This week you begin writing those contracts yourself, and you meet tests in three ways that will recur for the rest of the course. You will keep using the ready made contracts through the widget when you work on the projects. You will start writing your own assert statements to pin down functions you or the assistant have written. And you will run those tests directly with the pytest command in the terminal and read its report, which lists exactly which contracts held and which were broken, so that you are no longer dependent on a friendly widget to tell you and can read the raw verdict for yourself.
Put the whole discipline together on a small problem before you rely on it. Here is a specification for gc_content written as a contract, before any implementation exists, with the awkward cases decided on purpose:
assert gc_content('GGCC') == 1.0
assert gc_content('ATAT') == 0.0
assert gc_content('GGAT') == 0.5
assert gc_content('ggcc') == 1.0 # lowercase counts too: a deliberate decisionOnly now do you ask the assistant for the function. If it hands you a version that walks through the strand counting bases equal to the capital letters G and C, that version will satisfy the first three clauses and fail the fourth, and the failure is not a nuisance; it is the contract doing its job, catching a decision the assistant made silently that does not match the decision you made deliberately. Without the fourth clause you would never have known the two of you disagreed.
Exercise 43-2
Take a vague request and turn it into a contract. Choose a small function you might ask an assistant to write, such as one that counts how many times a given codon appears in a longer strand, and before writing or requesting any code, write down three concrete examples with the answers worked out by hand, then turn each into an assert statement. Make sure at least one of your three examples is an awkward one, such as an overlapping occurrence or a codon that does not appear at all, and be ready to say what decision that awkward example forces. Keep this contract; the next three exercises all use it.
Exercise 43-3
Now with no assistant at all, so that the mechanics are firmly yours. Take a function you have already written earlier in the course and write two tests for it: one that you are confident it passes, and one that you expect it to fail, perhaps because it exposes a case the function never handled. Run both. The point is to feel the difference between a test that confirms and a test that accuses, because a contract that only ever confirms is not protecting you from anything.
Exercise 43-4
Run the full discipline in order, on the contract you wrote two exercises ago. You have the specification already, so now, and only now, ask the assistant to write the function. Give it the purpose and the signature. Do not give it your tests. Run your tests against what it produces. If they all pass, you have earned a limited and specific belief in the code, limited to the cases you actually specified. If one fails, notice that you found the disagreement before the code ever ran in anger, which is the entire reason for writing the contract first.
Exercise 43-5
Now aim the contract at a decision rather than at a bug. Write a contract for is_gc_rich(dna), a function that says whether a strand is GC rich, and make it decide two things the request itself leaves open: what happens at exactly one half, as in 'GGAT', and what happens for the empty strand. Decide both deliberately and write an assert for each. Then ask the assistant for the function using only the prose description, saying nothing about either case. Run your contract. Whatever it guessed, it guessed silently, and the only reason you can now tell whether the two of you agree is that you decided first. Record which of your two decisions it got wrong, if either.
Exercise 43-6
Last, watch the contract outlive the code. Take the same specification and ask the assistant for a second, deliberately different implementation of the same function, perhaps by asking for one that avoids a loop, or one that a beginner would find easier to read. Run your unchanged tests against it. Two different pieces of code, one contract, the same verdict. That is what it means to say the implementation is replaceable and the specification is not, and it is why the tests are worth more than any particular function you will ever get out of a machine.
For your logbook this week, record one specification you wrote as tests before requesting any code, note in particular which awkward case you had to make a deliberate decision about, and say whether the assistant’s eventual code agreed with your decision or quietly disagreed with it. If it agreed on everything, add the one further awkward case you wish you had thought to specify.