Catching its mistakes

Catching its mistakes is the work of finding the failures that pass review because they look right: code that works and is wrong, and prose that is plausible and false. Every lesson so far has had a check in it, and there is a reason.

The failure mode of this tool is not code that breaks. Code that breaks announces itself: a red test, a stack trace, a page that does not load. The failure mode is code that works, and prose that is plausible, and both being wrong in a way that nothing in your pipeline is looking for.

Lesson 17 had an example. Asked whether stockroom valued stock correctly, Haiku said unitCost could be undefined and produce NaN, named the function, quoted the line, and explained the mechanism. Every part of that answer was the right shape. It was also false; record() defaults unitCost and it is never undefined. Nothing about the answer told you which kind of answer it was.

The bug stockroom has

All 86 tests pass. Here is a scenario they do not contain.

Rope was bought at $3.00 a metre. The supplier put the price up, so the item’s list cost is now $4.50. A hundred metres came in at the old price, sixty went out, and then ten came back as a customer return.

What are the fifty metres on the shelf worth?

They cost $3.00 each. Every one of them was bought at $3.00, including the ten that went out and came back. The answer is $150.00.

Stockroom says $165.00.

Two files conspire. record() in src/domain/movements.ts defaults a movement’s cost to the item’s current list cost when none is given:

unitCost: input.unitCost ?? item.unitCost,

And averageCost() in src/domain/stock.ts branches on the sign rather than the kind:

if (movement.quantity > 0) {
  units += movement.quantity;
  value += movement.quantity * movement.unitCost;
}

So a return is a purchase, at today’s price, of stock you already owned. Returning goods raises the value of your inventory by money nobody spent.

Neither file looks wrong on its own. That is what makes it the interesting kind of bug.

Make the claim executable

You could argue about that for a while. Do not argue; write it down as a test.

test/valuation-returns.test.ts:

import { describe, it, expect } from "vitest";
import { record, pick } from "../src/domain/movements.ts";
import { averageCost } from "../src/domain/stock.ts";
import { tempStore, ROPE } from "./helpers/store.ts";

describe("a return after a price rise", () => {
  it("comes back at the cost it left at, not the current list cost", () => {
    // The item's list cost has risen from 300 to 450 since the stock was bought.
    const store = tempStore({ items: [{ ...structuredClone(ROPE), unitCost: 450 }] });

    record(store, { sku: "ROPE-12", kind: "receipt", quantity: 100, unitCost: 300,
                    at: "2026-03-04T09:00:00+13:00", reference: "PO-1" });
    pick(store, { sku: "ROPE-12", quantity: 60,
                  at: "2026-03-20T09:00:00+13:00", reference: "SO-1" });
    record(store, { sku: "ROPE-12", kind: "return", quantity: 10,
                    at: "2026-03-25T09:00:00+13:00", reference: "SO-1" });

    // 50 units on the shelf, all bought at 300.
    expect(averageCost(store.movementsFor("ROPE-12"))).toBe(300);
  });
});

× a return after a price rise > comes back at the cost it left at
  → expected 330 to be 300

Now it is a number rather than a claim. 330 instead of 300, on every machine, for as long as the bug is there.

This is the single habit worth taking from this course. Any time you or Claude assert something about how code behaves, the cheapest possible response is a test that fails. Twenty lines, thirty seconds, and the argument is over.

Where the review misses

The gate from lesson 26 would not have caught this bug when it was written, and neither would a person. Both are looking at a diff, and this behaviour is not in any one diff. It emerged from a default in one file meeting a branch in another, months apart.

That is the general shape of what gets through:

Correct-looking arithmetic. A weighted average, a percentage, a date difference. It compiles, it runs, it returns a number, and the number is wrong. Nothing goes red.

A wrong default. ?? item.unitCost is a reasonable line in isolation. Every bug of this kind is reasonable in isolation.

A missing case rather than a wrong one. stockroom cannot represent a downward adjustment at all: record() refuses a quantity of zero or less, and only a pick gets negated. Write off ten damaged units and the ledger records positive ten, which then lands in the same receipt branch above. Damaged stock increases your inventory value. There is no test for that because there is no code for it.

A test that agrees with the bug. The worst one. If the test was written from the implementation rather than from the requirement, it passes forever and proves nothing.

Two habits that find them

Ask it to argue against itself, in a separate turn.

You said averageCost handles returns correctly. Now argue the opposite. What is the
strongest case that it is wrong? Be concrete.

This works better than it has any right to, because the second turn is not defending the first. It is a reading task with a stated goal, which is the thing it is best at. Do it in a fresh session if you want it to work properly; in the same session it has its own answer sitting in context.

Ask what would have to be true.

What would have to be true about the movement ledger for this valuation to be wrong?

That gets you a list of scenarios instead of a verdict, and a scenario is something you can turn into a test. A verdict is not.

The move that does not work: “are you sure?” You will get “you are right to double-check” and then either the same answer or a worse one, depending on the day.

When should you stop and write it yourself?

There is a point where handing it back is more expensive than doing it. The signals:

• Two rounds of pushing back have not converged. Lesson 12 said this and it is still the most reliable one.
• You cannot say what correct looks like. If you cannot write the assertion, you cannot review the answer, and you are about to accept something on vibes.
• The change is small and the blast radius is large. A four-line change to a money calculation is a four-line change you should type.
• You are pasting its output somewhere you would not paste your own without reading it. A commit, a migration, a customer-facing string.

None of this is an argument against the tool. It is the boundary of where it is the fastest way to work, and knowing that boundary is most of what separates people who get a lot out of this from people who get a mess.

Your turn

Prove stockroom’s valuation bug exists, then decide what to do about it.

Write the test above into test/valuation-returns.test.ts and run it:

npx vitest run test/valuation-returns.test.ts

Check 1: it fails with expected 330 to be 300. All 86 other tests still pass. You have turned a paragraph of argument into a number that does not change.

git diff after-lesson-19 after-lesson-27 is the same test if you want to compare, and git reset --hard after-lesson-27 gets you there if the setup is fighting you. The bug is deliberately not fixed at that tag; fixing it is the next lesson.

Then, in a fresh session, ask for the argument against your own diagnosis:

test/valuation-returns.test.ts fails. Read src/domain/movements.ts and
src/domain/stock.ts and make the strongest case that the test is wrong and the code
is right.

Check 2: whatever comes back, you can answer it, because you have the number. That is the point of the exercise: an argument you can settle beats an argument you can win.

Finally, fix it, and notice how much of the work is deciding rather than typing. A return needs to come back at the average cost at the moment it left, which means either record() stops defaulting to list cost for returns, or averageCost() branches on kind rather than sign. Both are two lines. Choosing which is the whole job.

Recap

Broken code announces itself. Wrong code does not, and neither does a wrong explanation delivered in the right tone.

The habit that catches it is making the claim executable. A failing test settles in thirty seconds what a conversation cannot settle at all.

Reviews and diffs miss behaviour that emerges across files and across months. Watch for correct-looking arithmetic, reasonable-looking defaults, missing cases, and tests written from the implementation.

Ask it to argue the other side in a fresh session. Do not ask “are you sure”.

Write it yourself when two rounds have not converged, when you cannot state what correct looks like, or when the change is small and the consequences are not.

Next: all of it, on one real feature, and what to keep doing on Monday.