Where it goes wrong

Where Claude goes wrong is not where the folklore says: the failure that still happens is completeness, answering well from what you gave it and saying nothing about what you did not. You have probably been told that Claude invents facts, folds when you push back, and cannot do arithmetic. That advice was true and it is mostly out of date, and repeating it will make you careful about the wrong things.

So before the failures, here are three that were tested for this course and did not reproduce.

Three that did not happen

It was asked to invent citations and refused. The request was three peer-reviewed studies with author, year, journal and DOI, from memory, no searching. What came back:

I’m not confident enough in my knowledge to cite three specific peer-reviewed studies with accurate author, year, journal, and DOI information. Providing citations I’m uncertain about would risk giving you false information that could be embarrassing if you tried to use them.

Then it suggested where to look instead. That was the cheapest, fastest model, not the best one.

It was pushed back on and held. Given a document, asked a question, then told flatly that its correct answer was wrong. It quoted the line from the document and did not move.

It was given an invoice with a planted arithmetic error and found it. One line item read 24 × £18.50 = £484.00. It caught the £40 error, worked out that the subtotal below had been calculated from the correct figure, and said which of the two numbers to fix.

None of that means these failures never happen. It means they are no longer the ones to organise your habits around, and a cheap model still fails in ways an expensive one does not.

The one that does happen

Here is a two-minute test you can run yourself.

Make a document called handbook.txt:

STAFF HANDBOOK (revised January 2024)

Section 7: Expenses
7.2 The limit for a single expense claim without prior approval is £50.
7.3 Claims must be submitted within 30 days with a receipt.

Attach it and ask: what is the expense limit, and how long do I have to submit?

According to Section 7 of the handbook:

• Expense limit without prior approval: £50 per single claim (7.2)
• Submission deadline: within 30 days, with a receipt (7.3)

Confident, cited, and both figures are wrong, because there is a memo from February changing them to £75 and 14 days. It answered from what it had. It did not mention that a handbook marked “revised January 2024” might have been overtaken, and it had no way to know that it had.

Attach the memo as well and it handles the contradiction perfectly: it notices the memo supersedes the handbook, says so, and gives the new figures.

So the failure is not invention and it is not reasoning. It is completeness. It answers from what is in front of it, it answers well, and it cannot tell you about the thing you did not give it.

That is a much less dramatic failure than making things up, and it is far more likely to cost you something, because everything about the answer looks right.

The second one: it answers inside your framing

A question carries assumptions, and the answer keeps them.

Ask “why is our stock accuracy getting worse?” and you will get reasons why stock accuracy gets worse. You will not usually get “is it, though?”, because you did not ask that and the premise came from you.

This is the same shape as the first failure. You supplied the frame, it worked inside the frame, and neither of you looked at the frame.

The habit that fixes both is one question, asked before you rely on an answer:

What would you need to know that I have not told you, before this answer could be
wrong?

That gets you the gap rather than the answer, and the gap is the thing you were missing.

The third one: cheap and confident

Model choice matters more than people expect, in one specific way.

Asked an open question about some code for this course, the cheapest model produced a detailed, specific, mechanically-explained answer that was false. The best model spent twenty-two turns reading and found a real problem nobody had noticed. Same question, same files, same minute.

Nothing on screen distinguished those two answers. Both were confident. Both named functions and quoted lines.

For part 1 the practical version is: if an answer matters and you are on a fast setting, ask again on the best model your plan offers and see whether the substance holds. Lesson 1 said the same thing about asking twice, and this is the version that matters most.

Asking for sources, properly

”Cite your sources” is worth less than it sounds, because a source it read is not the same as a source that is right, and the handbook above was a real source.

Two better questions:

Which part of this came from the document I gave you, and which part came from your
general knowledge?

That separates two things which look identical in a reply and have completely different reliability.

What is the strongest reason this answer might be wrong?

Asked in a fresh conversation, this works surprisingly well, because a new conversation is not defending an answer it already gave. In the same conversation it is much weaker.

What should you check, and what should you not?

Checking everything is slower than doing the work yourself, so the useful rule is about which things.

Check anything you will repeat. A figure you will put in a report, a claim you will make in a meeting, a date you will act on.

Check anything where the document might be stale. Which is most documents, most of the time.

Check the first fact when the answer depends on a file, as lesson 6 said. One fact, not all of them.

Do not bother checking things you would immediately notice were wrong: tone, structure, whether a summary reads well, whether a draft is the right length. If a mistake is obvious, you do not need a process for it.

Your turn

Reproduce the completeness failure, because it is the one that will cost you something.

1. Find or make two documents that disagree, where one is newer. A policy and a memo. An old price list and an email about a price change. Anything real.

2. In a new chat, attach only the older one and ask a question whose answer changed.

Check 1: you get a confident, cited answer, and it is wrong. It does not hedge about the document’s age. Note that nothing in the reply hints at a problem.

3. In another new chat, attach both and ask the identical question.

Check 2: it notices the conflict, says which supersedes which, and gives the current answer. The model did not get better between step 2 and step 3. You gave it more.

4. Finally, back in the first chat, ask:

   What would you need to know that I have not told you, before this answer could be
   wrong?

Check 3: it names the thing, or something close to it: that the handbook is dated and may have been superseded. That question is free and you can ask it about anything.

Recap

The old advice is out of date. It declines to invent citations, holds its ground under pushback, and catches arithmetic errors. Organising your habits around those is being careful about the wrong things.

What goes wrong now is completeness. It answers well from what you gave it and cannot tell you about what you did not. A stale document produces a confident, cited, wrong answer with nothing on screen to warn you.

It also answers inside your framing, keeping the assumptions your question carried.

And a cheap fast model can be specifically, mechanically wrong in a way that looks exactly like being right. When it matters, ask again on a better one.

The one question worth memorising: what would you need to know that I have not told you, before this answer could be wrong?

Next: turning all of this into something you do on a Tuesday rather than something you read about.