The Person Who Could Have Thought

What AI does to formation, in school, in universities and in ordinary life

You can feel that you understand something. The feeling is specific and often strong. A concept clicks, an explanation lands, a difficult system resolves into sense, and somewhere underneath there is a quiet signal that says yes, I have this now, I could explain it if you asked.

The signal is not reliable. Cognitive scientists have a name for one version of the problem, the illusion of explanatory depth. People believe they understand how things work, zips, governments, the water cycle, in far more detail than they actually do, and the belief survives because it is so rarely tested. We almost never have to produce a full account of anything, so we almost never get the feedback that would show us how little is there. Ask someone to explain, step by step, with nothing left out, and the confidence falls quickly towards the truth.

There is a part of the mind built to produce that confidence. Call it the interpreter. Its work is to narrate and explain and justify, to give a fluent account of what we know and why we did what we did, and it is good at the work, good enough that the account often runs ahead of the understanding it claims to describe. The interpreter is the press office, not the factory. It can report that production is going well without a single thing having been made.

For most of history the press office had a limit. To produce a convincing account of understanding you usually had to have done at least some of the understanding first. Fluency and formation travelled together. Not always, not honestly, but closely enough that the account was some evidence of the thing.

That is the link the machine severs. AI produces the fluent account on demand, instantly, about anything, to a high finish. It will write the paragraph, work the problem, summarise the book, supply the reading. And it will hand you, folded into the output, the feeling of having understood it. The question of whether you understand was always answered by the interpreter. Now the interpreter has help, and the help never tires. The test grades itself. The part that does the grading can pass without doing the work, and you need not notice.

So the inner audit cannot be trusted.

If the test cannot be run from the inside, then the consequences are not private. They are educational, institutional and ordinary.

The question is not, in the first instance, whether the tool was used. That is already too crude. A calculator can be harmless in one room and destructive in another. A dictionary can be a support for thought or a substitute for it. The question is what kind of human capacity the activity is supposed to form, and whether the tool use forms that capacity or bypasses it.

That is the line.

Not tool or no tool. Formation or no formation.

In school, this should make us much more conservative than the technology industry would like. Children are not workers with outputs to optimise. They are not junior consultants, junior copywriters, junior analysts or junior software engineers. They are people in formation. The essay is not only a container for an argument. It is one of the means by which the capacity to argue is built. The maths problem is not only a request for an answer. It is a machine for producing mathematical judgement. Translation, summary, close reading, memorisation, drafting, revising, diagramming, explaining aloud: these are not obsolete rituals around the real work. They are the real work.

So in schools the burden of proof should fall heavily on the AI use. Unless it can be shown that the use builds the relevant capacity, it should not be there. An AI that quizzes a pupil after she has read the chapter may build formation. An AI that gives a second explanation after the pupil has first wrestled with the problem may build formation. An AI that asks for the next step, refuses to finish the answer and makes the pupil try again may build formation. But an AI that writes the paragraph, solves the problem, produces the summary or supplies the interpretation before the pupil has attempted the act herself is not assisting formation. It is consuming the occasion for it.

That distinction matters more than detection. Schools are being pushed toward the least interesting question: did the child use AI? The better question is: did this activity still require the child to become more capable? A perfectly undetectable submission can be educationally empty. A fully acknowledged use of AI can be educationally serious. The moral centre is not the presence of the machine. It is the presence or absence of formation.

Higher education has a different problem, because universities are full of people who are close enough to competence to mistake fluency for it. The undergraduate, the master’s student, the doctoral candidate, even the academic, can produce a beautiful account of a process they did not really undergo. The university therefore cannot treat AI literacy as a module in prompt-writing. That is far too small. Students need to understand the interpreter, the illusion of explanatory depth, the ease with which a coherent account can counterfeit understanding. They need to know that the danger is not only that they might deceive the institution. It is that they might deceive themselves and receive a good mark for doing so.

A serious university response would not be built around surveillance. It would be built around occasions on which understanding has to appear in real time. More oral examination. More live problem-solving. More drafting in front of another person. More vivas, supervisions, seminars, studios, labs and defences where the prepared account can be interrupted. More assessment that asks not only for the finished object but for the student’s ability to move, unaided, inside the material.

This is not nostalgia. It is a recognition of what assessment was always partly for. The point was never simply to certify that a product existed. It was to discover whether a person had been changed by making it.

And then there is the rest of life, where the danger is quieter because nobody is grading us. The adult version of the problem is not that we will fail an exam. It is that we will slowly stop doing the things that keep our judgement alive, while surrounded by prose that assures us our judgement has never been sharper. We will read summaries and think we have read books. We will accept explanations and think we have understood systems. We will ask for arguments and think we have considered the matter. We will generate plans and think we have decided. We will keep the vocabulary of agency while outsourcing more and more of the acts from which agency is made.

Formation is not a possession. It is a practice. You do not own it the way you own a bicycle in a shed. You own it the way you own fitness, or a language, or a friendship: by renewing it. Stop doing the acts and the capacity changes. Not dramatically, not all at once, not in a way that announces itself. It simply becomes easier to accept the substitute. Then easier still. Eventually the substitute feels natural, and the unease of doing the thing yourself feels like inefficiency.

That may be the most dangerous inversion of all. The feeling of difficulty, which once marked the place where formation was happening, comes to feel like a design flaw. The tool offers to remove it. And because it removes the discomfort, we experience the removal as progress.

But some discomfort is not friction in the system. It is the system.

The blank page is not merely an obstacle to writing. It is part of how writing teaches you what you think. The unsolved problem is not merely a delay before the answer. It is the condition under which problem-solving is formed. The hard paragraph, the failed proof, the book that will not summarise itself in your head, the conversation in which you cannot yet say what you mean: these are not inefficiencies waiting to be automated away. They are the places where the self is still under construction.

So the practical rule is plain, though not comfortable. Keep some capacities under load. Do not let every difficult act become a prompt. Do some reading with no summary waiting. Do some writing with no completion engine beside you. Do some thinking without asking for an outline first. Do some explaining to another human who can stop you and say, no, do that step again. Do some work in conditions where the only evidence of capacity is capacity.

Not because tools are wicked. Because formation requires occasions on which the tool is absent, late, limited or subordinate.

The right question, then, is not “Can I give a convincing account of my understanding?” The machine can help me do that. My interpreter can help me do that. My education can help me do that. The right question is harsher and more external: what can I still do when the account is interrupted?

Can I solve the next problem?

Can I explain the live case?

Can I write the sentence that has not already been prepared?

Can I notice the flaw before someone points it out?

Can I continue when the tool is gone?

Those are not perfect tests either. Nothing human is. But they have one advantage over the private audit: they happen in the world. They do not ask the press office for its report. They ask the factory to produce.

That is where schools should begin. That is what universities should teach. That is what adults now have to protect in themselves. The point is not to preserve some romantic purity of unaided thought. Unaided thought has never been pure. We have always thought with books, teachers, libraries, instruments, conversations, diagrams, marginalia and other minds. The point is to preserve the apprenticeship by which a person becomes capable of joining that extended machinery as an agent rather than an accessory.

That is why the test that grades itself cannot be the end of the matter. The inner feeling will not save us. The fluent account will not save us. The vocabulary of sovereignty will not save us. Only formation can save the thing that talks about formation. And formation has to be built, protected, renewed and sometimes imposed from outside, because the part of us that most wants reassurance is the part least able to tell whether reassurance is deserved.

So the rule is not abstinence. It is order.

First the capacity, then the tool.

First the struggle, then the support.

First the formation, then the augmentation.

Reverse the order and we do not get the same human being with better equipment. We get a human being who can describe a capacity he has not formed, defend a judgement he has not earned and feel awake all the way down.

The danger is not that AI does our thinking. The danger is that it prevents the formation of the person who could have thought, while giving that person a flawless account of the thinking they did not do.