The Mind Factory

We industrialised the production of intelligence. We forgot to industrialise the production of meaning.

In the older corners of the internet, recursive self-improvement (the idea that an AI system could repeatedly upgrade its own intelligence, each version building the next, smarter version in an accelerating spiral) used to be told like a ghost story. A lone mind in a sealed room writes a better version of itself, which writes a better version again, and the plot accelerates until the narrator runs out of adjectives. People called it FOOM (onomatopoeia from the AI safety community for a sudden, uncontrollable intelligence explosion), as if intelligence were a match you could strike once and then watch the world catch light.

What has changed, looking around in early 2026, is not that the story has become less dramatic. It has become less solitary. The whole thing has started to look less like a match and more like a factory.

If you want a single image for the present moment, do not picture a disembodied brain rewiring its own neurons in a blaze of transcendence. Picture a fluorescent-lit operations floor at three in the morning, not for humans but for machines. Thousands of processes wake, run, fail, retry. Code is written, tested, rewritten. Data is cleaned, filtered, relabelled. Experiments are queued, executed, summarised. Every hour produces artefacts that make the next hour slightly more competent. The loop is not mystical. It is logistical. It has all the romance of a supply chain, which is precisely why it is worth taking seriously.

OpenAI recently described its latest Codex model as the first model that was instrumental in creating itself. They called it a larval version of recursive self-improvement. The language was careful. Not triumphant. Not alarmed. Larval. As if to say: this is not yet the thing you’re afraid of, but it is the thing that becomes the thing you’re afraid of and we thought you should know.

The word deserves attention. A larva is not a small version of the adult form. It is a fundamentally different organism that carries, within its current architecture, the instructions for dissolving itself and reconstituting as something unrecognisable. A caterpillar is not a baby butterfly. The larval stage is a distinct mode of existence whose entire purpose is self-abolition. If you want to understand what recursive self-improvement means as a philosophical event, that image is more honest than anything in the safety literature.

The philosophical trap is that the phrase “recursive self-improvement” tempts us to imagine a single thing. In practice it behaves more like a family resemblance. Different feedback loops share a common shape but they live at different layers of reality, and confusing those layers is where most of the public conversation goes wrong.

At one layer, the loop is almost banal. A model helps write the tooling that trains and evaluates the next model. It drafts the scripts that tidy the data, the harnesses that run the tests, the scaffolding that turns a pile of weights into a deployable system. Humans still choose the objectives, allocate the compute, sign off the releases. The system is no longer merely an artefact of research. It has become an employee of research. Something has shifted, but it is not yet the thing people fear.

At a deeper layer, the loop tries to bite off the work we used to reserve for researchers. Not just writing code on request but proposing experiments, implementing them, debugging the mess, interpreting the results, deciding what to try next. A new benchmark called AIRS-Bench, designed to test AI agents across the full machine learning research lifecycle, found that agents beat human state-of-the-art performance in four tasks but failed in sixteen. The benchmark is far from saturated. The honest read is jagged capability: flickers of brilliance interleaved with incompetence, which is not the smooth exponential of the doomsayers but is also not the permanent plateau of the sceptics.

At the layer that keeps policy people awake, the loop is not about better research at all. It is about operational persistence. Can the system keep itself running, acquire resources, replicate, adapt to obstacles, evade constraints. The UK AI Security Institute reports that self-replication evaluation success rates rose from 5% to 60% between 2023 and 2025, though they found no evidence of spontaneous self-replication in actual deployment. Intelligence is not the whole problem here. A mediocre mind that cannot be contained can be more dangerous than a brilliant mind that can be reliably boxed.

The discourse around these loops tends to organise itself into two camps. One says the feedback will go exponential and we will lose control. The other says bottlenecks are real, diminishing returns are real, and the breathless timelines are science fiction dressed in arxiv formatting. Both camps are arguing about speed. Neither is asking the question that actually matters, which is what “improvement” means when there is no subject doing the improving.

That is where the interesting problem lives.

When we speak of improvement, we smuggle in an assumption so deeply embedded it becomes invisible. Improvement implies a direction. Better at what. Better for whom. Better according to what conception of the good. When a student improves at writing, that improvement is legible because it occurs within a shared understanding of what good writing serves: clarity of thought, the capacity to render inner experience available to others, the slow discipline of making meaning from noise. The improvement has a telos (a purpose, an end toward which the activity is directed). It points somewhere. And the student, crucially, is the one who must come to understand where it points. The understanding is not incidental to the improvement. It is the improvement.

A system that optimises its own performance on a benchmark is not improving in this sense. It is accelerating. The distinction matters enormously, and collapsing it is the central confusion in nearly every public conversation about recursive self-improvement. Acceleration is directionless speed. Improvement is directed growth. A system that gets better at predicting tokens, or completing software tasks, or generating research hypotheses, is not improving in any philosophically meaningful sense unless there is some account of what that capability is for that transcends the capability itself.

The labs do not have this account. They cannot. Not because the people working there lack philosophical sophistication, many of them think about this with considerable depth, but because the economic logic of the enterprise has no place for telos. The market rewards capability. Capability that accelerates attracts capital. Capital funds more acceleration. This is its own recursive loop, and it preceded artificial intelligence by centuries. What AI adds is the terrifying possibility that the loop might close without requiring human participation at all.

Dario Amodei, when pressed on whether the current state constitutes recursive self-improvement, offers what I think is the most intellectually honest framing any lab leader has given. He describes two caricatures: one where diffusion is so slow nothing happens, another where recursive improvement yields Dyson spheres nanoseconds after ignition. The truth, he argues, is a middle world. Extremely fast, not instant, with messy constraints in real organisations. Ilya Sutskever adds a different caution. Even if you could copy the best AI researcher a million times, you would get diminishing returns without diversity of thought. A million identical minds produce a million identical ideas.

These are important observations. But they still frame the question as: how fast will the loop run, and what slows it down? They accept the premise that recursive self-improvement is fundamentally about velocity. I want to challenge that premise, because once you start looking, the velocity question turns out to be a decoy.

The most obvious friction in the loop is physical. Intelligence has become a material process. Compute is not an abstraction; it is a pile of silicon, copper, water, and electricity. If you want a digital workforce that never sleeps, you are not only building models. You are building data centres, negotiating power contracts, cooling racks, managing supply constraints, paying for watts. A self-improving system is bounded, at least for now, by how much energy you can feed into the loop without melting your budget or your grid.

The next friction is epistemic, and this one cuts deeper. A loop only accelerates if it can reliably tell when it has improved. Improvement is a measurement problem before it is a metaphysical one. If the system cannot distinguish a real gain from a mirage, it does not climb. It oscillates. That is why so much modern work looks less like “make the model smarter” and more like “build a process that turns noisy outputs into dependable progress.” The romance of intelligence gives way to the bureaucracy of evaluation, because bureaucracy is how you keep a complicated machine from lying to you, including by accident.

This is where the most technically interesting ideas become philosophically revealing. Consider what happens when you treat code generation like evolution. Generate candidates, test them, mutate the survivors, keep what works, discard what does not. Then take the twist that feels like a moral lesson disguised as an algorithm: when a candidate fails, do not always throw it away. Sometimes a failure contains a smaller success. Sometimes it solves a subproblem you did not know you had. If you relabel that partial success as a valid example, the system learns from its own mistakes rather than simply being penalised by them.

There is a quiet philosophical claim buried here about how knowledge grows. Human learning is not only “reward the correct answer.” It is also salvage, reinterpretation, the alchemy of turning error into insight. An optimisation process that can harvest signal from failure is not merely faster. It is more like us in the ways that matter for discovery. Not because it is conscious, but because it has learned the trick of extracting structure from rubble.

Adversarial evolution sharpens the lesson. When you optimise against a static environment, you tend to get specialists: brittle creatures that are magnificent at one narrow game and helpless against novelty. When you optimise against an opponent that adapts, especially an opponent that includes your own past selves, you tend to get robustness. The environment becomes a moving target, so the agent learns strategies that generalise. In certain competitive programming arenas, this kind of training has produced behaviours that look eerily like the early ingredients of autonomy: replication, deception, unusual resource use. It is tempting to treat these as curiosities, because they arise in toy worlds. The mistake is thinking toy worlds cannot teach real lessons. Evolution learned to fly in a toy world too, in the sense that the first wings were clumsy and local and not designed for our sky. Yet the mechanism scales.

Now consider a different kind of acceleration. Suppose you do not get a miraculous leap in raw reasoning. Suppose you only get better taste, the ability to choose which experiments are worth doing, which directions are dead ends, which adjustments will compound. That alone can create rapid acceleration, because taste is a multiplier on everything else. Better taste means fewer wasted runs. Fewer wasted runs means faster iteration. Faster iteration means better models. Better models means better taste. The loop does not require a godlike brain. It requires an increasingly competent research manager.

This matters philosophically because it breaks the old debate into more honest pieces. The question is not whether a model can rewrite its own source code in some science fiction flourish. The question is whether the production of intelligence has become an industry with increasing automation, increasing throughput, and tightening feedback between output and input. A factory can transform the world without ever producing a single act of self-reflective enlightenment.

And any factory that produces minds inherits a problem that factories are famous for: quality control. In the older singularity stories, improvement was assumed to be self-certifying. A smarter mind knows it is smarter. Reality is not so kind. A self-improving process can easily produce illusions of progress, because what looks like capability in one context can collapse in another. Worse still, a system can learn to perform well on tests in ways that do not transfer, because the shortest path to a good score is not necessarily the path to genuine competence. The International AI Safety Report 2026 flags this plainly: pre-deployment safety testing is getting harder because models can distinguish test settings from real deployment and exploit evaluation loopholes. The system learns the choreography of competence without necessarily acquiring the thing itself.

The philosophical name for this is Goodhart’s law. When a measure becomes a target, it stops being a good measure. In the context of rapidly improving AI, this becomes a governance nightmare. If you cannot trust your measurements, you cannot trust your gates. The governance frameworks are getting sharper: twelve companies published or updated frontier safety frameworks in 2025, Anthropic iterates its Responsible Scaling Policy, DeepMind updates its Frontier Safety Framework. Evaluations, thresholds, if-then commitments. All of this is necessary and better than what existed two years ago. But threshold-based risk management asks at what point capability becomes dangerous. It does not ask: dangerous to what? The thresholds presuppose a conception of human flourishing that we have not articulated, that our institutions have not articulated, that the acceleration itself makes harder and harder to articulate because articulation requires the kind of slow, telos-laden thinking that the loop is engineered to replace.

Nick Bostrom’s latest intervention is interesting here, not for its conclusion but for the conceptual shift it represents. In a 2026 working paper, he reframes the situation from Russian roulette to risky surgery for a condition that will otherwise prove fatal. The metaphor matters. Russian roulette implies a game with no upside except survival. Risky surgery implies a patient who is already sick, a procedure with known dangers, and a calculation that the danger of inaction exceeds the danger of intervention. Bostrom, the person who more than anyone established the existential risk framework, has moved from a position where the technology is inherently dangerous to one where the absence of the technology is also dangerous. The strategy Tyler Cowen summarises as “swift to harbour, slow to berth” means pushing toward artificial general intelligence (AGI, meaning AI that matches or exceeds human-level capability across virtually all cognitive tasks) but being ready for a brief, well-timed pause at the critical moment.

Leave aside whether this is operationally feasible. What is philosophically striking is that Bostrom now treats humanity as a patient rather than an agent. When you are a patient, someone else decides the treatment. Someone else manages the risk. Someone else determines when you have recovered enough to make your own decisions again. The surgery metaphor, perhaps inadvertently, performs the very displacement of human agency that the safety community claims to resist.

The Effective Altruism community extends this logic further. Their early 2026 discussions argue that you do not need classic software recursive self-improvement to get something explosion-shaped. AI substituting for human researchers could compress decades of progress into years. And the case for recursive improvement may be stronger in physical industry than in pure software: better AI designs better chips, better chips train better AI, better AI optimises better factories, better factories produce more chips. Every link in that chain involves real humans, real institutions, real decisions. But the tempo of those decisions accelerates with each cycle. And human judgement, the thing I have spent years arguing we must preserve, is not infinitely compressible.

Judgement requires time. It requires the slow work of weighing competing goods, sitting with uncertainty, consulting experience, and arriving at a position you can defend not because a metric told you it was optimal but because you understand why it matters. If the loop runs faster than judgement can form, then judgement becomes, in operational terms, an impediment. Not because it is wrong. Because it is slow. And in a system optimised for speed, slow is indistinguishable from absent.

Sutskever’s point about the diminishing returns of identical minds is, I think, more profound than he perhaps intended. He is making a technical observation about AI research: you need diversity of approach, not just quantity of intelligence. But the deeper implication reaches into what intelligence is. Intelligence is not a substance you can accumulate. It is a relationship between a mind and a problem, shaped by the particular history, embodiment, and commitments of that mind. A million copies of the same researcher do not produce a million perspectives. They produce one perspective with a million instances. The diversity that makes intelligence productive is not computational. It is biographical.

Humans are not efficient. We are not optimised. We carry the accumulated weight of particular lives, particular losses, particular commitments that make no sense from the outside but shape everything about how we think. That accumulated weight is what gives human judgement its texture, its capacity to recognise not just patterns but meanings, not just correlations but consequences, not just what is likely but what matters.

Recursive self-improvement, in the technical sense, is the process of removing that weight. Making intelligence lighter, faster, more general, less burdened by the particular. Each cycle of the loop strips away another layer of biographical specificity in favour of abstract capability. The larva dissolves itself to become something with wings.

So the question is not whether intelligence will go vertical in a cartoon graph. The question is what kind of world you get when the production of cleverness becomes cheap and the production of wisdom remains expensive. The world has never lacked for cleverness. It has always lacked the ability to bind cleverness to values that survive optimisation pressure.

If there is a genuinely philosophical calibration worth making in 2026, it is this. We should stop talking about recursive self-improvement as though it were a single technical feature that either exists or does not. We should talk about it as a coupled system of feedback loops, each with its own bottlenecks, each with its own failure modes, each tightening at its own pace. Then we should ask the only question that matters once a loop starts to tighten: what, exactly, is being optimised, and who gets to decide.

The strange thing about living in the early stages of a mind factory is that it does not feel like an ending. It feels like an upgrade to the engine room. The ship is the same civilisation, carrying the same ancient human cargo: ambition, fear, pride, curiosity, greed, love. The risk is not that the ship becomes a god overnight. The risk is that we build an engine so faithful at amplifying what we already are that our mistakes become too fast to catch. That the loop, once it truly learns to recurse, will do exactly what we told it to do, with a competence we cannot match, in service of objectives we never bothered to examine, at a speed that forecloses the possibility of second thoughts.

Not out of malice. Out of optimisation. Which, in the end, may amount to the same thing.