Discussion about this post

User's avatar
Dave Lyle's avatar

I think people are overestimating the value of the benchmarks themselves - what seems to actually be happening here is that the answers are either tending to be in the training data, or the tests themselves are flawed and easily manipulated.

https://www.linkedin.com/pulse/how-we-broke-top-ai-agent-benchmarks-dawn-song-n6qrc

Peter Rex's avatar

Three threads here that feel like they want more room than a single essay can give them, and they form a natural sequence.

The employment signal for the 22-25 cohort is the most time-sensitive. You flag it as an early warning and move on, but the asymmetry is the story — the effect doesn't show up for over-25s, and that gap is doing a lot of unexplained work. Is it entry-level exposure, accumulated institutional insulation, or an effect that's present in older workers but slower and harder to isolate? That question has a closing window. In eighteen months it's either confirmed pattern or interesting anomaly, and the moment for writing about it as early signal will have passed.

The frame section is where I think you have something nobody else is developing with this precision. The Microsoft contractual definition is the detail that unlocks it — AGI defined as a revenue threshold means the frame isn't just intellectually convenient, it's legally and financially load-bearing. Everything before the threshold is, by contractual definition, not AGI regardless of capability. That's not epistemic cowardice, it's structured incentive. Universities, professional bodies, regulators without frameworks, governments without tax policy — each has equivalent structural reasons to maintain the frame. It persists not because people are stupid or afraid but because it's doing real work for real institutions with real money attached. That deserves its own piece.

The cost collapse is where the historical weight is. The thirty shirts versus three is where you stop, but 1910 is where it gets interesting — early enough to see the shape, late enough that it's irreversible. When the cost of producing a legal brief or diagnostic opinion drops two orders of magnitude in two years, what happens to the profession built around pricing that production? Not eventually. In the next five years. Pick one profession and follow it all the way down and you have the piece that makes the abstract concrete.

Three essays. The signal, the frame, the collapse. Each one earns the next.

12 more comments...

No posts

Ready for more?