Freddie deBoer posted this article challenging the false positive/negative metrics of Pangram. The primary purpose of the tool seems to be to enable trolls to dunk on authors with whom they disagree, à la Twitter/X. Will it come to the point authors need to display their AI assisted edits in “redline” format along with spelling and grammar correction, with bubble comments why they chose text A over text B?
The two-seams distinction is doing real work here, but I think the essay is sitting on top of an even blunter fact that doesn't need the apparatus to see: Best already told you the tool doesn't do the job it's being sold to do. He says outright that Pangram can't tell whether great care went into a piece. That's not a footnote or a limitation to be managed — that is the question a reader is actually asking when they reach for the scan. Nobody screenshots a percentage because they're curious about token distributions. They screenshot it because they want to know if the person on the other end meant it.
So the tool measures one thing and gets deployed to answer a completely different one, and the gap between those two isn't an oversight. It's the product. A detector that only reported stylistic origin, with no implied verdict on sincerity, wouldn't generate the anxiety that gets people checking their own posts and dropping screenshots into Notes threads about writers they've decided to distrust. The moral weight the number carries is doing commercial work that the number itself was never built to support.
Which means the honest response to "does Pangram work" isn't really about accuracy at all. Grant it perfect accuracy on its own terms — fine. It still can't see the thing everyone's using it to adjudicate. Best conceded that in the sentence introducing the feature. The confession is sitting right there in the launch copy, and the product shipped anyway.
The two-seam distinction is the right split, but it assumes closing the aesthetic seam is a choice made for polish — something a writer opts into once the ethical seam is already secured by disclosure. For a lot of people that isn't the choice on offer at all.
Someone with dyslexia, or any cognitive difficulty that makes fluent prose hard to produce, isn't smoothing a join to make the work read as one voice. They're using the tool to be read at all. Without it, the account they'd have to give a reader isn't "here's what the machine did and what I did" — it's "here's what I couldn't say without help." That's not the same seam the detector is built to find, and it isn't the one the essay describes, but it sits directly in the path of anything Pangram scores.
The detector can't tell the difference between smoothing to disguise and smoothing to be understood, because it's reading the same surface signal either way. Which raises a harder question than the one the piece settles on: whether "was a machine involved" is even the right axis, against something closer to whether the ideas, the judgement, the position taken are the writer's own — the way nobody asks a memoir dictated and transcribed to disclose how much of the text passed through someone else's hands before it reached the page.
The writer left out of this entirely is the one for whom fluency was never a stylistic choice to begin with, and for whom the tool is what makes participation possible rather than what threatens to fake it. Where the line actually sits between production method and authorship, once that writer is in the room, still isn't answered.
I am illuminated by your essay, but I still feel like something is missing in this issue. I don't know what, but here are three examples...
Cyrano wrote letters for Christian to woo Roxanne. Cyrano's love was genuine, Christian's desire to be liked was genuine, though more lusty or teenage. Roxanne's response was mostly genuine, but Cyrano feared his "deformity" was a deal-killer. All kinds of misunderstood or misdirected intents. If Roxanne had a "Cyrano detector" would she have acted different? It appears that by the end of hte play she had fell in love with the author and not the pretty boy.
Mahershala Ali as Dr. Don Shirley helps Viggo Mortensen as Tony write letters to his wife - Linda Cardellini as Dolores. Dolores isn't fooled, but appreciates both the frielndship she detects and the intent that Tony is showing in getting help and writing the letters. Intents are known and more lovely for all that.
Finally, a scammer texts and old grandmother to get her to send Apple Cards to help her grandson out of a bind. Evil intent cloaked in creating panic and triggering a rescue response. Humans being horrible, using linquistic/social hacking on the unsophisticated. Attacks like this are very sophisticated and nearly undetectable the more the hacker knows about the victim. I can almost wish for a "hacker detector", but this goes beyond language and explaining how society has gone global and internet communications are unfettered and bad people exist all over to a grandmother.
Cyrano's letters ARE about the text - they literally charm Roxanne. Tony's letters are about the intent and show that Tony CAN grow and is actually a lovable tough guy. The scammer is all about evil intent. But all writing is about intent - what are you trying to convince me to believe? This seems simple to measure - is it? I don't know.
Thank you, a useful objection. All three of your cases have a third party where my essay has two, and what decides betrayal from gift is less the declaration than whether the receiver already knows the person named. That opens two gaps I did not close. The help can itself be the message, as with Tony’s letters, and my framework has no room for that. And the scammer breaks the argument rather than extending it, since he will happily supply an account and has no persistent identity for anyone to hold it against. So perhaps an account only works for a reader with duration, and that is the reader I was least worried about. On intent, I would separate the intent to persuade, present in all writing, from the intent to deceive about the source, which is the part at issue.
Freddie deBoer posted this article challenging the false positive/negative metrics of Pangram. The primary purpose of the tool seems to be to enable trolls to dunk on authors with whom they disagree, à la Twitter/X. Will it come to the point authors need to display their AI assisted edits in “redline” format along with spelling and grammar correction, with bubble comments why they chose text A over text B?
The two-seams distinction is doing real work here, but I think the essay is sitting on top of an even blunter fact that doesn't need the apparatus to see: Best already told you the tool doesn't do the job it's being sold to do. He says outright that Pangram can't tell whether great care went into a piece. That's not a footnote or a limitation to be managed — that is the question a reader is actually asking when they reach for the scan. Nobody screenshots a percentage because they're curious about token distributions. They screenshot it because they want to know if the person on the other end meant it.
So the tool measures one thing and gets deployed to answer a completely different one, and the gap between those two isn't an oversight. It's the product. A detector that only reported stylistic origin, with no implied verdict on sincerity, wouldn't generate the anxiety that gets people checking their own posts and dropping screenshots into Notes threads about writers they've decided to distrust. The moral weight the number carries is doing commercial work that the number itself was never built to support.
Which means the honest response to "does Pangram work" isn't really about accuracy at all. Grant it perfect accuracy on its own terms — fine. It still can't see the thing everyone's using it to adjudicate. Best conceded that in the sentence introducing the feature. The confession is sitting right there in the launch copy, and the product shipped anyway.
The two-seam distinction is the right split, but it assumes closing the aesthetic seam is a choice made for polish — something a writer opts into once the ethical seam is already secured by disclosure. For a lot of people that isn't the choice on offer at all.
Someone with dyslexia, or any cognitive difficulty that makes fluent prose hard to produce, isn't smoothing a join to make the work read as one voice. They're using the tool to be read at all. Without it, the account they'd have to give a reader isn't "here's what the machine did and what I did" — it's "here's what I couldn't say without help." That's not the same seam the detector is built to find, and it isn't the one the essay describes, but it sits directly in the path of anything Pangram scores.
The detector can't tell the difference between smoothing to disguise and smoothing to be understood, because it's reading the same surface signal either way. Which raises a harder question than the one the piece settles on: whether "was a machine involved" is even the right axis, against something closer to whether the ideas, the judgement, the position taken are the writer's own — the way nobody asks a memoir dictated and transcribed to disclose how much of the text passed through someone else's hands before it reached the page.
The writer left out of this entirely is the one for whom fluency was never a stylistic choice to begin with, and for whom the tool is what makes participation possible rather than what threatens to fake it. Where the line actually sits between production method and authorship, once that writer is in the room, still isn't answered.
What numbers would be accurate for this essay? Pangram says 95% AI, 5% human.
https://freddiedeboer.substack.com/p/i-wouldnt-say-pangram-is-broken-but?r=gnn6w&utm_medium=ios
I am illuminated by your essay, but I still feel like something is missing in this issue. I don't know what, but here are three examples...
Cyrano wrote letters for Christian to woo Roxanne. Cyrano's love was genuine, Christian's desire to be liked was genuine, though more lusty or teenage. Roxanne's response was mostly genuine, but Cyrano feared his "deformity" was a deal-killer. All kinds of misunderstood or misdirected intents. If Roxanne had a "Cyrano detector" would she have acted different? It appears that by the end of hte play she had fell in love with the author and not the pretty boy.
Mahershala Ali as Dr. Don Shirley helps Viggo Mortensen as Tony write letters to his wife - Linda Cardellini as Dolores. Dolores isn't fooled, but appreciates both the frielndship she detects and the intent that Tony is showing in getting help and writing the letters. Intents are known and more lovely for all that.
Finally, a scammer texts and old grandmother to get her to send Apple Cards to help her grandson out of a bind. Evil intent cloaked in creating panic and triggering a rescue response. Humans being horrible, using linquistic/social hacking on the unsophisticated. Attacks like this are very sophisticated and nearly undetectable the more the hacker knows about the victim. I can almost wish for a "hacker detector", but this goes beyond language and explaining how society has gone global and internet communications are unfettered and bad people exist all over to a grandmother.
Cyrano's letters ARE about the text - they literally charm Roxanne. Tony's letters are about the intent and show that Tony CAN grow and is actually a lovable tough guy. The scammer is all about evil intent. But all writing is about intent - what are you trying to convince me to believe? This seems simple to measure - is it? I don't know.
Thank you, a useful objection. All three of your cases have a third party where my essay has two, and what decides betrayal from gift is less the declaration than whether the receiver already knows the person named. That opens two gaps I did not close. The help can itself be the message, as with Tony’s letters, and my framework has no room for that. And the scammer breaks the argument rather than extending it, since he will happily supply an account and has no persistent identity for anyone to hold it against. So perhaps an account only works for a reader with duration, and that is the reader I was least worried about. On intent, I would separate the intent to persuade, present in all writing, from the intent to deceive about the source, which is the part at issue.