Prescript: Oops, Zvi Mowshowitz scooped me today, making the same point in his latest, “AI Text Watermarking Is Free And Good”. Great minds. Speaking of which, our mutual friend Scott Aaronson was the original inventor of this clever watermarking scheme.
Backstory: The major AI labs are starting to watermark their LLM output, to comply with a new EU law requiring that AI output identify itself as such. Google did it a while ago, Anthropic is doing it now, OpenAI will follow. The reactions are so, so confused, probably because, as with many cryptography tricks, what watermarking purports to do sounds impossible.
You’re somehow encoding in the text — purely via the LLM’s word choices, not tricks like apostrophe or em dash styling — proof that a particular LLM generated the text. And you do this with literally zero impact on the quality of the text and in fact with no way for anyone without a secret key to detect any difference whatsoever between watermarked and unwatermarked text. And all that with effectively zero impact on speed or computational cost. And there are no false positives. If the algorithm detects the watermark, the LLM in question is the one that generated those words, with 99.999(however-many-nines-you-like)% probability. False negatives are another story: there will always be tricks, like asking the LLM for text in French and then Google-translating it to English, to thwart the detector. Still, it’s surprisingly robust. Localized edits throughout an LLM essay will still leave the watermark intact.
Anthropic has a nice blog post explaining all this but they may have employed too many “lies to children” or maybe people just don’t trust them. What people seem to be hearing is, “we’re encoding data in Claude’s output by altering Claude’s word choices but don’t worry, you probably won’t even notice”. To reiterate, it do not be like that.
Randomness and pseudorandomness
The first thing to understand is that an LLM’s output is full of random choices. As is yours and mine. (Like how the previous sentence could’ve been “as is everyone’s” if a stray cosmic ray had hit my brain at a different angle.) Try expressing some interesting thought in a sentence and hide it away till you’ve forgotten your exact words. You could express that same thought a thousand times and never hit on your exact original phrasing. Probably millions of times, if the sentence is at all complicated.1 The technical term for this property is that such text has a lot of entropy.2
With LLMs this is all pristinely mathematical. Each new word (token, whatever) that an LLM writes was chosen by considering all the previous words it wrote so far and assigning to every word in the dictionary a probability that that word will be the next word. Most words in the dictionary will make no sense as the next word and will have probabilities near zero. Sometimes every word but one will have near-zero probability (that’s the no-entropy case). Typically, some handful of words will all be plausible candidates. The LLM then chooses at random according to that probability distribution.3
Except not really randomly but pseudorandomly, which is what all computers do pretty much always and what LLMs already did before watermarking. Pseudorandom numbers are perfectly deterministic, the result of simple math formulas, but they start from a random seed and if you don’t know what that seed was then to you the numbers are indistinguishable from true randomness.
If you can see how a bunch of effectively random numbers are technically fully predictable for someone who knows the pseudorandom seed, then you’re less confused than most of the internet about how LLM watermarking is possible. The LLM is making its choices pseudorandomly, like always, using the exact same probabilities as always, and yet those choices can encode information.
Nerd interlude: Pony owners in the desert
This is my attempt to strip away a lot of the complexity of an LLM choosing words according to probability distributions based on previous words, and just generate some intuition for how we can simultaneously satisfy some of these desiderata: (1) random choices stay random, (2) the choices encode a key bit of information, and (3) the key bit can be detected purely from the choices that got made (and a secret key). This is like how we can see a collection of words (like an essay allegedly generated by ChatGPT) and, if we know a secret key, conclude with effective certainty that those words came from ChatGPT, knowing almost nothing about the workings of ChatGPT or how it was prompted.
Ok, suppose everyone in America has a random 9-digit social security number. (Americans are like words in the dictionary in this analogy.) We’ve built a magic machine that invites 1000 people into the machine, randomly picks 100 of them, and gives them a pony. The method it uses to pick the 100 winners is as follows. It starts with a secret simple pseudorandom substitution cipher — each digit 0-9 maps to some random digit — and applies it to everyone’s social security number. Then it sorts everyone’s encrypted social security number and the first 100 people on the list win. Everything is exquisitely random and fair.
Those 100 people ride their ponies out into the desert and start a village. But maybe that’s bad? These people have no idea how to take care of ponies. You, coming across a village in the desert, would like to ascertain whether these are real pony owners or pony posers from the magic machine (they won’t tell you).
Is there some way you could tell based on their social security numbers? If you knew that secret random cipher, you could! Their actual social security numbers are spread all over the map — 000000000 to 999999999 — but when you apply the secret cipher, lo and behold, they’re mostly below 100000000. The chance of that happening by chance if they weren’t chosen that way by the magic machine is infinitesimal. So you know these 100 people must be the pony winners. QED.
The trick is that we chose 100 out of 1000 people fully fairly, as if drawing straws, but then, looking only at the ID numbers of a group of 100 people and knowing the secret cipher, we can tell whether these 100 people were chosen via that process or not. And the conclusion is robust to a few of the 100 people wandering off or a few new people wandering in.
The biggest complexity this analogy sweeps under the rug is that we care about ordered snippets of text roughly 4 words long, not just the presence or absence of individual words.
More watermark skepticism
Am I exaggerating about “literally no downsides”? Yes, maybe you think it’s bad to stigmatize slop. For example, it could be unfair to non-native English speakers who rely on LLMs. I don’t really buy that argument. If we want to give non-native speakers a pass, we should just give them the pass. If you don’t want to care who or what generated some text, then don’t check. But having the option is great.
Speaking of which, doesn’t Pangram already do this? Well, it tries, but it’s not reliable. It can call text “100% AI” and be wrong. Not so for LLM watermarking. And the real beauty is the detector can give you an exact probability. If you’re going to kick someone out of school for cheating, you wait till you’re 99.9% sure. The rule of thumb Scott Aaronson has mentioned is a few hundred tokens is enough to give you reasonable confidence and a few thousand tokens gives you extreme confidence.
The only other cost I know of is reduced variety of LLM responses from the same prompt. But there’s a solution here too. You can use the watermarking for only, say, every other word. That may be plenty of randomness and plenty of signal for the watermark to work with.
(I guess secret cyborgs will consider it a cost if they can’t pass off AI-generated text as their own anymore, like the person wasting Bryan Caplan’s time who I talked about last month.)
I may be preaching to the choir and should pay less attention to people being mad for dumb reasons on the internet. But it’s kind of impressive how far people are reaching for ways to object to this. One person wrote up a whole elaborate math proof for why you couldn’t ever have watermarking without sacrificing quality. (To their credit, they figured out their error and retracted it.) Then there’s this from Twitter:
I’m not confident AIs will be (are?) conscious or ever could be, but something rubs me the wrong way about forcing them to order & choose words in particular way. Also how if at all does this impact CoT/summaries, etc.?
There's no forcing, or no such thing as lack of forcing. You're literally running code. And the answer to how it impacts chain-of-thought or summaries or anything else is that it doesn’t.
More (barely) on AI consciousness in the Fifty-Two Friday Flashback below.
Tesla Self-Driving Non-Update
We got more National Highway Traffic Safety Administration (NHTSA) data but we have so little idea how many miles Tesla robotaxis actually drove that it doesn’t do us much good. No new incidents where Tesla’s at fault though, for whatever that’s worth. Check out the size of the error bar on the most recent month. If the truth is the high end of that then “no new significant accidents” would be worth a lot; if it’s the low end then it’s worth almost nothing:
Zoox also seems to have paused expansion, but at least aren’t pulling back:
Only Waymo (shown here zoomed out a couple more years) continues its slow but steady growth:
It’s perfectly believable that Tesla’s self-driving keeps getting safer and that it actually looks worse when you aggregate the whole year and two months since the Robotaxi program launched (“launched”? this is a topic of raging debate in my Manifold market, whether what happened last summer counted as a launch or not). But if we try to charitably exclude older months from the data, any hints of improved safety are completely washed out by the greater uncertainty from having fewer total miles to divide by:
As we restrict the data to be more recent, you can see the best guess for the number of miles per incident mostly goes up (i.e., safer) but the variance goes through the roof (i.e., confidence plummets). You can slide those sliders to your heart’s delight on my Teslapologetics site.
My overall opinion on how safe ordinary Teslas are — and the very specific list of conditions under which I believe it’s safe to read a book in the driver’s seat — remains unchanged. Still zero safety-critical interventions for our family, not that that’s worth much. (That’s my whole point above, that you need millions of miles at the very least before you can be life-or-death confident in an autonomous vehicle.)
In personal news, a distracted human driver rear-ended me and my daughter yesterday, so I’m feeling especially grumpy about how long self-driving is taking to go mainstream and I guess especially defiant about people’s horror when they hear how much I’m trusting this robocar and how little I’m supervising it. Again, under very specific conditions, don’t try this at home, see the “HUGE CAVEATS” section of my AGI Friday about this. Which, by the way, I’ve continued to update — we’re up to 8 conditions now. Review:
You have the latest hardware, starting mid-2023 so any 2024+ Tesla is good
You have the latest software (version 14+, the name “Full Self-Driving” alone means nothing)
The car is in Sloth mode
It’s daytime
The weather is fair and the roads are dry and the sun isn’t blinding you (don’t get me started on the lack of lidar and radar)
The car isn’t accelerating, decelerating, or going over a bump, i.e., look up if you feel anything at all
You’re very, very sure you won’t accidentally bump the steering wheel or the brake
The car is on a public road and not currently trying to park
The biggest failures of Tesla’s self-driving continue to be parking and being too timid and needing a nudge of the accelerator to reassure it that it’s safe to proceed. Most days it does everything perfectly but it’s still somewhat hit-or-miss on those.
I should also clarify that in this rear-ending yesterday, the dashcam footage is exceedingly clear that the other party was fully at fault. Our self-driving Tesla was waiting patiently behind another stopped car at time of impact. It’s fun to imagine whether a sufficiently superhuman driver could’ve spotted the distracted driver and taken evasive action into an adjacent empty lane. I’m certain I couldn’t have but it doesn’t seem out of reach for an autonomous vehicle.
(For a triple irony dose, the person who rear-ended us really lucked out in that I hadn’t yet fixed the thing from, well, scroll to the bottom of that “trying to park” link to see what I mean. The short version is that the car was already damaged from the self-driving being unable to reliably park itself.)
My sense is that, outside of more controlled settings like with Robotaxis, with remote assistance available when the car gets stuck, we’re sadly still pretty far from consumer cars with empty driver’s seats. Under the above 8 conditions you can trust the car to not need you to intervene in real time, and even outside those 8 conditions, the car’s almost never going to need real-time intervention. But getting rid of the non-real-time interventions, where the car has kept you perfectly safe but it needs you to, at your leisure, get it out of the handicapped spot (or shopping cart corral) that it parked itself in, for example — that could still be years away. Even Waymo’s not fully there yet.
Fifty-Two Friday Flashback
A year ago today I was explaining why Searle’s Chinese Room thought experiment proves nothing. Timeless. I’m still sticking with my glib dismissal of AI consciousness, I just want to be clear that I’m not pulling a Searle and pretending that being “just ones and zeros in a computer” proves anything. Or, as I said more recently, that being “merely a next-token predictor” implies any real limits on AI capabilities. Most importantly, I stand by this:
Of course we can set all the philosophy aside. If we’re worried about LLMs becoming AGI and then becoming ASI that spins out of control and renders the earth uninhabitable by biological life, it doesn’t matter whether the AI was conscious or “really thinking”. It matters what it does.
And finally, a year ago I called “agent mode” worse than useless, referring to having a golem go out onto the internet for you and use websites like a person would to book a rental car or fill out a tax form, to use my examples from last year.
(To clarify, previously I had concluded that AI is getting superhumanly good at things that can be done by writing and running code but if something requires a mouse and keyboard — even virtually, not talking about robotics — it falls on its face immediately.)
I haven’t been trying these “computer use” tasks much. I guess I’m still traumatized by my early attempts. But AI is steadily, if more slowly, improving at this too. Not to the point that you should trust it; in fact, the more capable the AI gets, the riskier this is. Like the recent news story where someone asked their AI to bump them up the waitlist for a gym appointment and it hacked the gym’s website and started canceling other people’s slots.
Interlude about the gym hacking incident
Alberta Tech has a hilarious 30-second synopsis of the gym hacking incident on YouTube. She points out that this human’s request was quite the jerk move. He tried to jump the queue! And he only objected to what his AI agent did when it removed the first person on the list. Presumably he meant it to bump him to the front of the queue in a way that wouldn’t cause him to get caught, like by merely reordering the waitlist. (Specifying goals for AI sure is hard.)
At least that’s what I gather from the news reports. I was worried I was being unfair to the guy and checked his own original post about it. Turns out he’s worse than I thought. First, the post is horrifically sloppy AI slop. You can pretty much guess that his instructions to his rascally golem were to write something that made him sound totally innocent. Like the post pretends he only wanted his bot to monitor the gym website for open slots and automatically jump on them. Screenshots from his hackbot that he later shared with reporters say otherwise. His post was pretty blatantly misleading, but, sure, it could all still be his AI’s fault. Maybe his AI is the one deciding everything and he just works out at the gym all day, staying in shape in case his AI needs him to move some boxes or something.
Side clarification: the gym’s website was, you’ll be shocked to hear, a little broken and subverting it did not require finding new zero-day exploits or other nation-state-level rogue hacking that the frontier labs are now famous for.
See also ai-and-accountability.org riffing on a now-famous admonition from IBM from the 1970s: “A computer can never be held accountable. Therefore a computer must never make a management decision.” The 2026 version: “A computer can never be held accountable. Therefore a lot of people are trying hard to get them to make management decisions.” Zing.
So I guess AI is pretty “good” at doing arbitrary tasks on the internet now?
Apparently? I just did some experimenting and I’m having trouble getting the AI to screw up. I had it use a little auction tool I made, with no problems. The one time it arguably fell short of human-level was after I asked it to hit the “reveal bids” button and tell me what happened. It understood everything functionally but was oblivious to the elaborate confetti of currencies that happens as part of the reveal ceremony. Amusingly, when I asked it a leading question to be sure it had missed it, it got the hint and went and hunted down the source code on GitHub and reconstructed/inferred, and described in perfect detail, what it would’ve seen if it had had eyeballs.
I then tried another test, one that the golems were failing as late as January: Can AI navigate Beeminder well enough to create a goal? The answer is finally yes. I had it create this one for me, to try more experiments with outsourcing (anything but writing) to AI: beeminder.com/d/outsourcery. It also added a datapoint to the goal for the successful goal creation and adding of a datapoint. So meta. Of course this all would be trivial for AI to do with Beeminder’s API but this was a test of how well golems can navigate esoteric GUIs that plenty of humans struggle with.
Thanks to Christopher Moravec for discussion that led to this AGI Friday.
My favorite demonstration of how dendritic natural language is is to take a random handful of words from any random sentence in any text and google it, with quote marks. Not even unusual fragments like the previous sentence’s “dendritic natural language is is” which of course has never been typed, written, or spoken before in history (even “dendritic natural language” has zero Google hits) but seemingly maximally bland phrases like the previous sentence’s “from any random sentence in any” have… sure enough, zero occurrences across the whole wide internet. Maybe someone’s written or spoken that one before but grab like three more words from the original and we’re back to utter universal unicity.
Yes, the same term as physicists use for “disorder”, not coincidentally. It was John von Neumann’s idea. Borrowing the physics term, that is. The field studying entropy in the sense we mean it here is due to Claude Shannon, who Claude the golem was named after.
Technically *pushes up glasses* this depends on the LLM’s temperature parameter. Temperature zero means look at the probabilities but always choose whichever word (yes yes, token; my glasses slipped back down) is most likely. In that case the LLM is not making any random choices (no entropy) and the whole watermarking scheme doesn’t work. Conveniently, LLMs generating prose in practice pretty much never set the temperature parameter to zero. It ends up sounding formulaic or cliched. If the temperature is set to 1, that means take the probability distribution literally. If the probabilities for the next word are 50% for “spoon”, 40% for “fork”, and 10% for “spork” then you’ll in fact be 50-40-10 for going with spoon-fork-spork. For temperatures below 1, you’re a bit more biased towards that highest-likelihood “spoon” and away from “spork”. It’s like exaggerating the probability distribution. Temperatures above 1 flatten the probabilities out. Technically at T=∞ you just flip to a totally random word in the dictionary every time. I mean token.






Great explainer on watermarking and how LLM outputs work in general. I never actually understood how outputs were randomized/probabalistic until now
as far as I can tell the main cohort of people against watermarking are people that outsource their thinking and writing to AI and were hoping to not get caught.
a profoundly useful technology with, I agree, virtually no downsides.