This is getting tiring. Watermarking has no effect on model output quality when implemented correctly. It's somewhat like swapping a random RNG seed to the seed 42, and detecting what the seed was from a random sequence. The sequence generated from the seed 42 is just as random as any other seed. There couldn't be a quality difference. And yes, the output from an LLM is a conditional random sequence of tokens from a distribution determined by a model.
“When implemented correctly” is probably what people are complaining about.
Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.
In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Model companies are doing this for themselves anyways, it’s so they don’t feed generated content back into the slopper and collapse the model. From that angle it over time contributes to better model quality.
Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).
So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).
This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
Benchmarked output quality versus actual output quality are very different things. Some usecases are at the very fringe of model intelligence and depth of intelligence and logic suffers.
That's not true. Watermarks are messing with the next token generation probabilities based on some random seed. The quality is neccesarily lower, the difference is simply too small to notice, typically.
This article reads like it was written at least partly by AI to me. Specifically it reads like an article written by AI with edits made by a human further prompting the AI.
> Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.
Relevance and irrelevance are not introduced above this comment. This reads like an LLM-ism (particularly a GPT-ism) editing a document, removing something, and leaving a note about why it was removed, which doesn't really make sense when reading it.
> Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.
Also a GPT-ism which appears when it draws a counter-conclusion in the text because it feels the need to be honest and a human tells it to remove it because it's not true because of "reason".
Overall interesting research, however, I think it's great that model output is getting watermarked. I was skeptical of this at first, but Opus 5.5 is so good, it seems like it's a non-issue in practice.
The reason I think watermarking is great is because it's a really good way of preventing training on it's own output indiscriminately and Ouroboros-ing itself.
I am unable to understand what happens if the watermarked output goes as input to another agent. Let's say we asked Claude a question and got a watermarked response. If we pick that response and append it to the question we're asking ChatGPT, then will it answer or refuse to do so?
If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.
In _1984_ the Big Brother regime has the idea that by controlling language you can influence what is possible to think, and thus becomes a key tool of political repression.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
> Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. It’s nothing more than an attempt to broaden what’s considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
It’s been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. There’s no struggle possible because there is no intent. It’s also about memories and emotions, so if it WAS an intentional act, it’s not an attempt to control language. It wouldn’t be about trying to not think of the word cat, but trying to not think about that time when your cat died.
It’s hard to tell because the writing quality is garbage.
Second paragraph:
> Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.
This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.
> At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.
Claim support, if it appears, is pages later.
> At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.
?
> Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.
Wtf. Non-sequitor. Where does this come from?
> Such a watermarking procedure can therefore affect both what the model says and what an agent does.
Duh? In the literal sense of outputting different tokens.
> We call this behavioral effect sampling drift.
I think they should have used an LLM for writing help, or paid more for the one they used.
The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that it’s either entirely AI generated, or at least 100% AI assisted.
I’m fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AI’s real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.
I think they goofed on the conclusion: “These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.”
The article’s entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.
Also, it’s not AI’s fault that URLs in the references section aren’t clickable. Come on.
Watermarking sounds like a good idea, but it's not. Token drift from watermarking will degrade the quality of outputs and could allow clever people to circumvent guardrails.
You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.
Changing a random seed could either improve or degrade the output. In theory, better and worse outputs should be equally probable, depending on your luck.
Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).
So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).
This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
> Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.
Relevance and irrelevance are not introduced above this comment. This reads like an LLM-ism (particularly a GPT-ism) editing a document, removing something, and leaving a note about why it was removed, which doesn't really make sense when reading it.
> Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.
Also a GPT-ism which appears when it draws a counter-conclusion in the text because it feels the need to be honest and a human tells it to remove it because it's not true because of "reason".
Overall interesting research, however, I think it's great that model output is getting watermarked. I was skeptical of this at first, but Opus 5.5 is so good, it seems like it's a non-issue in practice.
The reason I think watermarking is great is because it's a really good way of preventing training on it's own output indiscriminately and Ouroboros-ing itself.
If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. It’s nothing more than an attempt to broaden what’s considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
It’s been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. There’s no struggle possible because there is no intent. It’s also about memories and emotions, so if it WAS an intentional act, it’s not an attempt to control language. It wouldn’t be about trying to not think of the word cat, but trying to not think about that time when your cat died.
Second paragraph:
> Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.
This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.
> At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.
Claim support, if it appears, is pages later.
> At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.
?
> Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.
Wtf. Non-sequitor. Where does this come from?
> Such a watermarking procedure can therefore affect both what the model says and what an agent does.
Duh? In the literal sense of outputting different tokens.
> We call this behavioral effect sampling drift.
I think they should have used an LLM for writing help, or paid more for the one they used.
The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that it’s either entirely AI generated, or at least 100% AI assisted.
I’m fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AI’s real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.
I think they goofed on the conclusion: “These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.”
The article’s entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.
Also, it’s not AI’s fault that URLs in the references section aren’t clickable. Come on.
You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.