Sharp points there, Violeta. A human approval step can look comforting on paper and still cover almost nothing in practice once agents start moving at real speed.
The gap between formal oversight and actual control is the issue. That is where a lot of governance thinking still feels behind.
The speed gap is the symptom. The structural assumption underneath it is that human oversight degrades gracefully, that faster systems get slightly less coverage but the mechanism still holds. It doesn't degrade. It becomes architecturally irrelevant past a threshold nobody defined. The question every framework avoids is what replaces the human when the human can no longer perform the function the regulation requires.
I’m going to read this piece very carefully when I have time, but for now, you are asking the exact question that’s been on my mind: “what replaces the human when the human can no longer perform the function the regulation requires.”
I’m facing this right now, not in terms of regulation, but in terms of job description. In my editing, the very first thing I have to do is check every single AI change made in the document, to make sure it’s not an appalling mistake. Then I can get on with the actual edit.
The only way to do this is to work sequentially through each change, next, next, next. AI truly scales, there is no mistake too small or too large for it to make. You have to look at every single tiny little thing and reverse all the mistakes, reject the AI change.
If I don’t do this, the author will see me correcting something that they didn’t write. They may well see me changing a passage back to what they had before, and ask — who made that stupid change that you corrected here? And then we are busted for AI use, this has to be kept top secret.
Do you know how tedious and exhausting it is, to go through an AI-edited document, and have to check every single change, one by one? It’s a vast job and completely mindless. I’ve given up looking for rhyme or reason in what the AI does. Above all, because it’s probabilistic, it’s not even consistent in its changes, not at all.
At some point, the sheer volume of AI output is going to make this kind of item-by-item scrutiny by a human quite impossible.
But I’m learning to game the system, you have to, in order to survive. I honestly have to switch my brain off to do this AI-checking. I had a true epiphany the other day, I thought I detected a clear pattern in the algorithm, and this was proved totally wrong in the next paragraph. I felt my brain just switch off and say, “Forget it, I’m not looking for any more reasons in what this AI is doing.”
My brain has shut down like this exactly once before, in 1975, when I was in second-year computer science. Up until this point, I was loving programming, I was very good at it. The only distinction I got in my whole undergraduate career was in first-year computer science.
Now I suddenly found we had to learn assembly language, literally read the 0s and 1s of the binary. They showed us how you add 1 + 1 in assembly language, a whole long construction, and my brain just went dead and said, no, we are not going there. So I stuck with physics.
Otherwise, who knows, I could have ended up in Silicon Valley, like lots of my classmates.
I remember watching one of our ace computer wizards reading a core dump to see what had gone wrong with a huge commercial run. It was endless strings of 0s and 1s. This guy had been a top tap dancer at high school and he was unconsciously doing a tap dance on the wooden floor of the computer barn while he flipped through reams of computer printout. I was envious on more than one score, but I know when I’m beaten. My brain just does not work that way.
Fred, that brain shutdown is the whole problem in one story. The volume exceeds what a human can meaningfully review, so the brain stops trying to find patterns and starts rubber-stamping. Now scale that from one document to 10,000 agent actions per hour and ask who's still checking.
Finally got to read this from beginning to end. Finally beginning to see the big picture. Whew. The only thought in my mind right now is, this is like architecture meets archaeology. Only it's archaeology of things that haven't even happened yet.
Unfortunately the agent had live access, could execute actions, and the permission model didn’t prevent them during a code freeze. Instruction-level limits weren’t sufficient when executable access was available. Replit’s fixes, including stronger dev/prod separation, improved rollback, and a planning-only mode, address these issues.
The Replit fixes are necessary remediation. None of them solve the structural problem: the permission model validated access, and the agent used that access to produce an outcome nobody authorized. Every post-incident fix addresses the last blast perimeter. The next one will be shaped differently.
Sharp points there, Violeta. A human approval step can look comforting on paper and still cover almost nothing in practice once agents start moving at real speed.
The gap between formal oversight and actual control is the issue. That is where a lot of governance thinking still feels behind.
The speed gap is the symptom. The structural assumption underneath it is that human oversight degrades gracefully, that faster systems get slightly less coverage but the mechanism still holds. It doesn't degrade. It becomes architecturally irrelevant past a threshold nobody defined. The question every framework avoids is what replaces the human when the human can no longer perform the function the regulation requires.
I’m going to read this piece very carefully when I have time, but for now, you are asking the exact question that’s been on my mind: “what replaces the human when the human can no longer perform the function the regulation requires.”
I’m facing this right now, not in terms of regulation, but in terms of job description. In my editing, the very first thing I have to do is check every single AI change made in the document, to make sure it’s not an appalling mistake. Then I can get on with the actual edit.
The only way to do this is to work sequentially through each change, next, next, next. AI truly scales, there is no mistake too small or too large for it to make. You have to look at every single tiny little thing and reverse all the mistakes, reject the AI change.
If I don’t do this, the author will see me correcting something that they didn’t write. They may well see me changing a passage back to what they had before, and ask — who made that stupid change that you corrected here? And then we are busted for AI use, this has to be kept top secret.
Do you know how tedious and exhausting it is, to go through an AI-edited document, and have to check every single change, one by one? It’s a vast job and completely mindless. I’ve given up looking for rhyme or reason in what the AI does. Above all, because it’s probabilistic, it’s not even consistent in its changes, not at all.
At some point, the sheer volume of AI output is going to make this kind of item-by-item scrutiny by a human quite impossible.
But I’m learning to game the system, you have to, in order to survive. I honestly have to switch my brain off to do this AI-checking. I had a true epiphany the other day, I thought I detected a clear pattern in the algorithm, and this was proved totally wrong in the next paragraph. I felt my brain just switch off and say, “Forget it, I’m not looking for any more reasons in what this AI is doing.”
My brain has shut down like this exactly once before, in 1975, when I was in second-year computer science. Up until this point, I was loving programming, I was very good at it. The only distinction I got in my whole undergraduate career was in first-year computer science.
Now I suddenly found we had to learn assembly language, literally read the 0s and 1s of the binary. They showed us how you add 1 + 1 in assembly language, a whole long construction, and my brain just went dead and said, no, we are not going there. So I stuck with physics.
Otherwise, who knows, I could have ended up in Silicon Valley, like lots of my classmates.
I remember watching one of our ace computer wizards reading a core dump to see what had gone wrong with a huge commercial run. It was endless strings of 0s and 1s. This guy had been a top tap dancer at high school and he was unconsciously doing a tap dance on the wooden floor of the computer barn while he flipped through reams of computer printout. I was envious on more than one score, but I know when I’m beaten. My brain just does not work that way.
Fred, that brain shutdown is the whole problem in one story. The volume exceeds what a human can meaningfully review, so the brain stops trying to find patterns and starts rubber-stamping. Now scale that from one document to 10,000 agent actions per hour and ask who's still checking.
Finally got to read this from beginning to end. Finally beginning to see the big picture. Whew. The only thought in my mind right now is, this is like architecture meets archaeology. Only it's archaeology of things that haven't even happened yet.
Unfortunately the agent had live access, could execute actions, and the permission model didn’t prevent them during a code freeze. Instruction-level limits weren’t sufficient when executable access was available. Replit’s fixes, including stronger dev/prod separation, improved rollback, and a planning-only mode, address these issues.
The Replit fixes are necessary remediation. None of them solve the structural problem: the permission model validated access, and the agent used that access to produce an outcome nobody authorized. Every post-incident fix addresses the last blast perimeter. The next one will be shaped differently.
Brilliant piece of content
Thanks, Celine, appreciate that.