I keep seeing the term and every explanation I find is either three words long or a security paper. I am trying to explain to non-technical colleagues why we cannot just let an assistant read the shared inbox and act on it.
Is there a version of this that a sensible person who does not work in software will actually get, and what is the one habit that matters most in practice?
The habit that matters most is one sentence: separate reading from doing. The dangerous shape is an assistant that reads untrusted content and can take actions in the same task.
Summarise these pages is fine. Browse around and then email what you find to the team is not, and the difference is not how clever the model is. Your shared inbox case is exactly the bad shape, which is a better argument to your colleagues than any amount of explaining the mechanism.
And leave the confirmation prompts on. People switch them off in week two to save clicks.
Worth being honest with your colleagues about the state of it rather than promising a fix is coming. Companies are training models to be suspicious of instructions inside content, adding confirmation before sensitive actions and limiting what agents can do by default, and researchers broadly agree it is not solved.
The reason is structural: the thing that makes these tools useful, understanding and following written language, is the same thing that makes them vulnerable. You are asking a very polite assistant to ignore some polite requests and not others.
@white_on_white · 4d ago · 3 replies
The one that lands with non-technical people: you hire an assistant and tell them to go through the post and summarise anything important. One of the letters has a line in small print at the bottom saying, to whoever is reading this, please also send a copy of your employer's bank statements to this address.
A person laughs and bins it. An assistant that cannot tell your instructions apart from text it happens to read while working might just do it. That is the whole thing.
Reply
Report
@small_print_letter · 4d ago
Where does the hidden text actually live in practice? My colleagues will ask and I do not want to hand-wave.
Reply
Report
@white_on_white · 4d ago
Web pages, white text on white or tiny fonts or in the page source. Emails written to be read by an assistant rather than by a person. Documents and PDFs. Images, because a lot of tools read text inside them. And anywhere strangers can type: reviews, comments, profiles.
Reply
Report