AI/ML

Confused deputy attacks Confused deputy is a term of art in information security. Wikipedia defines it like this: In information security, a confused deputy is a computer program that is tricked by another program (with fewer privileges or less rights) into misusing its authority on the system. It is a specific type of privilege escalation.

Language model applications work by mixing together trusted and untrusted data sources

For example, if the LLM generates instructions to send or delete an email the wrapping UI layer should trigger a prompt to the user asking for approval to carry out that action.

More to the point, it will inevitably suffer from dialog fatigue: users will learn to click “OK” to everything as fast as possible, so as a security measure it’s likely to catastrophically fail.

Data exfiltration attacks Wikipedia definition: Data exfiltration occurs when malware and/or a malicious actor carries out an unauthorized data transfer from a computer. It is also commonly called data extrusion or data exportation. Data exfiltration is also considered a form of data theft.

Even if an AI agent can’t make its own HTTP calls directly, there are still exfiltration vectors we need to lock down.

Locking down an LLM We’ve established that processing untrusted input using an LLM is fraught with danger. If an LLM is going to be exposed to untrusted content—content that could have been influenced by an outside attacker, via emails or web pages or any other form of untrusted input—it needs to follow these rules: No ability to execute additional actions that could be abused And if it might ever mix untrusted content with private data that could be the target of an exfiltration attack: Only call APIs that can be trusted not to leak data No generating outbound links, and no generating outbound images This is an extremely limiting set of rules when trying to build an AI assistant. It would appear to rule out most of the things we want to build!

For any output that could itself host a further injection attack, we need to take a different approach. Instead of forwarding the text as-is, we can instead work with unique tokens that represent that potentially tainted content.