AI/ML

At a high level, there are two primary methods for referencing specific data:Insert data as context in the model prompt, and direct the response to utilize that informationFine-tune a model, by providing hundreds or thousands of prompt <> completion pairs

The design above goes by various names, most commonly "retrieval-augmented generation" or "RETRO"

Retrieval-augmented generation a) retrieves relevant data from outside of the language model (non-parametric) and b) augments the data with context in the prompt to the LLM. The architecture cleanly routes around most of the limitations of fine-tuning and context-only approaches.

In January 2023, online learning platform Study surveyed more than 1,000 American students and over 100 educators. More than 89% of the students said they had used ChatGPT for help with a homework assignment. Nearly half admitted to using ChatGPT for an at-home test or quiz, 53% had used it to write an essay, and 22% had used it for outlining one.

While 17 states in the U.S. have banned contract cheating, it has not been a problem for freelancers in Kenya

Christopher Kanan, an associate professor in the department of computer science at the University of Rochester, has started giving in-person, in-class quizzes due to the popularity of ChatGPT

Others, like Ethan Mollick, an associate professor at the University of Pennsylvania’s Wharton School, have chosen to take a more open approach to ChatGPT in class. “The truth is, I probably couldn’t have stopped them even if I didn’t require it,” he told NPR.

Confused deputy attacks Confused deputy is a term of art in information security. Wikipedia defines it like this: In information security, a confused deputy is a computer program that is tricked by another program (with fewer privileges or less rights) into misusing its authority on the system. It is a specific type of privilege escalation.

Language model applications work by mixing together trusted and untrusted data sources

For example, if the LLM generates instructions to send or delete an email the wrapping UI layer should trigger a prompt to the user asking for approval to carry out that action.

More to the point, it will inevitably suffer from dialog fatigue: users will learn to click “OK” to everything as fast as possible, so as a security measure it’s likely to catastrophically fail.

Data exfiltration attacks Wikipedia definition: Data exfiltration occurs when malware and/or a malicious actor carries out an unauthorized data transfer from a computer. It is also commonly called data extrusion or data exportation. Data exfiltration is also considered a form of data theft.

Even if an AI agent can’t make its own HTTP calls directly, there are still exfiltration vectors we need to lock down.

Locking down an LLM We’ve established that processing untrusted input using an LLM is fraught with danger. If an LLM is going to be exposed to untrusted content—content that could have been influenced by an outside attacker, via emails or web pages or any other form of untrusted input—it needs to follow these rules: No ability to execute additional actions that could be abused And if it might ever mix untrusted content with private data that could be the target of an exfiltration attack: Only call APIs that can be trusted not to leak data No generating outbound links, and no generating outbound images This is an extremely limiting set of rules when trying to build an AI assistant. It would appear to rule out most of the things we want to build!

For any output that could itself host a further injection attack, we need to take a different approach. Instead of forwarding the text as-is, we can instead work with unique tokens that represent that potentially tainted content.