AI/ML

As announced earlier this week, Senator Edward Markey (D-MA) and Representatives Ted Lieu (D-CA), Don Beyer (D-VA), and Ken Buck (R-CO) have introduced the Block Nuclear Launch by Autonomous AI Act, which would “prohibit the use of Federal funds to launch a nuclear weapon using an autonomous weapons system that is not subject to meaningful human control.” The act would codify existing Pentagon rules for nuclear weapons, which, as of 2022, read thusly:

Its sponsors note that a 2021 National Security Commission on Artificial Intelligence report recommended affirming a ban on autonomous nuclear weapons launches, not only to prevent it from happening inside the US government but to spur similar commitments from China and Russia. Publicizing the bill calls attention to the potential dangers of current-generation autonomous artificial intelligence systems, a going concern in Congress and the tech world alike. And as indicated by the press release, it offers a chance to highlight the sponsors’ other nuclear non-proliferation efforts — like a recent bill restricting the president’s power to unilaterally declare nuclear war.

Under US law, the issue is quite clear: you need an inventor and an inventor needs to be human, to get a patent. There are many good reasons for this: mainly because the entire point of the patent system is to create incentives to invent. An AI system… doesn’t need that incentive. It just responds to inputs.

I’m sorry, but what?!? Larry Lessig arguing that not giving out patents to AI “jeopardizes billions in current and future investments?” This is the same Lessig, after all, who created Creative Commons and spent years and multiple books explaining how locking up knowledge via intellectual monopolies was harmful

First of all, most inventions wouldn’t be protected by trade secrets anyway, because most things can be reverse engineered. Second, the value in inventing something is in bringing that product to market. That’s where you make the money, in selling the product, not the patent. Third, the point of the patent is to help the inventor recoup the capital expenditure in creating the invention in the first place. But, AI is cheap and can generate tons of ideas quickly. The capex is minimal.

If the makers of AI are concerned that they can’t make money without patents, just ask their AI to invent products they can sell in the market. They don’t need patents. They need products.

At a high level, there are two primary methods for referencing specific data:Insert data as context in the model prompt, and direct the response to utilize that informationFine-tune a model, by providing hundreds or thousands of prompt <> completion pairs

The design above goes by various names, most commonly "retrieval-augmented generation" or "RETRO"

Retrieval-augmented generation a) retrieves relevant data from outside of the language model (non-parametric) and b) augments the data with context in the prompt to the LLM. The architecture cleanly routes around most of the limitations of fine-tuning and context-only approaches.

In January 2023, online learning platform Study surveyed more than 1,000 American students and over 100 educators. More than 89% of the students said they had used ChatGPT for help with a homework assignment. Nearly half admitted to using ChatGPT for an at-home test or quiz, 53% had used it to write an essay, and 22% had used it for outlining one.

While 17 states in the U.S. have banned contract cheating, it has not been a problem for freelancers in Kenya

Christopher Kanan, an associate professor in the department of computer science at the University of Rochester, has started giving in-person, in-class quizzes due to the popularity of ChatGPT

Others, like Ethan Mollick, an associate professor at the University of Pennsylvania’s Wharton School, have chosen to take a more open approach to ChatGPT in class. “The truth is, I probably couldn’t have stopped them even if I didn’t require it,” he told NPR.

Confused deputy attacks Confused deputy is a term of art in information security. Wikipedia defines it like this: In information security, a confused deputy is a computer program that is tricked by another program (with fewer privileges or less rights) into misusing its authority on the system. It is a specific type of privilege escalation.

Language model applications work by mixing together trusted and untrusted data sources

For example, if the LLM generates instructions to send or delete an email the wrapping UI layer should trigger a prompt to the user asking for approval to carry out that action.

More to the point, it will inevitably suffer from dialog fatigue: users will learn to click “OK” to everything as fast as possible, so as a security measure it’s likely to catastrophically fail.

Data exfiltration attacks Wikipedia definition: Data exfiltration occurs when malware and/or a malicious actor carries out an unauthorized data transfer from a computer. It is also commonly called data extrusion or data exportation. Data exfiltration is also considered a form of data theft.

Even if an AI agent can’t make its own HTTP calls directly, there are still exfiltration vectors we need to lock down.

Locking down an LLM We’ve established that processing untrusted input using an LLM is fraught with danger. If an LLM is going to be exposed to untrusted content—content that could have been influenced by an outside attacker, via emails or web pages or any other form of untrusted input—it needs to follow these rules: No ability to execute additional actions that could be abused And if it might ever mix untrusted content with private data that could be the target of an exfiltration attack: Only call APIs that can be trusted not to leak data No generating outbound links, and no generating outbound images This is an extremely limiting set of rules when trying to build an AI assistant. It would appear to rule out most of the things we want to build!

For any output that could itself host a further injection attack, we need to take a different approach. Instead of forwarding the text as-is, we can instead work with unique tokens that represent that potentially tainted content.