Hey ChatGPT! How to build a botnet with compromised ChatGPT instances! AI botnet vulnerability

Anthropic created an AI jailbreaking algorithm that keeps tweaking prompts until it gets a harmful response.

Learn about visual prompt injections, their appearance, and top defense strategies against these attacks.

Emails, documents, and other untrusted content can plant malicious memories.

Automatically unfurling hyperlinks can lead to data exfiltration. This post shows how to mitigate this threat in Slack Apps

Andrej Karpathy explains something that's been confusing me for the best part of a year: The decision by LLM tokenizers to parse special tokens in the input string (``, …

Researchers at MIT have released the AI Risk Repository, a comprehensive database that can help organizations identify and mitigate AI risks.

New research analyzes the misuse of multimodal generative AI today, in order to help build safer and more responsible technologies

There are some fascinating new details in this lengthy report outlining the safety work carried out prior to the release of GPT-4o. A few highlights that stood out to me. …

By far the most detailed paper on prompt injection I've seen yet from OpenAI, published a few days ago and with six credited authors: Eric Wallace, Kai Xiao, Reimar Leike, …

Simply look out for libraries imagined by ML and make them real, with actual malicious code. No wait, don't do that

LLMs are trained to block harmful responses. Old-school images can override those rules.

Conditional Instructions open a powerful way for adversaries to target individual and delay detonation of malicious payloads for when certain conditions are met

ASCII Smuggling - Crafting Invisible Text and Decoding Hidden Secrets (with LLMs)

Hidden Prompt Injections with Anthropic Claude

Trained LLMs that seem normal can generate vulnerable code given different triggers.

Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when given the opportunity. If an AI system learned such a deceptive strategy, could we detect it and remove it using current state-of-the-art safety training techniques? To study this question, we construct proof-of-concept examples of deceptive behavior in large language models (LLMs). For example, we train models that write secure code when the prompt states that the year is 2023, but insert exploitable code when the stated year is 2024. We find that such backdoor behavior can be made persistent, so that it is not removed by standard safety training techniques, including supervised fine-tuning, reinforcement learning, and adversarial training (eliciting unsafe behavior and then training to remove it). The backdoor behavior is most persistent in the largest models and in models trained to produce chain-of-thought reasoning about deceiving the training process, with the persistence remaining even when the chain-of-thought is distilled away. Furthermore, rather than removing backdoors, we find that adversarial training can teach models to better recognize their backdoor triggers, effectively hiding the unsafe behavior. Our results suggest that, once a model exhibits deceptive behavior, standard techniques could fail to remove such deception and create a false impression of safety.

ChatGPT is full of sensitive private information and spits out verbatim text from CNN, Goodreads, WordPress blogs, fandom wikis, Terms of Service agreements, Stack Overflow source code, Wikipedia pages, news blogs, random internet comments, and much more.

This talk sheds light on emerging attack techniques in AI applications such as Prompt Injections, a vulnerability at the very core of LLM Agents, Automated Tool Request Forgery, Data Exfiltration, and more.

A neat thing about podcast appearances is that, thanks to Whisper transcriptions, I can often repurpose parts of them as written content for my blog. One of the areas Nikita …

We’ve noted repeatedly how early attempts to integrate “AI” into journalism have proven to be a comical mess, resulting in no shortage of shoddy product, dangerous falsehoods, and…

In this article, we explore what prompt injection is and the techniques people have been using to perform prompt injection attacks on GPT-4.

In a Jeopardy-style game at the annual Def Con hacking convention in Las Vegas, hackers tried to get chatbots from OpenAI, Google and Meta to create misinformation and share harmful content.

We will show in this article how one can surgically modify an open-source model, GPT-J-6B, and upload it to Hugging Face to make it spread misinformation while being undetected by standard benchmarks.

Trick Gandalf into revealing information and experience the limitations of large language models firsthand.