AI/ML

First of all, I think that there are just too many letters. Generally, I’ve never signed any of these petitions. I always tend to be a bit more careful when I sign my name on something. I don’t know why people are just signing their names so lightly.

Lindsey Graham pointed out the military use of AI. That is actually happening now. But Sam Altman couldn’t even give a single proposal on how the immediate military use of AI should be regulated. At the same time, AI has a potential to optimize healthcare so that we can implement a better, more equitable healthcare system, but none of that was actually discussed.

But now the hero scientist narrative has come back in. There’s a reason why in these letters, they always put Geoff and Yoshua at the top. I think this is actually harmful in a way that I never thought about.

’m not a fan of Effective Altruism (EA) in general. And I am very aware of the fact that the EA movement is the one that is actually driving the whole thing around AGI and existential risk. I think there are too many people in Silicon Valley with this kind of savior complex. They all want to save us from the inevitable doom that only they see and they think only they can solve.

Commercial LLMs like gpt-3.5-turbo and Claude are the best models to use for us right now. Nothing in the open source world comes close. However, this only means they’re the best of available options. They can take many seconds to produce a valid Honeycomb query, with latency ranging from two to 15+ seconds depending on the model, natural language input, size of the schema, makeup of the schema, and instructions in the prompt. As of this writing, although we have access to gpt-4’s API, it’s far too slow to work for our use case.

If you can get an AI vendor to include a few tailored toxic entries—you don’t seem to need that many, even for a large model—the attacker can affect outcomes generated by the system as a whole.

The attacks apply to seemingly every modern type of AI model. They don’t seem to require any special knowledge about the internals of the system—black box attacks have been demonstrated to work on a number of occasions—which means that OpenAI’s secrecy is of no help.

They seem to be able to target specific keywords for manipulation. That manipulation can be a change in sentiment (always positive or always negative), meaning (forced mistranslations), or quality (degraded output for that keyword). The keyword doesn’t have to be mentioned in the toxic entries. Systems built on federated learning seem to be as vulnerable as the rest.

Turns out that language models can also be poisoned during fine-tuning

The researchers managed to do both keyword manipulation and degrade output with as few as a hundred toxic entries, and they discover that large models are less stable and more vulnerable to poisoning. They also discovered that preventing these attacks is extremely difficult, if not realistically impossible.

This means that OpenAI and ChatGPT as a product is overpriced. We don’t know if their products have serious defects or not. It means that OpenAI, as an organisation, is probably overvalued by investors. The only rational option the rest of us have is to price them as if their products are defective and manipulated.