Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
Large Language models and the reverse turing test
Mastering Diverse Domains through World Models
Unsupervised One-shot Learning of Both Specific Instances and Generalised Classes with a Hippocampal Architecture
Jiddu Krishnamurti clone
https://github.com/louis030195/krishnamurti-clone
Chat gpt diagram
Point·E: A System for Generating 3D Point Clouds from Complex Prompts
Roko's basilisk - Wikipedia
Roko's basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development.[1][2] It originated in a 2010 post at discussion board LessWrong, a technical forum focused on analytical rational enquiry.[1][3][4] The thought experiment's name derives from the poster of the article (Roko) and the basilisk, a mythical creature capable of destroying enemies with its stare.
Chinese room - Wikipedia
The Chinese room argument holds that a digital computer executing a program cannot have a "mind," "understanding" or "consciousness,"[a] regardless of how intelligently or human-like the program may make the computer behave. The argument was presented by philosopher John Searle in his paper, "Minds, Brains, and Programs", published in Behavioral and Brain Sciences in 1980. Similar arguments were presented by Gottfried Leibniz (1714), Anatoly Dneprov (1961), Lawrence Davis (1974) and Ned Block (1978). Searle's version has been widely discussed in the years since.[1] The centerpiece of Searle's argument is a thought experiment known as the Chinese room
![#84 LAURA RUIS - Large language models are not zero-shot communicators [NEURIPS UNPLUGGED]](https://rdl.ink/render/https%3A%2F%2Fi.ytimg.com%2Fvi%2F5Yd28ssDutA%2Fmaxresdefault.jpg?width=350&height=&ar=16:9&mode=crop&format=avif&dpr=2)
#84 LAURA RUIS - Large language models are not zero-shot communicators [NEURIPS UNPLUGGED]
In this NeurIPSs interview, we speak with Laura Ruis about her research on the ability of language models to interpret language in context. She has designed a simple task to evaluate the performance of widely used state-of-the-art language models and has found that they struggle to make pragmatic inferences (implicatures). Tune in to learn more about her findings and what they mean for the future of conversational AI.
Pod: https://anchor.fm/machinelearningstreettalk/episodes/84-LAURA-RUIS---Large-language-models-are-not-zero-shot-communicators-NEURIPS-UNPLUGGED-e1rri6k
Support us! https://www.patreon.com/mlst
Laura Ruis
https://www.lauraruis.com/
https://twitter.com/LauraRuis
BLOOM
https://bigscience.huggingface.co/blog/bloom
Large language models are not zero-shot communicators [Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, Edward Grefenstette]
https://arxiv.org/abs/2210.14986
[Zhang et al] OPT: Open Pre-trained Transformer Language Models
https://arxiv.org/pdf/2205.01068.pdf
[Lampinen] Can language models handle recursively nested grammatical structures? A case study on comparing models and humans
https://arxiv.org/pdf/2210.15303.pdf
[Gary Marcus] Horse rides astronaut
https://garymarcus.substack.com/p/horse-rides-astronaut
[Gary Marcus] GPT-3, Bloviator: OpenAI’s language generator has no idea what it’s talking about
https://www.technologyreview.com/2020/08/22/1007539/gpt3-openai-language-generator-artificial-intelligence-ai-opinion/
[Bender et al] On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
https://dl.acm.org/doi/10.1145/3442188.3445922
[janus] Simulators (Less Wrong)
https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators
Training language models to follow instructions with human feedback
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
CICERO: An AI agent that negotiates, persuades, and cooperates with people
PROGRESSIVE DISTILLATION FOR FAST SAMPLING OF DIFFUSION MODELS
ChatGPT: Optimizing Language Models for Dialogue
We’ve trained a model called ChatGPT which interacts in a conversational way. The dialogue format makes it possible for ChatGPT to answer followup questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests. ChatGPT is a sibling model to InstructGPT, which is trained to follow an instruction in
We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.
We randomly selected a model-written message, sampled several alternative completions, and had AI trainers rank them. Using these reward models, we can fine-tune the model using Proximal Policy Optimization.
A Gentle Introduction to Pooling Layers for Convolutional Neural Networks - MachineLearningMastery.com
Convolutional layers in a convolutional neural network summarize the presence of features in an input image. A problem with the output feature maps is that they are sensitive to the location of the features in the input. One approach to address this sensitivity is to down sample the feature maps. This has the effect of […]
Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map. Two common pooling methods are average pooling and max pooling that summarize the average presence of a feature and the most activated presence of a feature respectively.
Restricted Boltzmann Machines for Collaborative Filtering
Training Products of Experts by Minimizing Contrastive Divergence
LAION-5B: An open large-scale dataset for training next generation image-text models
Neuro-symbolic AI - Wikipedia
Neuro-symbolic AI integrates neural and symbolic AI architectures to address complementary strengths and weaknesses of each, providing a robust AI capable of reasoning, learning, and cognitive modeling. As argued by Valiant[1] and many others,[2] the effective construction of rich computational cognitive models demands the combination of sound symbolic reasoning and efficient machine learning models. Gary Marcus, argues that: "We cannot construct rich cognitive models in an adequate, automated way without the triumvirate of hybrid architecture, rich prior knowledge, and sophisticated techniques for reasoning."[3]. Further, "To build a robust, knowledge-driven approach to AI we must have the machinery of symbol-manipulation in our toolkit. Too much of useful knowledge is abstract to make do without tools that represent and manipulate abstraction, and to date, the only machinery that we know of that can manipulate such abstract knowledge reliably is the apparatus of symbol-manipulation
The Principles of Deep Learning Theory
This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how...
Perplexity Intuition (and Derivation)
Never be perplexed again by perplexity.
Less entropy (or less disordered system) is favorable over more entropy. Because predictable results are preferred over randomness. This is why people say low perplexity is good and high perplexity is bad since the perplexity is the exponentiation of the entropy (and you can safely think of the concept of perplexity as entropy).
Why do we use perplexity instead of entropy? If we think of perplexity as a branching factor (the weighted average number of choices a random variable has), then that number is easier to understand than the entropy
Perplexity - Wikipedia
In information theory, perplexity is a measurement of how well a probability distribution or probability model predicts a sample. It may be used to compare probability models. A low perplexity indicates the probability distribution is good at predicting the sample
MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields
The spelled-out intro to language modeling: building makemore
We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT....
No Language Left Behind: Scaling Human-Centered Machine Translation
Open-Ended Learning Leads to Generally Capable Agents
Artificial agents have achieved great success in individual challenging simulated environments, mastering the particular tasks they were trained for, with their behaviour even generalising to maps and opponents that were never encountered in training. In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. Training an agent that is performant across such a vast space of tasks is a central challenge, one we find that pure reinforcement learning on a fixed distribution of training tasks does not succeed in. We show that through constructing an open-ended learning process, which dyna
We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks
Algebra, Topology, Differential Calculus, and Optimization Theory For Computer Science and Machine Learning
The spelled-out intro to neural networks and backpropagation: building micrograd
This is the most step-by-step spelled-out explanation of backpropagation and training of neural networks.
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction