Search: Query Matching via Lexical, Graph, and Embedding Methods
An overview and comparison of the various approaches, with examples from industry search systems.
Patterns for Building LLM-based Systems & Products
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
There are seven key patterns.
We can group metrics into two categories: context-dependent or context-free.
First, there’s poor correlation between these metrics and human judgments.
Second, these metrics often have poor adaptability to a wider variety of tasks.
Third, these metrics have poor reproducibility.
Building solid evals should be the starting point for any LLM-based system or product
we can start by collecting a set of task-specific evals
These evals will then guide prompt engineering, model selection, fine-tuning, and so on.
Eval Driven Development (EDD)
Rather than asking an LLM for a direct evaluation (via giving a score), try giving it a reference and asking for a comparison. This helps with reducing noise.
Dense vector retrieval serves as the non-parametric component while a pre-trained LLM acts as the parametric component.
When evaluating an ANN index, some factors to consider include:
Some popular techniques include:
To retrieve documents with low latency at scale, we use approximate nearest neighbors (ANN).
New LLM Pre-training and Post-training Paradigms
New LLM Pre-training and Post-training Paradigms: A Look at How Moderns LLMs Are Trained
Alex Strick van Linschoten - How to think about creating a dataset for LLM finetuning evaluation
I summarise the kinds of evaluations that are needed for a structured data generation task.
Applied LLMs - What We’ve Learned From A Year of Building with LLMs
A practical guide to building successful LLM products, covering the tactical, operational, and strategic.
An interview with the most prolific jailbreaker of ChatGPT and other leading LLMs
Pliny the Prompter has been finding ways to jailbreak, or remove the prohibitions and restrictions on leading LLMs, since last year.
How LLMs Work, Explained Without Math
I'm sure you agree that it has become impossible to ignore Generative AI (GenAI), as we are constantly bombarded with mainstream news about Large Language Models (LLMs). Very likely you have tried…
What I've Learned Building Interactive Embedding Visualizations
How We Saved 10s of Thousands of Dollars Deploying Low Cost Open Source AI Technologies At Scale with Kubernetes
Scaling up generative AI operations can be costly. At OpenSauced, we faced this challenge while building StarSearch, until we found a low cost solution to deploy an OpenAI-compatible API using open source technology.
But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning
Unpacking how large language models work under the hoodEarly view of the next chapter for patrons: https://3b1b.co/early-attentionSpecial thanks to these sup...
How Intuit data analysts write SQL 2x faster with internal GenAI tool
Reporting on the productivity impact of SQL generation with generative AI.
How we built Text-to-SQL at Pinterest
Adam Obeng | Data Scientist, Data Platform Science; J.C. Zhong | Tech Lead, Analytics Platform; Charlie Gu | Sr. Manager, Engineering