A few prompting techniques have consistently helped improve performance across various models and tasks: n-shot prompts + in-context learning, chain-of-thought, and providing relevant resources.

Have small prompts that do one thing, and only one thing, well

Prompting an LLM is just the beginning. To get the most juice out of them, we need to think beyond a single prompt and embrace workflows. For example, how could we split a single complex task into multiple simpler tasks?

The most successful agent builders may be those with strong experience managing junior engineers because the process of generating plans is similar to how we instruct and manage juniors. We give juniors clear goals and concrete plans, instead of vague open-ended directions, and we should do the same for our agents too.

Hallucinations are a stubborn problem. Unlike content safety or PII defects which have a lot of attention and thus seldom occur, factual inconsistencies are stubbornly persistent and more challenging to detect. They’re more common and occur at a baseline rate of 5 – 10%, and from what we’ve learned from LLM providers, it can be challenging to get it below 2%, even on simple tasks such as summarization.

First of all, you need <em>the expertise to judge an output</em>, and second you need <em>the discipline to exert the effort </em>required to assess an output.

<p>More thoughts to come on this, but for now, I think there are a few questions we should be asking:</p> <ul> <li>Does this person have the knowledge and expertise to judge this output?</li> <li>Is it reasonable to expect this person has the discipline to evaluate the outputs in detail?</li> <li>What is the risk if output errors are not caught?</li></ul>

Modern professional work consists of a wide range of activities, rather than a single specialization.

AI acts as a leveler, raising everyone to a minimum level of performance.

Just because early results for AI suggest that only lower performing people benefit does not mean that this is the only possible pattern. It may be that the reason only lower performers gain from AI currently is because the current AI systems are not good enough to help top performers. Or, alternately, it might be that top performers need more training and work to get benefits from AI. If either of these conditions prove true, and they certainly seem plausible, then AI might act more as an escalator, increasing the skills for everyone, from top to bottom performers. After an adjustment period, the relative skill positions stay similar, but everyone gets more done, faster.

<span>Alternately, it might be that some people are just really good at working with AI. They can </span><a href="https://www.oneusefulthing.org/p/centaurs-and-cyborgs-on-the-jagged" rel="">adopt Cyborg practices </a><span>better than others and have a natural (or learned) gift for working with LLM systems.</span>