A guide to LLMs for my father

Dear Dad,

Everyone at your work is excited about AI. Your younger coworkers are starting to send you AI generated reports and decks at a pace that's frankly impossible to keep up with. Someone set up a collection of "helpful prompts" you open in Notepad on your computer. You sent us (your children) an email with the "something big is happening" blog post, and I responded with Ed Zitron's annotated version. You sent us a Word document with "the most commonly used AI thinking frameworks" and associated helpful prompts. It is quite clearly AI generated.

I know you are worried about my future. I know you don't want me to fall behind. For the record, I use LLMs a lot at work. At my previous job, I even worked on quite a bit of LLM tooling. I don't particularly like using them, but I'm a member of a team. The play has been called to actively use these tools to be more productive, and I intend to hit my assignment. I find it strange that LLMs are the only product where the customer is responsible for figuring out what they're useful for, but that's neither here nor there.

At your work, you were recently given the task of evaluating the effectiveness of using LLMs to write proposals. As someone close to you who happens to work in the one industry these things really took over, I want to help you get all the context you need to make that decision.

The word guesser

So what is an LLM?

A large language model is a computer program designed to predict words. If you give it a sentence to read, with the last word erased, and ask it to guess what the final word will be, you have used the LLM exactly how it was trained and meant to be used. Fundamentally, any time you ask an LLM a question, you are saying "here are some words, give me the statistically likely next words to follow".

These models are really quite excellent at guessing what the next words will be, because they have been trained on the largest corpus of written information ever assembled in the history of mankind, much of it taken without the consent of the authors and in direct violation of copyright laws. But that doesn't change the fact that the models are not thinking, they are calculating statistically likely next words.

The prompt framework document you sent me is broken down by use case. Here's the example prompt for making a Bid/No Bid decision:

Act as a Federal Capture Executive. Conduct a SHRED analysis. Assess customer need, incumbent position, competitors, probability of win, risks, required investments, and provide a Bid/No-Bid recommendation.

You need to know that no large language model is capable of doing any single item on that list. It cannot recommend something. It cannot assess customer need. It can take in this series of words and guess what the next words are.

It cannot think.

Large language models simply produce plausible text. There are times when that's good enough, and times when it's not.

The coding harness

It may seem like LLMs are rapidly improving, but in reality the language models have pretty much stalled. The seeming improvement is partly due to a barrage of nonsense benchmarks, but also because there has been genuinely massive financial investment in the software around the word guessers. That wrapping software is often called a harness. You've heard of these. Claude Code is a harness that wraps Anthropic's models.

The job of the harness is to provide a myriad of tools for actually doing things on the computer in response to the text output of an LLM, as well as gathering text based context for it to read before guessing at more words. This works shockingly well in software engineering, because code is just words the computer knows how to translate into actions. When the LLM predicts a series of words that looks an awful lot like code, programmers can run the code and validate that it does what we want it to. We have all sorts of automated ways of doing that.

The point I really want to get across here is that two things are feeding into LLMs being relatively good at coding compared to how they perform at other tasks.

None of this makes LLMs good at anything beyond producing "candidate code" for engineers to throw at the wall of automated checks and see what sticks. None of it gives LLMs the ability to use logic. Do not think "LLMs good at coding" implies "LLMs good at tasks generally". Neither statement is true, but the latter statement is ultra mega especially not true.

Autonomous AI agents

We also need to talk about agents. There is no such thing as an AI agent. It's a marketing term (like "AI" which generally means "thing I'm selling"). LLMs are not and have never been "autonomous". You have to give it words, and ask it to guess more words. When people want their AI agents to act autonomously, they set up a bunch of regular old software to constantly source and stuff words into the LLM and record what comes out. Most harnesses have modes for doing this.

Think of it like a game of telephone, where you say the first word and the rest of the line is just the agent talking to itself over and over. Eventually, it loses the plot completely. Believe me, I've really tried to find a set and forget workflow, but 99% of the time, to get a result you remotely want, you need to babysit it the whole way through.

"That's AI"

LLMs are fundamentally agents of averageness. They produce statistically likely text, which means they seldom produce something that triggers a novel idea. They also produce text that has a certain feel to it, similar to how one can quite easily spot AI art. If the content of a proposal is AI generated, there's a solid shot the person reading it will immediately recognize that it's LLM generated text. Some people will find this frustrating, others who are fond of AI may not (although they probably had their agent summarize the doc for them anyway). It's worth noting that teenagers these days will call something "AI" as slang for it being of low quality. Food for thought.

My bias is that if it's worth doing right, you shouldn't use AI. If it's not worth doing right, you shouldn't bother doing it at all. Perhaps a less rigid intepretation is that you have to decide if you need "a deliverable" or "a quality deliverable". There are plenty of things we do at work to tick a box. There are plenty of tasks we timebox to preserve space for other priorities. One of the promises of AI is time to focus on the most interesting or critical pieces of our jobs.