What I have found, after building a lot of agentic systems, is that asking an LLM for the “analytical” recommendation is one of the least reliable things you can do with it. Not because the models are bad. Because you are asking the wrong thing of them.
Here is the correction I want to make up front, because I used to explain this the lazy way and the lazy way is wrong.
Randomness is not what causes hallucination
The popular explanation is that models hallucinate because of the randomness in sampling, and that if you turn the temperature down to zero you get the reliable version. That is not true. A model decoding greedily at temperature zero still fabricates. It fabricates confidently, and it fabricates the same thing every time you ask.
The reason is in the objective. The model is trained to produce a plausible continuation of the text, and a fabricated citation, a fabricated statistic, and a fabricated executive name are all extremely plausible continuations. On top of that, the way models are scored during training generally rewards producing an answer over saying “I don’t know,” so the behavior you get is a system that is very good at sounding like it knows.
Sampling randomness does not create that. It just varies which fabrication you get on any given run.
This matters because the diagnosis determines the fix. If randomness were the cause, the fix would be a temperature setting. It isn’t, so it isn’t.
The randomness is still the creative spark
The part of the original idea that holds up completely is this. The variability in these models is genuinely where the useful non-obvious output comes from. Ask for twenty campaign angles, or ways to segment a market, or things that might explain a weird pattern in the data, and the variability is doing real work for you. Run it twice, get two different sets, keep the good ones.
So the question is not how to suppress it. The question to ask is: on this particular step, do I want the model being inventive or not?
That question has a clean answer on almost every step of a real workflow, and most people never ask it. They write one prompt that mixes both modes together and then wonder why the output looks confident and turns out to be partly made up.
Where creativity belongs and where it doesn’t
| Creative is the point | Creative is a defect |
|---|---|
| Generating hypotheses about what might drive an outcome | Extracting a number, a name, or a date from a source document |
| Proposing what to measure and which features might matter | Arithmetic of any kind |
| Designing the search strategy: where would this data even live | Scoring, ranking, and thresholding |
| Naming and describing segments once the math has produced them | Eligibility and exclusion decisions |
| Drafting outreach copy, subject lines, invitation language | Anything that gets written into a system of record |
The trap is that the same model does both, in the same conversation, in the same tone of voice, and gives you no signal about which mode it was in.
The actual method: ask for the framework, not the conclusion
Here is the reframe that changed how I build these systems.
Do not ask the model for the recommendation. Ask the model to find the real data, write the code that analyzes it, and explain the framework. Then run the code. The recommendation falls out of the execution, not out of the model’s opinion.
The model is the analyst who designs the method. It is not the calculator that produces the number.
That single change gets you something an opinion can never give you: a result that can be recomputed. Someone can take your inputs, run your code, and land on the same answer. If they disagree, they disagree with a specific line of a specific transformation, which is a conversation you can actually have. That is what analytically defensible means, and no amount of prompt engineering on “please be accurate” will get you there.
Question
The job, stated as a question.
Where does the data live?
The model, used creatively, proposes what to measure and where the data would be.
Real data
Stored, hashed, and dated.
Write the analysis
The model, used creatively, writes the code. It does not produce the number.
Execute
The code runs. This step is deterministic.
Answer
Every input traces to a source.
Explain
The model, used creatively, says what the result means.
How to actually make it repeatable
Temperature is the wrong lever, so here are the ones that work. All of them move the determinism out of the model and into the system around it.
| Lever | What changes |
|---|---|
| Pin the model version | A pipeline that says “the newest model” changes under you, and nobody can say why. |
| Store the evidence, not just the answer | Save every retrieved document with its URL, its retrieval date, and a hash. A live web that has changed is not a re-run. |
| Code and fields on the analytical steps, never a prose conclusion | A number in a field with a source URL can be checked. The same number inside a paragraph cannot. |
| Verify citations mechanically | Fetch the URL, confirm the claim is in the document, and fail the record if it is not. |
| Use disagreement as a signal | Run an extraction three times. Agreement is probably fine. Disagreement is the queue for a person. |
| Put the human where the ambiguity is | A person reviews the records the system already flagged, not everything and not nothing. |
What this looks like in practice
In the next posts in this series I am going to run this all the way through on a hard, real problem: finding new wealth management clients in a specific wealth band, working out what they can actually invest, and setting up a recurring engagement loop around them that doesn’t cross legal or creepiness lines.
It is a good test case because the naive version is so tempting. “Find me ten high net worth prospects for my firm” is a prompt that will produce ten confident-looking names in about four seconds, and every one of them is a liability. The analytical version takes more design and produces something you can put in front of a compliance officer.
That is the whole difference, and it is worth understanding where it comes from.