Why AI gets stuck in weird explanations
This explains everything about the bizarre responses people sometimes get.
People are understandably concerned when ChatGPT is given a simple question and produces an answer that’s completely bonkers.
There's a reason for that weirdness.
And understanding it will make you much better at using AI.
The problem most people miss
Here's what's happening behind the scenes: AI language models like ChatGPT write one word at a time, from left to right.
Once they "say" something, they can't take it back.
It's like writing with a pen instead of a pencil—no erasing allowed.
Why this creates bizarre responses
Imagine you're working on a legal brief and you write down your first analysis. Halfway through, you realize that opening line was completely wrong.
What do you do? You delete it and start over.
AI can't do that.
Instead, it has to keep building on that wrong foundation—creating increasingly elaborate explanations to make everything fit together somehow.
A real example
Let's say you ask: "If a client owes $5,000 and pays $3,000, then incurs $2,000 in additional fees, what's the total balance?"
The AI might start with a mistaken interpretation: "The client owes $5,000 and pays $7,000..."
Now it's stuck. It can't erase that wrong "$7,000."
So it doubles down: "Well, the client must have made an overpayment from a previous account, so when we factor in the credit balance..."
The explanation gets weirder because the AI is trying to make bad math work.
This explains everything
Those moments when ChatGPT gives you perfectly reasonable legal analysis for three paragraphs, then suddenly veers into something that makes no sense?
It made an early error and spent the rest of the response trying to justify it. (Not unlike many humans we know, right?)
The bottom line
Even the newest AI models have this limitation. They're better at avoiding early mistakes, but when they make one, they're still trapped by it.
This is why you always need to fact-check AI output—especially for complex legal work where one wrong assumption can derail everything.
Understanding how AI "thinks" is the key to using it effectively. Once you know its limitations, you can work around them.
Want to learn more tricks for getting better results from ChatGPT? Check out my ChatGPT Lab, where we meet weekly to share what actually works.
;-)
Ernie
P.S. Check out my Inner Circle, which is a great way to learn about the best tech tools for improving your law practice. And this 5-question survey is also a good way to start (if you take it, I'll give you some recommendations tailored to your situation at no charge)