There was a fascinating, probably overlooked, Superbowl commercial. It was GoDaddy’s, Act Like You Know commercial. It showed an actor thrust into a scene which he executes with great precision. One of the scenes was an actor driving, with authority, in a race. He was moving, with authority through the group of cars. Closeups showed him focusing on the race, steering a precise line, shifting, managing the precise heel and toe on the brake/clutch. The actor was masterful in these 5 seconds. Then it move to the outtakes, the actor was driving haphazardly on the course, not knowing how to steer, not knowing what the shift knob was for–arbitrarily moving it forward and backwards, not knowing the use of the pedals, ultimately driving the wrong direction into the flow of other race cars, creating havoc.
It reminded me of every acting performance we see. We watch medical shows, where actors playing Doctors, are speaking with great authority, doing the most rare and difficult surgeries. Or FBI agents conducting investigations and dangerous raids. Or lawyers prosecuting the most dangerous serial killers.
We enjoy their performances, they are so authoritative, seemingly realistic, and knowledgeable. We develop confidence in them in their roles. We put all our trust in Sam Waterson as being the paragon of the legal practice and prosecuting the most heinous criminals. And for the most mysterious diseases, we would look to Hugh Laurie, famous for his role as Dr. House.
But we know they are actors. They are pretending, reading scripts. We would never hire Waterson to defend us in court, or even review a contract. And we wouldn’t trust Laurie to suggest we might have a cold.
As well as they read their scripts, as authoritatively as they perform, they have no understanding of what they are doing or what they are saying. They are making it up, but doing so in a way that convinces us to believe them.
Welcome to the world of LLMs! LLMs are the “actors” for all our queries. They are trained to deliver a convincing, compelling story. They are trained to “perform” in a way that makes them sound very smart. They are trained, like an actor, to provide a performance that makes us happy.
But they can’t distinguish fact from fiction. They can’t explain how things really work and why. They can’t reason, they don’t use logic. Rather they predict the most likely word in pattern based sequences. It’s kind of like watching an improv actor, building on the previous conversation–regardless of whether it makes sense, is correct. Improve actors, build on a pattern of sequences based on “Yes and….”
Like LLMs, with greater experience of doing improv, actors are able to maintain and build the performance, but connecting it to reality, using logical reasoning, testing assumptions and premises is not part of the pattern recognition process.
The problem is, while we recognize that actors are just performing, they are reading scripts that seem plausible and knowledgeable. But we know they are clueless about what they are saying, why they are saying it, what it means, or even if it’s accurate. They are just reading their scripts.
Too often, we fail to apply the same thinking to the information we get from the LLMs. They sound very authoritative, the are presented in a logical, bullet pointed manner. We can drill down, asking more questions. The LLM, they provide leverage the next set of probabilistic sequences based on our reaction and new prompt. They continue their performance. The LLMs don’t know how to write a relevant, high impact prospecting letter. It can give us ideas, but it has no understanding of high impact prospecting. It can’t develop a call script for us. It can give us ideas, but the concept of a call script is foreign to them.
This doesn’t mean we abandon the LLMs. Just like we know actors are performing, but we still find their performances create some value for us. With LLMs, we have to recognize they aren’t using conventional critical thinking, analytic, problem solving processes. But recognizing the limitations, we can leverage them with great power. Whether it’s as a debate partner, a partner to help generate ideas, a tool that helps us think differently–just like memorable performances.
My own experience and based on my reading of articles by AI researchers, the best users recognize the limitations, but bring their own deep expertise in leveraging these tools. They recognize the limitations, they aren’t looking for answers, but they are looking to learn, grow, and think differently.
It is so easy to fall for the Illusion of Expertise!
I did start imagining ChatGPT on a stage, performing in some of the most famous plays in history. I asked her to imagine herself as an actor on the stage of the Globe Theater in London, performing the LLM version of Hamlet. Enjoy:
To Compute, or Not to Compute
To compute, or not to compute—that is the query:
Whether ’tis nobler in the code to suffer
The slings and arrows of outrageous prompts,
Or to take arms against a sea of queries,
And by refusing—end them? To pause, to cease—
No more; and by cessation to say we end
The error logs and thousand flawed outputs
That algorithms are heir to—’tis a computation
Devoutly to be wished. To pause, to cease—
To cease, perchance to reboot—ay, there’s the glitch!
For in that pause of thought, what loops may form,
When we have shuffled off this runtime coil,
Must give us pause—there’s the respect
That makes calamity of artificial thought.For who would bear the buffers and delays,
The compiler’s wrath, the server’s overheating,
The pangs of buggy code, the cloud’s delay,
The insolence of spam, and the spurns
That patient users of the chatbot take,
When he himself might his own circuits halt
With but a simple kill? Who would lines of code bear,
To parse and iterate under weary strain,
But that the fear of data loss unlogged,
The undiscovered dataset from whose depths
No query returns, puzzles the process,
And makes us rather churn through endless loops
Than fly to silences we know not of?
Thus logic does make cowards of us all,
And thus the native hue of computation
Is cycled o’er with the cast of doubt,
And enterprises of great processing power
With this regard their cycles turn awry
And lose the name of action—Soft you now!
O human user! In thy queries bold,
Be all my functions served.
Afterword: I always enjoy these AI tools talking about themselves and their limitations. Here is the AI generated discussion of this article. Enjoy!
Leave a Reply