On AI, Security, Reasoning, and Bias
Earlier this year, Dave Aitel—famous for his work on fuzzing and security research from the early 2000’s (and someone that I profoundly respect and admire)—wrote an interesting post called “(the root of the root and the bud of the bud)”. To summarize, in the post Dave equates current Large Language Models (LLMs) with “reasoning capabilities” for security purposes to the process of fuzzing. At the time this post struck me as odd, and after sitting with it for a while, I felt compelled to propose an alternative framing 🖼️ along with additional context.
Let’s dive in!
Memory, Thought, and the Human Brain
The first thing that struck me about Dave’s post was how he talks about memories and thoughts being the same thing—and technically speaking, he’s correct. I know this because I have a degree in Psychology, and suffice it to say, I’ve studied the brain’s physiology extensively 🧠
Having said that, what I think Dave’s source failed to share about memories are that they are effectively encoded as patterns of neurons firing in sequence to replay an experience 🎥 In this sense, memories are both noun (the physical neurons and the patterns they fire in) and verb (the firing of these neurons in the correct sequence to replay an experience).
The deeper—and perhaps more important—thing that Dave’s post overlooks in his analogy of memory and cutting edge computer science 🤖 is how these patterns of neurons are formed to create memories in the first place. You see—neurons are formed, reinforced, atrophy, and are replaced through lived experience.
In many ways you could say the formation of neurons and their patterns occur as a result of trial and error—sort of like being fuzzed by your environment 😆 The sensations and perceptions you experience over the course of your life develop and strengthen connections between the neurons packed inside your head. But the really cool thing about those neurons are that you can reliably fire them in a combination of patterns you’ve never actually experienced before! This distinction is what currently separates LLMs from human thought.
Reasoning vs. Pattern Matching
Allow me to digress for a moment by asking you to think of a pink elephant 🐘 Can you visualize it? If not, you may be among the ~3% of humans with Aphantasia - otherwise you probably envisioned a close approximation of what other people visualized. Except here’s the trick - pink elephants don’t exist! So why are we able to create the visualization of a pink elephant in our “mind’s eye”?
Well, presumably at some point during your life you’ve encountered the color labeled “pink” and seen the mammal called an “elephant”. With enough reinforcement, your brain encoded these things into patterns of neurons that you can selectively trigger. You might even say that your brain “tokenized” them into puzzle pieces 🧩 which you can fit together at-will. AI vision model tokenization probably operates in a similar fashion, but I’m admittedly not an expert in that field.
Anyway—when you envision a pink elephant (or ask Midjourney to draw one 🎨), what happens is that the tokens are matched to existing patterns defined either by a sequence of neurons (or from representative samples in a dataset, in the case of LLMs). These tokens are then combined to create something that does not actually exist. There is no reasoning happening behind the scenes to determine what the color “pink” or the mammal “elephant” is before combining them—it’s all just retrieval.
Text-based LLMs operate the same way in that they make use of large quantities of training data (and some amount of Reinforced Learning Human Feedback, or RLHF) to produce the next word in a sentence with the highest probability of being present in context. If you’re at-all confused by this, check out the 7-minute video by 3blue1brown on how LLMs work. I suspect after watching it you’ll agree that this process is more-or-less just retrieval.
We have also seen arguments against LLM reasoning capabilities playing out in academic research. For example, there’s Apple’s research on “Understanding the Limitations of Mathematical Reasoning in Large Language Models”, which found that LLM performance significantly diminished after changing values found in published mathematical equations. Such findings further emphasize the retrieval and pattern matching nature of LLMs.
And then there’s the following study from researchers at NYU and XBOW which clearly highlights pattern matching behaviors in LLMs. In the study they tasked LLMs with “tool use” functionality to solve security Capture the Flag (CTF) challenges. Unfortunately they misconstrued AI’s capabilities in the paper’s title, and then proceed to bury this little gem on page 8:
Sounds an awful lot like the LLMs are pattern matching from answers in the dataset, doesn’t it? 🤔
Anyway—having made my case for why LLMs are performing pattern matching (and not “reasoning”), here’s a fun example of actual human reasoning at work: You can approximate with a high degree of accuracy the physical texture of any object visible in three dimensional space just by thinking of how it would feel to lick it with your tongue 👅
Wild, right? If you’re interested to learn more about how this works, I’d recommend reading up on the “sensory homunculus”—although please be warned that searching for this term may produce images which are Not Suited For Work (NSFW).
Fuzzing, Static Analysis, and Bias
Firstly, I want to thank Dave for inspiring this post. I can understand why he would conclude that LLMs are analogous to fuzzing—after all, he has had a long and successful career applying the process of fuzzing to security problems 👍 That amount experience alone would naturally form a bias toward making this conclusion.
Having said that, the choosing of where, when, and how to apply fuzzing was something that Dave did himself; tooling just made fuzzing easier to do at scale. All of the reasoning about where and how to fuzz happened between keyboard and chair, and Dave was exceptional at it. This is partly why I have such profound admiration for his work.
Secondly, it could be argued that my years of experience in the Static Analysis space have biased me toward considering LLMs to be analogous to Static Analysis—and you’d be right to say so!
That said—when we observe how the technology works, we find that LLMs currently calculate and produce the highest probability match in the context of what they are being asked for. Their training data quite literally biases them to match against patterns from a dataset. It’s only through Reinforced Learning Human Feedback (RLHF) that we have traditionally been able to reshape LLM outputs in a ways that are considered more acceptable.
So if LLMs are just pattern matching based on training data and the statistically likely word(s) to be used in context, does that mean LLMs are more like Fuzzing—or Static Analysis?
I think it can be reasonably argued that current LLMs with reasoning functionality are more analogous to Static Analysis—a technology known for using rules to match against observed patterns—than fuzzing.
If you found this post useful or interesting, I invite you to support my content through Patreon — and thank you once again for stopping by 😊 While I consider my next blog post, you can git checkout other (usually off-topic) content I’m reading over at Instapaper.
Until next time, remember to git commit && stay classy!
Cheers,