I’ve been using generative AI tools to write code since day one.
They write substantial amounts of my code today.
But besides code, they also make bugs.
Not syntax errors or missing includes. Real, behavioral bugs. Something is wrong in the code, something that requires human intervention.
Then, someone says: “Oh, don’t use that model, use this model.” So I use the new model, and it makes bugs again.
They say: “Wait for the new version, it will fix everything.” Summer comes, and the bugs are still there.
I know why. The story is very simple. Bugs are literally built into every LLM. They will never stop.
To understand why, we need to look under the hood. This article will show you what’s inside any LLM in existence. And it will reveal the mathematical foundations that make bugs inevitable.
Peeling Back the Layers: What’s Inside an LLM?
We will remove all the refinements and tools added to an LLM on its outside. We’ll look into the bare model.
When you peel off the chat interfaces, IDE integrations, agents, and plug-ins, you find the core. At the heart of every LLM lies a deep neural network.
Deep neural networks are topologically equivalent to multi-layer perceptrons. This is historically the oldest topology of artificial neural networks.
A multi-layer perceptron consists of layers. Each layer consists of neurons. Each neuron has multiple inputs and one output.
graph TD
subgraph Multi-Layer Perceptron
A[Input Layer] --> B{Hidden Layer 1};
B -- Weighted Sum & Activation --> C{...};
C -- Weighted Sum & Activation --> D[Output Layer];
end
subgraph Neuron Detail
I1[Input 1] --> N;
I2[Input 2] --> N;
I3[...] --> N;
N{Sum(Inputs * Weights)} --> AF(Nonlinear Activation Function);
AF --> O[Output];
end
The first layer only receives inputs. The transforms happen in each neuron.
- A neuron receives its inputs.
- It multiplies them by weights.
- It sums those up.
- It passes the sum through a nonlinear activation function.
Do that for all neurons in a layer, and you have the layer’s output. Pass it to the next layer, and the next, until you reach the output layer. That gives you the output of the entire network.
Those weights are the parameters of the artificial neural network. You know them as the parameters of the LLM.
A Brief History of Neural Networks
The history here is fascinating.
It started in 1943 with the first artificial neuron. Then, in 1958, the first single-layer perceptron was designed. The multi-layer perceptron followed.
Deep neural networks, as we call them today, were built during the 1960s. This technology is more than half a century old.
Training vs. Running: A Critical Distinction
There’s a crucial distinction I must clarify. Training a neural network and running it are completely different activities.
We train these networks using supervised learning. A network sees large amounts of data on its input. For every input, there is a known, correct output.
During training, we adjust the weights. The goal is to reduce the discrepancy between the network’s actual output and the desired output. That is training. It can take a long time and complex algorithms.
But once training is complete, you get your model. That’s it. The training has no more effect once you start using the model.
So, what does the network do in practice? It accepts inputs, forwards them through the layers, and produces an output. That’s it.
Want the next token? Just pass a new input and wait for the new output. Any “knowledge” the network has is baked into the weights of its neurons.
An improved training algorithm could make it cheaper to produce the next model version. That’s nice. It could also help make a model with less error.
But is it possible to create a network with zero errors? How much can we truly reduce the error?
The Mathematical Certainty of Imperfection
This is where the famous Universal Approximation Theorem comes to the table.
The nonlinear transforms in the neurons are not arbitrary. They make the entire network a nonlinear transform of its inputs. This is vital because such functions have provable properties.
The Universal Approximation Theorem points out a few of these properties. It first appeared in 1989. But its roots go back to the mathematical theory of the 19th century, like Weierstrass’s theorem from 1885.
The Universal Approximation Theorem Explained
Here is what the theorem says:
An artificial neural network can approximate any continuous function on a bounded input range at an arbitrary level of accuracy.
What does this mean in practice?
If you approximate the probabilities of words in a language, you can train the network to speak that language. If you train it on the probabilities of the next token in code, it will write code.
Artificial neural networks learn the statistical patterns of a language. Then, they reproduce those patterns when used.
Consequences for Code Generation
Now, you can see the consequences.
-
There is always an error. While it can approximate a function to an arbitrary level of accuracy, it’s never perfect.
-
It only knows what it has seen. The outputs always resemble the outputs it saw during training. It cannot work on things it didn’t see. You can’t show it a book and expect it to write software based on principles from that book. That’s science fiction.
-
It fails outside its comfort zone. The network only has that low error rate on the bounded input it was trained on. Move out of that input, and the network will start making gross mistakes. Show it words it never saw, a new programming syntax, or a library it never trained on, and the model will start hallucinating, big time.
It’s built into them.
Why Your Code is Just a Statistical Guess
So, how does an LLM write code? It’s essentially simple.
It takes the code it sees already. It takes your prompt. Then it tries to predict the next token. And the next, and the next.
But you must understand certain aspects of this behavior.
[!NOTE] The code you pass in a prompt has never been seen during training. The artificial neural network will always operate in interpolation mode.
It will always try to recognize fragments of someone else’s code it saw during training. Then, it produces a fragment of your code. Therefore, the output will never be exact.
Is that code correct? Is it incorrect? There is no way for the model to tell.
It requires a human in the loop to see the code and judge what is good and what is bad. If a human doesn’t notice an issue, the bug remains.
The Final Verdict: Humans Remain in the Loop
The Universal Approximation Theorem also tells us when to expect bugs.
Stable error rates are expected within the bounded region of inputs. But even there, the situation isn’t uniform.
In regions with millions of training samples, the LLM will be confident. It will write perfect code because it can interpolate from numerous flawless examples.
But if you’re working on anything experimental or with advanced algorithms, real samples in the training set are rare. The error rates will naturally grow. You cannot expect an LLM to produce exotic code with confidence. It will make many, many bugs.
And if you step outside the region it was trained on, the error rates will go wild. It will probably create more problems than it solves.
The message is clear. LLMs will never stop making bugs.
No matter how large the model, how good its architecture, or how advanced the training… what you see today is essentially the ceiling. That’s what LLMs are.
There won’t be large improvements in LLM accuracy anymore. That’s my prediction.
They will get faster. They will be more convenient, yes. But they will never become absolutely correct.
They will keep adding bugs at some rate. And you will always have to be there to read the code, and to find and correct those bugs.