LLMs, and transformers in general, work based on statistical autoregression, not strict reasoning. The output they produce is not logical. It’s “speculative” or “intuitive” at best. They do not “sometimes hallucinate”. They hallucinate all the time, and it’s only a result of pure statistics that their answers in many cases appear to be right, or at least somehow useful.

We are constantly bombarded with speculative answers, which absolutely, every single time need to be verified using some external conscious reasoning. In the current deployments this processing is done on the integration (harness) side, thus the actual scraps of conscious intelligence in those systems are in fact massive if-else ladders written by a programmer, not some large neural network brain.

Many people ignore this fact, thinking that LLMs are the actual intelligence, and the rest is merely an “integration”. In reality it is completely the opposite. You have a model that gives you a ton of suggestions, for which you have no guarantees whatsoever, and this part is relatively easy to integrate into your system. But you still need some very capable intelligence to process and verify all those suggestions. And it cannot be another autoregressive agent, so writing a while loop feeding the model with its own output will not cut it. It has to be something conscious. Trivializing it leads to the belief that you can just simply wrap a model into a product. That once you have a model, most of the work is already done. It is not.

You need to review the code. You need to double-check every single fact. And it becomes increasingly more difficult with search engines giving you LLM-generated summaries by default instead of giving you the source. And the source is more and more often LLM-generated, so the objective truth is harder and harder to obtain.

The limit is not how many or how complex tasks can be completed by LLMs. The progress made by the models is astounding. The limit is that you will never be able to trust their answers. No matter how much and how complicated output LLMs will produce, the bottleneck will always be the external conscious intelligence required to process it. And the only thing like that you currently have is you.