❯The Paradoxes of AI-first and Everything-LLM
Reasoning About the Logical Limits of AI in Engineering Design.
In this article I want to attempt to analyze a logical inconsistency I've noticed concerning AI Engineering and the design of AI-first systems. These narratives are of various nature, but they have a common theme: AI proposes a fundamentally different way of doing engineering, to the point that we can replace human intervention with LLM output. Writing software feels easier, because the mental labor and the costs associated with implementing logical rules with software can be bypassed replacingthem with an API call to an LLM model.
This is not a victory, there is a flaw in this way of thinking. Complexity hasn't gone away, it has just replaced business logic with output control logic, increased security and a more complex architecture.
The paradox is that with AI, we may end up having to manage more complexity than before. If we want adaptability, we pay with complexity, if we want simplicity, we end up with an inflexible and dull system. This is a logical problem, not a technological one, and it cannot be fixed with more technology.
It's useful to introduce the following point of view which is foundational for the rest of this article. Systems can be broken down and analyzed from two perspectives, the logical and the technological. Logical aspects come first, since technology implements and presupposes the logical ones.
Logic vs Implementation
The main observation I want to make is based on the conservation of complexity: there are fundamental limits to how we can organize the logic of our design before we hit a limit. We can move complexity to other subsystems but we can't eliminate it, this is true irrespective of the technology we use. The problem and its constraints demand e certain set of logical steps and state transitions. As long as we talk about AI though, we're looking at our system from the perspective of implementation rather than logic, and, for the most part, we can only hope to bring small improvements over time, directly related to the performance increase of the latest software releases. The logical layer is where major improvements can be made.
If we accept this point of view, certain conclusions become immediate.
From a technological perspective AI is surely a novelty, from the logical one, hardly so. I would challenge the idea that a tool, even AI, represents a novel logic that fundamentally changes how we think about engineering. In this sense, AI does little to "solve" design challenges.
The point is not being anti AI, but it's incorrect to point at the tool when the limit is the logic. This is a categorical mistake.
There is a small selection of tasks where AI unlocks new possibilities. Its value comes when the problem itself is fuzzy, ambiguous, or expensive to encode with rules. But even in those cases, AI is just one unit of a complex machine. As soon as we venture outside the implementation of the individual building block, we're again faced with the same issues I've pointed at above, we're still bound to logical limits, we still need to understand state transitions and failure modes and design accordingly. If nothing else AI exacerbates these problems. As of today, AI hasn't really impacted the logic of design, not yet, and the limitation is not technological and cannot easily be eliminated.
Conservation of Complexity
It's tempting to replace every component of our system with an LLM call via an MCP server or an AI equivalent, but that turns a local microsecond operation into a blocking network request. You are burning through compute and tokens, increasing security concerns, multiplying your API costs and exposing yourself to novel failure modes.
Prototyping with AI can feel like a breeze, no logic to program into your software, just let AI make the decision. That approach may work in development, but production systems are a very different environment. Once you have to meet reliability and performance guarantees, constraints become more apparent and the trade-offs begin to accumulate. At that point the idea that AI isn't bringing those magical advantages starts to seep in.
It's paradoxical that to resolve engineering problems we compensate with more complexity. Now that we have a central AI component, we must fine-tune it, generate datasets of features and versioned training datasets. Then we must evaluate the model. If the performance isn't satisfactory we must generate a dataset of rules and update the "long term memory" to continuously improve the model. All of this must be maintained of course.
AI first systems require an array of tools to be production ready:
→ Feature datasets
→ Test/Training datasets
→ Synthetic data generation
→ Fine-tuning
→ Prompt revisions and versioning
→ Datasets with Improvement rules
→ Monitoring output drift
→ Monitoring input dataset drift
→ Compensation logic for output randomness
Claiming you have an 'AI-first' system is appealing, but dialing back the rhetoric reveals a massive foundation of traditional software. Most of the heavy lifting is actually standard code and API calls designed to tame LLM randomness, measure performance, and keep the system running.
Conclusion
AI was supposed to simplify things. I'm arguing that accounting for errors, supervision, verification and model improvement, it often doesn't make things better, and that's not for superficial reasons that can be easily dismissed. The AI fever is leading us to paradoxes and architectural monstrosities. We may be persuaded by the fact that the initial design steps are simplified by replacing the costs of implementing logic with a fuzzy LLM system, but the conservation of complexity is soon going to bite back, forcing you to resort to compensatory systems that often add more burdens rather than solving problems.