09:41:02.118 [DEBUG] $ cd .. → back to homepage

09:41:02.121 [INFO] loading module:machine…

09:41:02.152 [INFO] found 2 posts matching module="machine"

Runtime Quality Requires Humans

module=", "author=Ben Zeiglerpublished=2026-11-17read_time=16m
An abstract illustration of squares and circles
# Abstract representation of a complex system by geralt on Pixabay

As I was getting ready to publish this article, Microsoft published a preliminary analysis (Reddit copy) of what caused the massive Outlook outage last week. It turns out it was triggered by a runtime performance issue with lock contention in a Windows process running on the networking gateway servers. This caused a cascade of failures which took more than a day to recover from. It’s hard to know if Microsoft’s recent AI-driven layoffs and policy changes caused this failure, but it is a great example of how subtle runtime issues like lock contention can destroy the health of a complex software system.

If you’ve ever tried to actually use a computer, you’ve run into shitty software that crashes or breaks in confusing ways. Sometimes, this is because of functional bugs in the code that can be prevented with good unit tests. Current AI agents like Claude Code are great at writing and maintaining unit tests, often better than the average programmer. But unit tests will not find the critical bugs that only manifest when features interact with real users and external systems in complex, hard to predict ways. When building a complex system that needs to work in many different situations, you can’t solely depend on AI agents because they are not good at maintaining high Runtime Quality. Companies that try to have AI do everything will keep running into major problems that can kill entire projects.

Runtime Quality?

Runtime Quality is not an established term, but I am using it to mean the aspects of Software Quality that only manifest when trying to actually use a system in the real world. You could also use the term Dynamic Structural Quality but that’s too many words. In Software Engineering terms, I am talking about the non-functional requirements that affect program execution:

  • Reliability is the health of a software system in normal conditions. If a program crashes randomly due to memory corruption it is not reliable.
  • Robustness is the ability of software to handle unusual user inputs and unexpected errors. If a program fails when you enter weird characters in a text box it is not robust.
  • Performance is how quickly the system can perform critical tasks in real world situations. Systems that become unusably slow under load are not performant.
  • Efficiency is how productively a system uses expensive resources like CPU and RAM. If a system costs too much money to run it is not efficient.
  • Security is also a critical runtime quality, but there are a ton of articles about this already so I’m not going to focus on it here.

The best book I have read about runtime quality is The Pragmatic Programmer, which definitely influenced my career as a game programmer and is well worth reading today. I enjoy working on complex problems with real-world implications, which means that I have spent most of my career working on performance, reliability, and robustness issues in complex systems instead of implementing sexy new features.

Runtime failures can be discovered by trained QA (or agents) running integration and smoke tests before release, but some issues will only be discovered after your system goes live and fails to actually work. In my experience so far, AI agents are very good at diagnosing the cause of runtime failures using crash dumps and logs, but they are pretty bad at anticipating possible failures before they happen. I’m sure they’re getting better at this over time, but there are some fundamental reasons why experienced programmers will outperform agents.

Open Goals or Explicit Requirements

The currently available AI agents tend to be good at two kinds of tasks: open-ended brainstorming tasks with relaxed requirements for success (like helping to design this website), and well-defined tasks with clear success criteria. If you ask an agent to do something with complicated implicit requirements for success (like writing a legal brief or research article from scratch), it will often hallucinate information or lie about what it actually did. This makes total sense given how LLMs work, and stressed-out humans often do the same thing when they are forced to solve problems they don’t know how to solve.

AI agents are good at open-ended tasks because they have learned a massive amount of background information on many technical topics. Claude knows far more about most programming languages than I do, because it was trained on millions of lines of code and specifically taught coding skills by thousands of hours of feedback from skilled programmers. I have a pretty wide knowledge base and experience with many different areas of programming, but AI agents have more background knowledge and learned experience than I do on most technical topics. This wide knowledge base is a big reason why agents are good at diagnosing crash reports and processing confusing bug reports.

But the amount of dynamic memory and task-specific learning available to an individual Claude instance is severely limited compared to a human programmer. Some of this is due to the raw cost and technical difficulty in using large amounts of computer RAM, which is why no one can afford to buy a new graphics card. This can be addressed by increasing the amount of RAM allocated to each instance, but that does not solve the entire problem. Even frontier agents with tons of memory still need to have their working memory regularly reset to avoid getting saturated with useless information. This is an active area of research (like teaching AI how to sleep), but it will be years before agents can compete with experienced humans at task-specific learning and memory.

The reason that agent systems like Claude can handle difficult coding tasks with clear requirements is that they were taught how to write and use tools like unit tests to check that the code they write works correctly. This is great for meeting well-defined functional requirements or comparing with existing software, which is why they’re so good at reverse engineering and data conversion tasks. Some of the fear and hype about AI replacing human jobs is overblown, but any job devoted to performing predictable data transformations or building simple custom systems is at high risk for AI replacement. That’s a serious problem for the health of many industries, but AI will not be replacing humans any time soon when building complex systems that require creativity and long term technical vision.

Runtime Complexity

What makes software complex in the first place? Complex System has a formal definition, but the TLDR is that a system is complex if it has many different components with different inputs and outputs that interact to create emergent and hard to predict properties. Online multiplayer games are a good example because they are basically always complex systems. You can’t build a successful complex system by optimizing for a single success factor because there are always multiple competing factors that need to be balanced. In a multiplayer game, some key factors are the enjoyment and engagement of each player, the costs of hosting and developing the game, and the revenue that allows the game to continue. Runtime quality affects all of these factors and too many runtime failures will kill any project.

It’s always been hard to write automated tests for multiplayer games, because they involve real-time interactions with players that are extremely difficult to predict and understand. You can build comprehensive unit tests for simple real-time systems where all possible states are well understood, but this is not feasible for a multiplayer game. To fix runtime failures in a multiplayer game you need to rely on a combination of static code analysis (which can detect data races and uninitialized variable use), custom tests (like bot matches) that simulate player behavior, and data collected during pre-release user testing.

DiagramEnjoymentEngagementRevenueDevelopment CostHosting CostRuntime Quality
# In a multiplayer game, runtime quality interacts with the other important success factors

Because of their overall complexity, developing a successful multiplayer game usually requires a substantial team of experienced developers who specialize and work on different parts of the game system. If you tell a junior developer or AI agent to create features that “increase per-player revenue” they are likely to damage some other success factor like player enjoyment or cause runtime efficiency issues that dramatically increase hosting costs. While optimizing for revenue can make sense in the short term, single-minded focus on a single success factor will kill the long term viability of any complex system. How should project leaders deal with this?

One method is to formalize a list of hard rules saying what developers (human or AI) can or can’t do, and create an enforcement process to ensure those rules are always followed. This is a good idea for dealing with critical ethical and safety concerns, but as the complexity of a system increases it becomes increasingly difficult to create and enforce hard lists of rules. At some point you are going to have to trust individual developers to use their own best judgment. When leaders refuse to trust developers, it leads to excessive micromanagement that slows down projects and makes everyone real mad.

One big advantage of hiring experienced developers is that leadership does not need to micromanage them to ensure the health of complex systems. Instead, if you tell an experienced game developer to implement a new feature they will naturally make decisions that take into account possible runtime failures and the importance of other project success factors. Experienced programmers generally hate writing inefficient and fragile code, which means that when they add a new feature they will avoid risky decisions that can kill the reliability or enjoyment of the overall game. Unfortunately, management often ignores the value of experienced developers.

Business School Hates Complex Systems

If you’ve worked at any American-style company run by business majors, you have probably seen that MBA types and consultants hate complex systems and will push developers to focus on a single success factor like user growth or quarterly profit. They don’t like it when you try to explain that hacking in a bunch of features will make the whole system fall apart. I think this is one big reason that corporate executives love the idea of replacing developers with AI agents: AI agents do not push back when they are asked to achieve unrealistic or counterproductive goals. They will happily do what you tell them without stopping to worry that it could kill the long term health and success of a company (or its customers).

I’m not an idealistic leftist who thinks that focusing on profit is always bad, but I do believe that American capitalism drove off a cliff when business leaders started focusing exclusively on shareholder value in the 80s. Focusing on a single factor inevitably leads to business decisions that completely disregard other important factors like worker and customer quality of life. As I talk about in AI Is Not the End of the World, many of the dangers of increased AI use are directly related to the false belief among business leaders that companies should always be simple systems devoted exclusively to increasing shareholder value.

A cartoon with the caption: Yes, the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders.
# Cartoon from The New Yorker

In reality, most companies are responsible for building complex systems with unclear goals, multiple factors that need to be balanced for long term success, and a significant risk of runtime failures. In turn, those systems are embedded inside massively complex human and environmental systems. As an experienced programmer, I believe that tasks to sustain the long term health of company-critical complex systems are usually more important than hacking in features to help meet quarterly revenue targets. If business executives force their workers to ignore quality concerns and their obligations to customers, eventually those workers and customers will leave and go somewhere else that does respect them.

Actually Improving Quality

Oh right, I’m supposed to be writing about runtime failures in software systems. Ignoring AI, runtime quality has often been neglected in software companies because it is hard to expose and advocate for at the business and management level. We’ve never found great metrics for measuring the impact of senior programmers (lines of code and bugs fixed are both awful) so it is hard to push back against management demands. If an executive or manager does not understand the importance of runtime efficiency, they are not going to respect it when making HR decisions to promote experienced developers or replace them with AI.

Reading up to this point, you might think that managers should always listen to the advice of experienced software developers and prioritize fixing all quality concerns they bring up. No, this is an absolutely terrible idea. Many senior developers are great at implementation but have a poor understanding of how their code is actually used in practice. If you let those developers fix everything that bothers them, they will spend years reimplementing systems and pointlessly refactoring code that doesn’t improve the actual software system. As complex systems evolve over time, they will inevitably develop maintainability and consistency issues which make programmers mad. Because it is easier to identify problems in current systems than in theoretically perfect future systems, rewriting an existing system often lowers runtime quality.

This tendency is part of why “Software Engineering” can cause more harm than good. If you’ve been around the software industry for a few decades, you’ve been exposed to some of the religious wars over paradigms like Object-Oriented Programming and Functional Programming. These debates rely on abstract arguments and theoretical problems as opposed to learned experience working in real codebases. The point of all software systems is to actually do something, and proponents of software engineering often downplay the importance of practical concerns. They are really good at making charts that become useless when anything changes though.

DiagramSoftware Engineering Class+variablestring+method(a uint64)(x, y int)Pointless UML Diagram+variablestring+method(a uint64)(x, y int) only taught how to make this
# This is a good excuse to make fun of UML

There is one group of technical workers that are explicitly taught to care about the health of complex systems in real world environments: professional Engineers. Most programmers don’t have engineering degrees or belong to professional societies with ethics and certification requirements, but maybe we should. I don’t think we want to force game developers to meet formal quality requirements, but if “AI infrastructure” is as important to the future as everyone claims, we should take the process of developing AI agents at least as seriously as building a bridge. The history of technology is full of cases where unfettered development caused horrible mistakes which created a need for engineering discipline and government regulation, and we’re right around that stage in the development of AI.

We Can Fix Things

I don’t want this to be a rant about how AI is destroying the software industry, so I should probably suggest how to actually fix some of these issues going forward. This part is inherently speculative and incomplete, and I’m sure I am missing something obvious that you can complain about in the comments wherever you found this (I am definitely not going to enable comments on this site).

As I mentioned above, I think that the runtime quality of software (and the general health of the industry) would benefit from bringing in more of the concepts and practices used in professional engineering fields. Quality factors like reliability and performance need to be treated as critical to overall success, and the best practices for maintaining them should be treated seriously. Professional engineers are taught that the long term success of a building project is more important than meeting project deadlines, which encourages them to push back when management tries to cut corners and bypass critical safeguards.

Going forward, large-scale software development will be a collaborative process between human programmers and AI agents, and that needs to be acknowledged when building systems. One positive side effect I’ve noticed with increased AI usage is an improvement in the quality and scope of plain-text documentation. Because of the limited task-specific memory available to AI agents, documenting how a system actually works and why it was designed that way is even more important than it was before. Markdown files stored in a code repository readable by both humans and agents are the perfect place to document concerns about runtime quality. In theory, information about meeting non-functional requirements can be usefully shared across many projects, which is part of why I wrote this in the first place.

Of course there are AI-specific methods for improving runtime quality. To get around the task-specific learning and memory issues it should be possible to train agents in specific problem domains (which is an active area of development). Custom-trained agents are not really viable for smaller organizations, so most of us will need to use off-the-shelf agents for the time being. To best do this, the developers I talked to while researching this all recommended using a workflow with 3 types of agents, following this basic outline:

  1. You can chat with a planning agent (like Fable) to collaboratively build the initial architecture of a system and work out the functional and runtime quality requirements for a specific project. These concerns need to be documented somewhere accessible and regularly updated when the project changes over time.
  2. You can then use implementation agents (like Sonnet) to write and maintain the lower level code using that documentation. Humans are apparently optional for most of this part.
  3. Before that code is committed, you can run it through an AI-driven code review process to identify technical issues and concerns about runtime quality. This should be integrated with a suite of automated tests specific to your project. Once that code passes automated review, humans need to review check-ins to keep on top of what is going on.

If you are a programmer or tech worker who has been tasked with building a new feature, you have the ability to think about runtime quality (and larger ethical issues) and make decisions that can help your development team, your company, and humanity overall. It often feels like we have no control over our jobs and need to do the exact stupid things our boss tells us to do, but most of the time we have significant influence over the final product. We can’t stop the tidal wave of vibe-coded slop on our own, but we can make specific development decisions that maintain the health of complex systems and provide real value to the humans who actually use our software.

--VeryVerbose

Thanks to James, Kenton, Jason, Aaron, and others I talked to about AI coding. Everything in this article (including the errors) was written by a human.

If you want to read more words about this subject, here are some links that inspired this article:

[INFO] loading older machine posts…

$ cat archive.log