09:41:02.118 [DEBUG] $ cd .. → back to homepage

09:41:02.121 [INFO] loading module:games…

09:41:02.152 [INFO] found 2 posts matching module="games"

The Human Brain Is a Game Engine

module=", "author=Ben Zeiglerpublished=2024-09-17read_time=11m
A brain with a circuit board pattern
# Image by Kohji Asakawa from Pixabay

I have spent the last few years (while getting a Masters in Psychology and working as a game programmer) working with two very complicated software systems: the Unreal Engine and the Human Brain. While there are many obvious differences, both systems are used to generate interactive simulations of reality in real time. As game programmers have improved the complexity of their reality simulations over the last few decades, they have come up with some solutions that are surprisingly similar to what evolution came up with for the human brain. I thought I would use my experience in both fields to survey what I see as the structural similarities and link to some relevant scientific research if you want to learn more.

The current version of the Unreal Engine (UE5) is a very large software system that has evolved over 25 years through the efforts of thousands of different developers, so it is basically impossible for one person to fully understand it. By “understand” I mean the ability to create a detailed mental model of how every module works and how they all interact with each other. I’ve worked with UE for 15 years now and I understand how the core engine, gameplay, and networking modules work but there are hundreds of other modules that I do not understand. The human brain is like this but massively more complex. Although scientists have developed a good understanding of some parts of the brain, the entire system is far too complex for us to fully understand at a detailed level.

Like complex computer software, the human brain can be analyzed as a collection of different Cognitive Modules that handle different kinds of tasks. Some of these modules are implemented in specific low-level hardware that we share with many other animals. Other modules like visual processing are implemented in specialized areas of the cerebral cortex that evolved over time to handle complex mathematics. The highest level modules seem to be implemented in general areas like the frontal lobe where the exact location and configuration is different for each individual (which makes them difficult to analyze with brain scans). Neuroscientists now have a good understanding of how the simpler brain modules work at a detailed biological level, but those approaches do not scale up to understanding how the high level modules work because they are too complex and variable. Fields like Cognitive and Evolutionary Psychology are better for explaining how the brain as a whole functions.

After working on thousands of different bugs and tasks, I know that many parts of the Unreal Engine are used in ways that are very different from how they were originally designed to work. A lot of the code that confuses developers now made perfect sense when it was first written 10 years ago to solve some specific problem. The code then evolved over time as it was used by other programmers as a foundation for similar tasks or repurposed to solve very different problems. Because all of these systems are interrelated, it’s very difficult (and often harmful) to completely redesign a system from scratch. The human brain is like this at a much grander scale, where our brain hardware and software evolved over millions of years in situations that are very different from modern reality. But via cultural evolution, education, and science we have learned how to reuse that software to create new and exciting things like video games.

Real Time Reality

The Unreal Engine is a real time software system, which means that it has to work within specific time limits. Real Time (RT) software can be roughly split into two categories, Hard RT and Soft RT. Hard RT systems are used for applications like car engines where reliability is critical and the problems are complicated but well defined. This software is designed to be fully documented, audited, and understood, so this type of programming is often handled by Computer or Electrical Engineers. On the other hand Soft RT systems are designed to handle situations that are too complicated and unpredictable to perfectly solve in a fixed amount of time, so they need to use optimizations and heuristics to find good-enough solutions in constantly changing conditions. Modern game engines and the human brain are both Soft RT Systems, and this has interesting implications for human psychology.

Developers may not realize it, but many video game systems can be described using the concepts of Control Theory that were originally developed for mechanical feedback-driven systems. Many parts of the human brain can also be understood using control theory, especially the lower level neural systems related to perception and movement. Historically, most attempts to create Artificial Intelligence were not based on control theory and real-time feedback systems, which is one of the reasons that logic-based AI has gone through a cycle of bubbles and AI winters for the last 50 years. This has started to change with the recent advances in generative AI, but systems like ChatGPT do not need to process real time feedback. But because video game engines are designed to handle real time interactive feedback, game engines work more like the human brain. It is not a coincidence that the hardware used for modern AI is based on the GPU technology originally designed for video games.

Early video games were Hard RT systems because they had to run simple code quickly and consistently to meet the strict timing demands of CRT standards like NTSC. The details of this are fascinating and I enjoy watching channels like DisplacedGamers that explore how this worked on systems like the NES. Some parts of game engines still work with hard time constraints, but games mostly transitioned to become Soft RT systems with the creation of 3D engines (like the original Unreal). Instead of running everything at a fixed frame rate, most games now update their simulation of reality based on how much real time has passed since the last frame. A high frame rate is still important because it enables low input latency and smoother displays, so shortcuts need to be taken to keep up with reality.

Unreliable Optimizations

One of the simplest ways that both game engines and the brain deal with real time constraints is to use models of physics that are efficient in normal situations but can fail in unusual situations. This is why video game objects often behave very strangely when they move too fast or get too far away. Recently, research has shown that the brain models the physics of reality in a surprisingly similar way that is optimized for the requirements of daily life. Also, most famous optical illusions are caused by generally useful perceptual heuristics that fail in specific situations. It is not a coincidence that both the brain and video games tend to generate confusing visual artifacts in weird situations, because those artifacts happen when the small errors of physical simulation interact to create larger problems.

The brain also uses many time-saving heuristics at the cognitive and conceptual level, which create various forms of cognitive bias. For example it is actually extremely difficult to determine if two perceived objects should be treated as “the same thing” so it takes many months after birth for humans to develop an effective sense of object permanence. If we tried to treat each newly perceived object as entirely unique, keeping track of important things in our environment would take an excessive amount of processing power and memory. As we grow older we learn which types of similarities are important for predicting future behavior and which ones are irrelevant, but this process is impossible to do perfectly. Instead, we have to learn a series of heuristics over time that we combine with reasoning to help us understand the behavior of objects and people.

Based on what I’ve learned about Psychology, many of the problems we face in modern society are related to failures of these cognitive heuristics. For example, at a deep level humans are programmed to generally trust our family and distrust outsiders. This is critical for developing the family bonds needed to help children live and learn (and later to keep us safe). But as we get older, it is much harder to know who to trust, and many harmful cases of racism and other discrimination in the modern world are caused by the failures of heuristics that evolved many thousands of years ago.

Time Traveling Software

One way to increase game frame rate is to process multiple things at once on multiple threads (running on different processors). In a modern 3D engine, the code that handles game behavior and logic (gameplay) runs at the same time as the code that creates the images for the player (rendering). Because the rendering code needs to use the position and state information created by the gameplay simulation, the rendering code is often several milliseconds “behind” the gameplay code. This gets more complicated with audio processing and input handling, but the main point is that different parts of the game engine are handling different “points in time” at once. The engine regularly jumps between different simulation time frames so you could say that the software is “time traveling”.

The operations of a computer happen at specific frequencies that allow communication across the entire system, and the human brain works in a similar way. The timings of low-level systems like visual perception are tied to the physics of light and electricity and then information is encoded in signals (like Alpha waves) that are emitted at specific frequencies to communicate with distant parts of the brain. However, the timings for brain computation are more variable than for a computer because they are based on complex neuron processes instead of a fixed crystal oscillator. Instead of one fixed frame rate, different processes in the brain run at very different frequencies and need to somehow synchronize their operations.

Because different brain processes run at very different speeds and it takes significant time for signals to travel across the brain, there is no coherent definition of what the “current time” is in the brain. Instead, our conscious understanding of time is an illusion created by the narrative parts of our brain that try to make sense of what is going on. Psychology research into the illusions of time perception show how the brain merges events from different time frames to create a coherent stream of consciousness. It is fairly easy to manipulate events in experiments to convince people that some behavior was “caused” by something that actually happened much later in reality. Human time perception gets even less accurate over the longer time frames of memory, as untrained humans are awful at estimating the passage of time. For game engines and the human brain, time is always an illusion.

The Brain is a Multiverse

Some games need to simulate multiple possible versions of a game world at the same time. This is commonly used for network prediction in multiplayer games, because the latency of network communication is much higher than the latency of local player interaction. In multiplayer games, each online player lives in a slightly different version of reality based on actions that have not yet been sent over the network. The game engine needs to predict what remote players are doing in order to maintain the illusion of a single shared reality for the local player. When the remote player information is finally received, the game engine then needs to merge information from these multiple realities and resolve any conflicts. There are many ways to do this, but they all have some edge cases where the final outcome does not match what one player experienced. The design and implementation of multiplayer games requires a good understanding of human psychology, because it is hard to merge these conflicting realities in a way that feels right.

The human brain also has to simulate and merge multiple conflicting versions of reality. One way to see this in action is when you have a conversation with someone else. There is no way to directly communicate thoughts, so our brain needs to simulate the thought processes of the other person in real time to understand their words and predict their behavior. There is some evidence that this evolutionary need to simulate the thoughts of others is what created human consciousness in the first place. The process of simulating what another person is thinking and integrating it with our local perceptions is very similar to the process of network prediction in multiplayer games. Not everyone is great at this kind of “mind reading”, but it’s frankly amazing that we’re able to do this at all.

On top of that, most of us can have a conversation at the same time that we do something else like drive a car. This is because the sensorimotor part of the brain can simulate the “car” version of reality while the social part of the brain is simulating the “conversation” reality. There is a limit to this capability, and I am a worse driver when there are other people in the car. When I can focus on driving, I drive defensively by simulating what would happen if the other drivers did something stupid. However, when I am talking to someone else the parts of my brain that try to predict what other people will do runs out of bandwidth and I am more likely to make a mistake. We need to simulate multiple versions of reality when we make complex decisions or evaluate the truth of statements, but we can’t do this while we are dreaming or drunk. The inability to simulate and compare multiple realities is part of why current generative AI systems (which only simulate a single reality) have a “dreamlike” quality and often lie to users.

Scientists can debate if we are physically living in a computer simulation, but it is clear that our conscious minds live inside the simulations of reality created by the software of the human brain. The conscious mind cannot directly interact with reality, so all of our experiences are based on electrical signals organized into a complex system of interactive real-time feedback loops. Not only that, but we live inside a multiverse because the brain is often simulating and integrating multiple versions of reality at the same time. In other words, we all live in a massively multiplayer online world powered by the game engine that is the human brain.

[INFO] loading older games posts…

$ cat archive.log