Unraveling Complexity: An Anecdotal Dive into Controllability and Observability of Distributed Multivariate Systems
The entire universe is a distributed system. But in software engineering we are interested in a minute subset of distributed systems and how to observe and control them to make our lives easier and often to make money and drive economy.
Observability, in this context, is about gathering the necessary data to understand the system's state and performance. This might involve tracking metrics like request rates, errors, and latency, or using logging and tracing to understand the history of system events and interactions. The facets of observability are often categorised as the 3 pillars - Logging , Tracing and Metrics
"The internal variables of a complex system are rarely in-dependent of each other, as the interactions between the system’s components induce systematic interdependencies between them"
- (Liu, Slotine & Albert-László Barabási 2013)
Controllability on the other hand, is about making adjustments to the system to influence its behaviour. This could be scaling resources up or down in response to demand, deploying new code to add features or fix bugs, or tuning system parameters to optimise performance. Observability and Controllability often go hand in hand because in order to steer a system efficiently towards a logical goal, we need to observe it.
Separating the Evergreen from the Specifics
In this article we are focusing more on general, evergreen principles, to steer ourselves away from context bound specific concepts that become obsolete at the speed of innovation and trends. At the time of writing this, a loosely-coupled, vendor neutral implementation of observable systems is achieved by using Open Telemetry. But the essence of the subject matter remains evergreen.
By achieving high levels of observability and controllability, we can build more reliable, efficient, and user-friendly software systems. And as you noted, this isn't just about making our lives easier—it's also a key driver of economic value. As our world becomes increasingly digital, the ability to build, manage, and optimise distributed systems is more important than ever for businesses across all industries.
Observability and Controllability are not only anecdotally duals of each other but they have a duality ingrained in mathematics.
Unraveling a magician's secret
Imagine you're at a magic show. The magician places a coin in her palm, closes her hand, and when she opens it again, the coin has vanished! Now, you didn't see where the coin went or how the trick was done, but you know something happened - the coin is gone. The trick isn't observable to you because the magician has hidden the important details.
But what if you could see every detail? What if you had x-ray vision and could see the coin up the magician's sleeve? Suddenly, the system - in this case, the magic trick - becomes observable. You can infer what happened from what you can see.
In essence, observability is like having x-ray vision for systems. Whether it's a magic trick, a car, or even something as complex as a city's traffic system, observability allows you to infer the entire state of a system based on what you can measure or see.
And here's the mind-blowing part: In systems like weather prediction or traffic control, observability allows us to "see" the state of things we can't physically observe. By measuring certain variables, like temperature, wind speed, or traffic flow, we can infer a larger picture - whether it's predicting a storm or anticipating a traffic jam. In a way, it's our superpower for understanding and navigating the complex world around us.
Concluding with an Anecdote
There was once a master sculptor named Magnus who had an apprentice, Cora. Magnus was an expert at moulding stone into exquisite figures, while Cora, young and eager, was just beginning to understand the art.
One day, Cora asked, "Master, how is it that you can take an unformed stone and bring such beauty out of it?"
Magnus smiled and replied, "Well, my dear, it's a delicate dance between controllability and observability. In the same way that a captain must understand the sea and steer his ship with confidence, so must we understand the stone and guide our hands accordingly."
Over time, Cora became adept at sculpting, observing the stone and controlling her tools, but she found herself dissatisfied. Despite her growing skill, her sculptures did not seem as lifelike as her master's. Seeing her struggle, Magnus shared a deeper wisdom.
"Cora," he began, "controllability and observability are duals of each other, but the true mastery lies in understanding their balance. Observability is to truly see, to understand, and to appreciate the world and its complexity. But controllability is the ability to navigate within this complex world, influencing outcomes, shaping reality according to our understanding. To bring life into stone, you need to observe, to see its essence, but also control your hands to reveal it."
As she grew older, Cora realised the wisdom behind these words was not confined to the studio. It echoed throughout her life. She noticed that understanding people—observing their needs, fears, joys—made her relationships more meaningful. Controlling her reactions to life's ebbs and flows resulted in a more peaceful existence.
The dance between controllability and observability became Cora's guidepost, reminding her of the delicate balance between accepting the world as it is and striving to make it better. For she understood that life, like her sculptures, is shaped in the nexus of observing and controlling. Each movement of understanding, every subtle shift in action, brought forth the beauty she sought to create.
In the same way we should strive to bring efficiency to the systems we build by making them more controllable with a robust foundation of observability.
Happy Engineering 🍋
PS - the entire universe is a distributed system that recurses in down to Planck length and recurses up to the edge of the observable universe.
