This is the fourth part of my tech debt series. It might even be the last part, but I sure have a lot to say about it so don’t get your hopes up. You can read parts one, two, and three as well. To sum up the series so far:
“The rework required by past choices” is an inadequate definition of tech debt.
It is inadequate because it is overly focused on litigating the past decisions of others and neglects the (much greater) amount of rework required by changes in the world that don’t stem from previous choices made by system designers.
Part three was about how software systems fall apart just like everything else. Financial accounting has a way to track the loss of value from stuff falling apart called depreciation.
When your business plunks down some money to buy a new piece of equipment, your accounting department does not immediately classify it as an expense (a decrease in your assets). It’s true you have less in your bank account, but now you have that shiny new piece of awesome. You transformed one asset (cold, hard cash) into another (a cool doomahickey you are going to run part of your business with).
But your accounting team is well aware that things fall apart and so, every year, they will depreciate the doomahickey by reducing the value of it on the books and adding an expense of the same amount it was reduced. After a somewhat arbitrarily chosen period, the doomahickey’s value will be zero, at least as far as your accounting department is concerned.
Which is just fine – everything does fall apart so why shouldn’t the rules of accounting reflect that? Depreciation is just the accounting profession’s nod to the Second Law of Thermodynamics, which comes for us all, even doomahickeys.
So what, exactly, is the problem?
If we can account for things falling apart, then is tech debt an illusion?
If you buy your kid a new car, it will lose value over time, starting as soon as it’s driven off the lot. But you don’t owe your kid a new car just because you bought them one years ago and it is now worth a lot less. You both knew that was going to happen when you bought it. Such is life and cars, kiddo.
So is tech debt just a verbal tic that whiny engineers trot out for stuff they don’t like? I do not think so. Recall from Part three that complex systems fall apart in three ways:
The solution system decays.
The problem system decays.
The coherence between the solution and the problem decays.
The rules of depreciation focus on the first kind of decay (since those rules are about the value of the solution system as an asset that you own), and that makes sense because those rules were designed for physical equipment, which tends to wear out before the world it is used in changes enough to matter.
But software decay is dominated by changes in the world and organizations struggle to “see” the decay that is happening to solutions which appear to more or less work the same as they always have1.
Consider the Southwest Airlines holiday travel meltdown. This extended and widespread disruption in Southwest’s operations was ultimately deemed to be the result of long-obsolete software, a fact clearly known by some at the airline, but possibly not known by leadership. At least I suspect leadership did not realize just how badly out of date the system was as it led to an enormous loss of money and customer goodwill on top of the money they have to spend to upgrade, and it was a priori likely to do so if you knew the sorry state of the software.
Southwest’s software was so out of date one might ponder whether its value was literally negative. I am guessing their financial statements did not have a liability for “critical software that has decayed to the point where our entire operations are one bad weather day away from an embarrassing and expensive total system failure".
Some people who almost certainly did not know that was the state of affairs were the travelers who bought Southwest Airlines tickets right before the meltdown. And Southwest’s shareholders were probably a bit surprised too.
Tech Debt as a missing accounting entry
Southwest pledged $1.3 billion in software maintenance and upgrades in the wake of that disaster.
When you pay for a service, there is a basic presumption that it is fit for purpose, that it actually provides an acceptably working version of the thing it is nominally offering. As it turned out, Southwest’s service was not fit for purpose and it’s costing them $1.3b to remedy that. I think it is entirely fair to say that, prior to the meltdown, there was a $1.3b hole in Southwest’s books that the world only became generally aware of after the fact.
Let’s suppose that Southwest had let the maintenance of its planes languish to the point where the paint was falling off, the interiors were dirty and frayed, etc. This would also represent a loss of value, but everyone would already have known that – travelers could just see it with their own eyes and would quickly decide to stop flying Southwest!
The key point of Part 3 is that while software systems fall apart like everything else, the way they fall apart is distinct from the way that physical equipment wears out. Software systems mostly decay by gradually losing coherence with the world they are meant to inhabit, and that makes it much harder to notice that it is happening.
This brings us back to my original claim in Part 1, where I unpacked the debt metaphor and concluded there had to be two parties involved in tech debt and one of them was probably the customer. Now I am in a position to explain what I think tech debt is and how it comes into being:
A software solution is created to solve a problem, creating something of value where the value is the worth of solving that problem.
Over time, the coherence of the solution / problem system decays, thus making the current solution less valuable than when it was created. The software solution may even have become a net liability because the coherence has decayed so much.
Because of the way that decay happens, the total extent of the loss of value is not widely known, particularly to customers and shareholders, but also within the organization that created the solution.
The technical debt in the system is the unacknowledged, unaccounted for difference in the actual current value of the solution and its assumed value.
The assumed value is what you would pay for a fit-for-purpose solution to the current problem. And the assumption arises because (most) people have failed to notice how far the current solution and current problem have drifted apart and thus treat the current solution as equally fit for purpose as when it was created.
Southwest’s systems were probably pretty good when Southwest first got going. At least, they could cope with the scale of the problem that early Southwest had. But as Southwest’s routes and hubs got big and complex, the software did not keep pace.
That happened gradually and, thus, was easy to miss if you weren’t working with those systems as part of Southwest’s operations teams. The mismatch between the solution system and the problem system grew to the point where a catastrophic breakdown was just a matter of time.
Sometimes whiny engineers have a point
Although I don’t agree with the current public definition of technical debt, it should be clear I think technical debt is very, very real. When engineers talk about tech debt, and worry about tech debt, I think they have (often correctly) intuited that the organization has failed to realize (or at least publicly acknowledge) how much value its technology solutions have lost and how much liability has accrued because of that failure.
Tech debt is an engineer’s way of saying “hey, we have a real problem building up here and you don’t seem to realize that!” to leadership. The essence of the message, what technical debt is, and my solution to the way we should understand the debt metaphor is this:
And the company owes that value because its customers and shareholders reasonably assume it is there. Maybe it used to be. But it is not there anymore. Sad!
Introducing The Crumble
The economist J.K. Galbraith defined “the bezzle” as the total quantity of undiscovered embezzlement in the economy. During periods with high levels of fraud, there is a net increase in psychic wealth as the fraudsters know they got what they wanted while the marks believe they got something too.
Charlie Munger, late of Berkshire Hathaway, expanded the bezzle2 to include any difference between the actual and reported values of all the assets in an economy, whether or not that difference was the result of fraud. His point being that the existence of that difference was relevant regardless of its cause because of its implications for the future (because one day the difference would be noticed, with consequences to follow).
In that spirit, let’s define the total amount of technical debt in the world as the crumble3. The crumble is that portion of the bezzle (Munger’s expanded version) that is the difference between the assumed value of all extant technical solutions and their actual value, a difference that has arisen because those systems have crumbled away over time as the solutions and their problems have drifted further and further apart while people failed to notice this happening.
Mostly failed to notice, that is. That “tech debt” is so ubiquitous a metaphor tells me that, like fraud, there are almost always people (aside from the fraudster) who know that something isn’t quite right, but they struggle to warn others before it’s too late. Whether “tech debt” as a term of art is actually helpful in that regard is a topic for another post.
Given that Southwest is just one example, I suspect the value of the crumble may be quite large…
The End, or at least I’m stopping for a while
Thanks for reading this far. I think I’ve managed to write a lot of what I wanted to say about technical debt. It’s possible I’ve spent too much time thinking about it. Because I’ll probably have more to say in the future!
If you have found value in reading this, even if as a useful counterpoint to your own, different opinions about tech debt, I’d appreciate hearing from you. In the meantime, here’s a very good boy to whom I owe some technical treat debt:
Peace.
I know that software a company creates can be capitalized. But having seen that process in action at multiple companies, I think it largely fails to capture both the value added and the value lost, mostly because it is a crude translation of the rules for physical asset depreciation to a situation where they are not a good fit.
While that article has a nice explanation of the original and subsequent definitions of the bezzle, I first heard about it in Dan Davies’s Lying For Money, which, if you haven’t read, you should go read right now.
Thank you, Gemini AI, for the excellent suggestion. And if Gemini grabbed that out of society’s hive mind because you put it there, well then I thank you for the contribution too.