I am Uncluttered(Yellow)

Preview Excerpt

3. Worse is better, but it’s also how we made this mess #

What really helped turn the software industry into The Software Industry™ trash fire it is today was the fundamental contradiction at the heart of pretty much every major decision in software over the past fifty years:

1. Worse really is better.
2. But it’s also how we got into this mess where everything is awful and broken.

If you aren’t familiar with the term worse is better – good god where have you been? Tell me now, so I can move there, lock the door, and crawl under the bed, you lucky stiff. I’m jealous.

At least I can make you suffer by forcing you to understand just how broken the rest of us are. The term “worse is better” comes from the early days of the software industry. Not quite the very beginning, but the moment it was coined definitely marked a beginning in software development: the beginning of modern programming practices.

Most fields, especially the ones that have some sort of creative dimension to them – of which I consider software development to be one – tend to have two opposing theoretical frameworks that are in constant conflict.

They don’t really. The emphasis on dichotomies in fields and academia is all bullshit to get a bunch of undergraduates to argue about shit. It’s the arguing that’s the skill they want them to develop and dichotomies are good at getting people to argue. People have a strong tendency to create factions around arbitrary totems and then proceed to murder (metaphorically or actually) the “other” and dichotomies have the advantage of collapsing the plurality of society into a single, focused “other”.

When factional debate is what you want, it’s an extraordinarily effective tool. When it isn’t – well, you’re gonna get it in the neck anyway, because people love to form factions and then fight over them. Over time, those factions often get polarised further and further apart and their respective roles and arguments take on a ritualistic dimension. The true “heterodox” thinker – the genuine alternative to whichever of the two is the “orthodox” faction – isn’t the Brainy McBrains that picks the less popular side of the dichotomy. The two factions exists mostly to establish each others boundaries and outlines, so the true “heterodox” is the viewpoint that rejects the duality of dichotomies altogether. But genuine heterodoxy is also a reliable way of getting thrown out of the room and the hallmark of the boring asshole who stands alone in parties, so… never mind. Go ahead, pick a side, and club the other.

In literature and drama, this dynamic often gets expressed as romanticism versus classicism.

Romanticism would be all about the larger context of the work, the emotions, the people, how it plays out, where the author of a romantic work matters almost as much as the work itself.

Classicism focuses on the meaning that’s found in the work itself, in its very structure and composition, without referring to the larger context. The author and the environment does not matter. It’s the work itself that matters. The text is what matters.

You can see why these two approaches are in opposition. One claims that the work primarily gets its meaning from context, while the other claims that the work primarily creates meaning internally through its structure and form, which is why adherents to these two movements tend to always be in a spiral of conflict and evolution, where they grow and evolve their ideas in constant argument with the adversary, who is also always evolving their ideas, eternally arguing and never coming together.

(Until their funding is cut and they all get jobs flipping burgers instead. Hard to seriously debate the intentional fallacy if you smell of lard and stale beef. Don’t mock. We’ll join them soon enough.)

Actual practice in literature, writing, and drama tends to be a mixture of the both approaches.

The author and context are both always going to matter to some degree. We exist in a society, and we can’t rid ourselves from it, though Thatcher certainly started the trend in the eighties of politicians pretending you could.

Meaning is also in the work itself.

Once we have a base level of understanding of language and text and how texts work, each text will carry a meaning of its own.

The structure of the text will tell you something on its own. You might not see all of it, but it’s still there.

Reality is always somewhere in between. Two things can be true at once. Both the inside and the outside matters.

Software development similarly has two opposing ideas.

You have the ideological kin to the literary classicists who focus on the correctness of the work and writing software that using approaches that are structurally correct.

Back in the day, this wing of software development bet on LISP – a hybrid of metastatic bracket kudzu and the self-important nerds’ love of maths.

The LISPers even had bespoke machines for running LISP code.

This faction of programming tended to be obsessed with proven validation of correctness of one kind or another. The idea was that you should be able to build software incrementally while its internal structures made sure all of it worked properly and was correct and stable. Whether it worked for actual end-users was often an afterthought. At least that’s one idea from the old LISPers modern programming has embraced with enthusiasm.

(I’m caricaturing the position a bit, but honestly not as much as you’d think.)

The idea is that software should be able to demonstrate to some degree its own correctness. An academic would cite an incompleteness theorem here, but I’m not an academic and you’re already a little bit bored (I can tell), so I’ll just leave you with a flat statement that fully proving the correctness of a piece of software with other software is both impossible in theory and in practice.

The question is how close you can get.

The idea that drove the code classicists was that the only way to make defect-free software was to make it impossible for it to have defects through language and platform design. Correctness came from internal structure and maths.

This is a more feasible goal than proving said correctness for reasons I’m still not going to go into because they are, honestly, still quite boring. (The I am Curious… movies tried to make up for their tediousness with sex and nudity. I don’t think attempting the same here would be to my advantage.)

The ideological adversary of the code classicists – the code romantics, if you will – would be the Unix, C and BSD side of the industry and the entire cavalcade of garbage that has followed, which basically said, what if we just get things done and build things out of simple concepts to begin with, and then try to improve it from there?

It’s going to have bugs, but we’re going to be able to fix them all afterwards, right?

Right?

That was, essentially, how the LISP crowd caricatured the success of Unix. They called it the worse is better approach.

The code romantic’s idea with worse is better was that by layering up relatively crude languages and systems, you would end with more correct software – as in more software that’s roughly correct – than with the alternative approach. Improved productivity gave you more time to work on the correctness of your code, all in less time than with the approach used by the hoity-toity “maths matters!” crowd. The classicist approach, conversely, gave you software that was arguably more correct, if agnostic in terms of User Experience, but with less productivity and slower overall because the work went at a much lower pace.

The worse is better approach has won out over the past 40 years. We just hammer things together and try and fix things afterwards.

That’s pretty much the only way for us to be realistically able to ship software at anything resembling the scale required by investors and employers.

If we wait until we can prove mathematically, with reasonable confidence, that a piece of software has no meaningful defects, we’re going to be waiting forever.

It’ll never ship, which obviously isn’t practical – you’ve got to at least pretend to ship if you are to con stock options out of somebody gullible – so instead we’ve decided to build an entire industry on the other thing. Not bothering to check at all. That thing.

We’ve built all of software on the concept that we’ll muddle through to begin with, and then try and fix things after the fact, at, y’know, that point in the future we all know will eventually arrive when you have tons of spare time to fix things?

These errors, mistakes, and flaws that litter modern software are baked into its very concept, and now we’ve built layers and layers and layers on top of that shoddy foundation. So, now we have rocket ships whose controls and steering is a React web app all wrapped up in an Electron container.

We have this tension where the only way to realistically ship anything that works (sort of) is to do it imperfectly, but those imperfections collect and build up over time which then begins to hinder our ability to ship anything ever again.

Now most, if not every, project in the software industry is dealing with decades of accumulated platform, language, process, or hardware defects.

There are security flaws in everything. There are user interface flaws in everything. “Ease of use” is an extinct concept.

Flaws accumulate. Things have become too complex and error-prone.

We can try to minimize the imperfections we ship with unit testing.

We can use type checking when that’s appropriate. That sometimes helps.

But the compromise we are struggling to overcome is baked into our industry – it’s the very ground we build on. Worse is better.

We ship imperfections because, on the whole, it’s better. At least for certain definitions of “better”.

The improvements we ship, on the whole, have a bigger impact than the defects we’re shipping with them.

But the defects are still there, and they’re still accumulating, and we’ve coasted for the past decades on the fact that we believe that the improvements are collecting faster than the defects.

Occasionally, for some software systems, you actually get the opportunity to stop and pause and try and fix some of the defects.

But you can never get rid of them all because “let defects ship” is baked into the system.

We’re working in a compromised industry.

There’s nothing we can do to truly fix it.

Not in this lifetime.

(I’m sure your fucking chatbot will talk you past all of the landmines.)

We live in a compromised world and there’s nothing we can realistically do, as individual developers, to fix the underlying issues. Especially now that we keep bouncing from job to job, just to get laid off next quarter while all the surplus funding gets sucked up by an avalanche of “magical” Elizas.

Except…

Taking my own advice above on rejecting dichotomies, we could treat the two as a sliding scale or spectrum. “Worse is better” approaches, at the behest of the stock market and investors, piling on dependencies, frameworks, and building complex monolithic software as a never-ending march of new features to drive the software industry’s stratospheric growth is the extreme end of that spectrum.

We can’t do much to salvage an industry that’s entirely geared at building upwards without paying any mind to foundations, but we can slide back on the scale for the work we control.

• We can avoid collecting JavaScript dependencies from an ecosystem that’s almost bound to collapse.
• We can focus mostly on the parts of the web stack that are less complex, more robust, and more reliable, like CSS, HTML, and HTTP.
• We can dial back on how much front-end code we use in the web services we make and the web apps we put together.
• We can shift our testing practices towards driving development instead of patching development after the fact.
• We can return to practices that are known to lead to software that’s easier to maintain in the long run, such as Separation of Concerns.

What we can’t do is absolve our industry of its “original sin” – the You-Only-Live-Once moment where corporate software platforms embraced “worse is better” for everything. We’d have to invent an alternate timeline. Where we made different choices 50 years ago and decided to prioritize correctness. Where the software we’re using is much less capable than the software we have today, but has fewer flaws, is easier to use, and doesn’t have the near-singularity, black-hole-singularity levels of defects that our current system has.

But that world doesn’t exist.

We have to live in this one, which means we have to find marginal improvements and incrementally fix what’s broken.