The Fallacies of Distributed Computing

Essentially everyone, when they first build a distributed application, makes the following eight assumptions. All prove to be false in the long run and all cause big trouble and painful learning experiences.

The canonical eight fallacies, as posted on James Gosling’s Website [1], which he attributes to Peter Deutsch:

  1. The network is reliable
  2. Latency is zero
  3. Bandwidth is infinite
  4. The network is secure
  5. Topology doesn’t change
  6. There is one administrator
  7. Transport cost is zero
  8. The network is homogeneous

Deutsch, himself [2] attributes the first four to various Sun luminaries as Bill Joy and Tom (or Dick or Bob) Lyon, but none of these authors has ever claimed attribution or spoken of their involvement. We can only really say that these first four, known as “The Fallacies of Networked Computing” emerged from the lore of Sun Mirosystems over the course of ten years or so, in the 1980s as Sun broke early ground in this field.

The story Peter Deutsch tells, is that these first four were just “floating around” when he joined Sun in the early 90s. Deutsch was tasked with evaluating the challenges posed by the burgeoning mobile networking segment, and so he took the first four, added his own 5-7 and presented them to then CEO Scott McNeally, who it is reported “binned them”.

James Gosling (widely recognised as the creator of Java) added fallacy number 8, and posted it on his website. They have been there from as far back as the internet archive goes [2004?], up to 2023 when he posted details of his retirement.

Relevance

It’s hard not to look at these nowadays and feel a pang of impatience. The presentation as fallacies themselves comes across as a little condescending. The language hails from a time when the Internet was a much smaller place, where technical discourse was dominated by people with expensive graduate school educations and in all liklihood one or more PhDs. It was a time when knowledge was passed between people in riddles, aping the academic sport of their tutelage. This might help explain how understanding even the problem the fallacies are trying to describe, is seemingly left in part as an exercise for the reader.

These fallacies describe what Kant would call “paralogia”, where our axioms for a reality we understand so well become delusions when faced with phenomena we don’t. Where our pre-existing knowledge crowds out our intuitions about something. We might call these “hallucinations” nowadays.

Yet they were of a different kind, The names that stilled your childish play, They have gone about the world like wind, But little time had they to pray

It is also important to remember, and hard to overstate, the kind of genius that was endemic to and concentrated in Sun Microsystems at that time. It cannot be stressed enough just how far ahead of the networked computing game they were. When the Internet blew up in the early 90s they had a good 10 year march on their competitors in that space. They had a technical vision, and the knowhow with which to implement to a degree that is rarely seen outside of strategic public institutions. They were a business that had an almost quasi-academic status.

The past is a foreign country; they do things differently there

A lot has happened since the early nineties though. Lots and lots of things have happened in the field of distributed computing and networking:

To the degree that issues around distributed computing actually occupy the public mind, the fallacies seem passe. Nobody goes onto their smartphone in 2026 and expects a seemless computing experience. We have all internalised many of these lessons at this stage, and most software engineering students will blow past these issues at undergraduate level.

Context

By the early 90s the business world, academia and the military had become acustomed to commonplace cheap, reliable, flexible and robust Local Area Networks (LAN). Ethernet is the best known of this gemeration of networking technologies, and it’s still in widespread use and the direct ancestor of Wi-Fi. Ethernet provided reliability, low latency, high bandwidth (while relying on physical security) user surface, for a very low infrastructure cost.

Most people at that time, according to Gage [3], that were flooding the industry out of universities, the military and the corporate world were familiar with modern high speed reliable networking. At the local level it was a solved problem. Reliable point-to-point links were also, arguably a solved problem.

But these things being solved problems didn’t mean that everybody (then as now) knew how to solve them, or understood the tradeoffs that made them work. Bill Joy, one of Sun’s founding luminaries implemented TCP for BSD, as well as coming up with “vi” and NFS (Networked File System). TCP is a protocol we are all still very familiar with to this day that specifically trades off latency and a little bit of bandwidth for reliability. BSD was Berkeley University’s distribution of Unix that became the foundation of most networked operating systems today, mostly because it was the product of geniuses but in no small part I’d say because it was free. Bill Joy also brought internet to Aspen as a dramatic proof of concept that presumably also drummed up a nice wedge of investor cash.

So here we had Sun, with demonstrated expertise developing seemless applications for the local network, also demonstrating competency in the wide area network.

Networked Computing

"The Network is the Computer"

There is an undeniable resonance to Sun’s slogan “The Network is the Computer”. The first four fallacies, know as “The Fallacies of Networked Computing” emerged from the engineering culture at Sun, during those first 10 years or so, as they apparently tried to sell a network that behaved like a computer. These first four fallacies propose that reliability, latency, bandwidth and security are four factors that become more prominent as you scale out from a single node to a distributed system.

Reliability is arguably the essential feature of a computer. A lot of coordination and integrated thought goes into making all those tightly coupled, interlinked components work successfully together. If a computer did not function reliably, it would not be useful. It would not be a thing that people would buy.

Cogito ergo sum

The first fallacy could even be considered a special case of the Cartesian formula “I think therefore I am”. For the computer it might read “I am reliable therefore I compute”. It is axiomatic but beyond the bounds of the single node (Turing?) computer it breaks down.

A big part of the joy of software development is the “safe space” that a computer provides to put together ideas and watch them manifest. Even when we are implementing, we are modelling some reality and there is a comfort in seeing rules activate and assert themselves unconstrained by the inconvenient and unreliable outer world.

As an experienced software engineer it can be helpful to reflect upon that world from time time. That green eden from which we all mostly sprang. That single core, single-threaded, secure, limited bandwidth, fixed latency world. That 8-bit universe (if you will), where we believed that all we needed to model was intent …

If reliability is the key concern (res “digitans”) of a computer then what is the primary concern of the network?

Well, it is reliability but the parameters are different.

For the point to point case it is message fidelity (quality). For the simplex (one way) link latency is barely a factor but as soon as you want the link to go both ways, to support a real time exchange be it conversation or telemetry then it starts to matter. Bandwidth becomes something to be traded off between cost and quality by the service provider.

What the network provides is a function of its topology. For a sequence of links end to end, if we can legitimately call that a network would prioritise transparency, again to uphold the message fidelity and latency concerns of a (logical) point to point link.

Every interconnection introduces latency and reduces the signal quality (which in the digital domain reduces the bandwidth). Increased compression introduces quantisation noise, that “grainy effect” you often see and hear in low quality, highly compressed media. Every interconnection compromises security, if your security model is physical.

Other features offered by network topology:

Thank You!

For reading this far. I’m enjoying putting this together and there is more detail to come I promise ;-)

AI Statement

This article is my own words and analysis. Insights developed as I researched this “the old fashioned way” using google, Wikipedia and archive.org. I used ChatGPT to help fill holes in my understanding where I couldn’t otherwise, and I used it for proofing.

A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.

Everything is broken

Save me Jebus