Lesson 4 of 4
Four Ways Architecture Fails in Practice
The big ball of mud, the distributed monolith, gold-plating, and the ivory tower — what each one looks like from the inside, and the early signal for each.
Most damaged systems are not damaged in novel ways. Four failure modes account for the overwhelming majority, and each has a recognisable early signal — long before the rewrite conversation starts. Learning to spot them is worth more than learning another pattern.
1. The big ball of mud
Nobody decided anything. Every feature was added wherever it was easiest to add, every module imports every other module, and the dependency graph is a hairball. It is the default state of any system where architectural decisions are never made deliberately, and it is the most common failure by a wide margin.
2. The distributed monolith
The most expensive failure mode, because it pays every cost of microservices and collects none of the benefits. The system was split into services, but the services must be deployed together, share a database, and call each other synchronously in long chains. You have added network partitions, distributed debugging, and deployment coordination to a system that still cannot be changed one piece at a time.
Distributed monolith: the shape to recognise
Independent deployment is the point of services. A shared database and a synchronous call chain remove it while keeping every cost.
Three services — orders, inventory and pricing — each call the next synchronously in a chain, and all three read and write the same shared database. A change to the database schema requires all three to be redeployed together.
Entry
API gateway
Services
Order service
calls inventory inline
Inventory service
calls pricing inline
Pricing service
Data
One shared database
All three read and write the same tables
3. Gold-plating
The system is built for a scale, a flexibility or a failure model that never arrives. Kubernetes for a service with four users. An event-sourced ledger for a settings page. A plugin architecture with exactly one plugin, forever. The work is competent, sometimes admirable, and it purchased nothing the business needed.
4. The ivory tower
Architecture is produced by people who do not build the system, handed down as diagrams and standards, and quietly ignored. The documented architecture and the running system diverge until the documents describe a system that has never existed. This failure is organisational rather than technical, and it is the hardest of the four to fix, because the people who could fix it are the ones causing it.
| Failure | Root cause | First remedy |
|---|---|---|
| Big ball of mud | No decisions were made deliberately | Enforce a small number of boundaries mechanically, in CI |
| Distributed monolith | Split by technical layer, not by ownership of data | Give each service its own data, or merge them back |
| Gold-plating | Designed for imagined requirements | Attach a number and a date to every quality attribute |
| Ivory tower | Deciders are separated from consequences | Architects write code and review pull requests in the system they govern |
In practice
Diagnosing before prescribing
A logistics company had split a monolith into eleven services over eighteen months. Delivery had got slower, not faster. The engineering director wanted to know whether to continue the split or reverse it, and both camps had strong opinions and no evidence.
Constraints
- Eleven services, one shared database
- Deployment runbook specifies an order
- Two teams own all eleven services
- Leadership wants a decision in two weeks
Decision
Neither. First measure: how often does changing one service require changing another, and what fraction of releases need the coordinated order? The answer was 70% and all of them.
Why
The problem was not the number of services, it was that the split had followed technical layers rather than data ownership, so nothing could move independently. Continuing the split would have made it worse; reversing it would have thrown away work that was fine. The fix was to give three services their own data and merge four that had no reason to be separate.
What it cost
The measurement cost two weeks that both camps wanted to spend building. It also stopped an eighteen-month rewrite that would have reproduced the same failure with a different service count. Diagnosing before prescribing is slower at the start and is the only thing that reliably works.
A team proposes splitting a service because "it has grown too large". What do you ask?RevealHide
What specifically has got harder, and would splitting fix that? Large is not a problem in itself. If the pain is that two teams keep colliding in the same code, a split along ownership lines will help. If the pain is that the code is tangled internally, splitting it distributes the tangle across a network and makes it dramatically worse. The same proposal is right in one case and a direct path to a distributed monolith in the other, and the only way to tell is to name the pain first.
Key takeaways
- Four failure modes — mud, distributed monolith, gold-plating, ivory tower — account for most damaged systems.
- Each has an early signal available long before the rewrite conversation: estimate variance, deployment ordering, speculative scale, unread documents.
- The distributed monolith is the most expensive because it pays every cost of distribution and collects no benefit.
- Boundaries enforced by the build survive; boundaries described in documents degrade.
- Diagnose which failure you have before prescribing a remedy — the four need different fixes.