Lesson 1 of 2
What You Lose, and Why 2PC Is Not the Answer
The single most under-estimated consequence of splitting a system: the atomic commit that made hundreds of operations correct without anyone thinking about it.
In a monolith, "reserve the stock, charge the card, create the order" is one transaction. If anything fails, nothing happened. Nobody designs that behaviour — it is a property of the database, applied automatically to every operation. Split those three steps across three services and the property is gone, permanently, and every operation that relied on it must now be designed.
Two-phase commit, and why it is rarely used
There is a protocol that provides atomicity across several resources. Two-phase commit has a coordinator ask every participant to prepare, and if all agree, tell them all to commit. It is correct in the absence of failures, it is genuinely available in some database and message-broker combinations, and it is very rarely the right choice for service-to-service operations. The reasons are worth knowing precisely, because "2PC is bad" is repeated far more often than it is explained.
- 1**It blocks.** Between prepare and commit, participants hold locks. If the coordinator fails after prepare, participants are stuck holding locks with no authority to release them, and the affected rows are unavailable until an operator intervenes.
- 2**It couples availability multiplicatively.** The operation succeeds only if every participant and the coordinator are up. Four participants at 99.9% each gives roughly 99.5% for the operation — worse than any individual component, which is the opposite of what splitting was meant to achieve.
- 3**It requires participant support.** Most modern services expose HTTP APIs, not transaction managers. You cannot enlist a third-party payment provider in your two-phase commit, and the external systems are usually exactly the ones you most wanted atomicity with.
- 4**It scales poorly.** Coordination cost and lock duration grow with the number of participants and with network latency, and the locks are held across the network round trip rather than for the duration of a local write.
Two responses when an operation spans services
Move the boundary
Merge the services, or move the data so that the operation is local to one service again.
- Restores real atomicity with no new machinery
- Removes an entire class of failure and its handling code
- Usually simpler than what it replaces
- Gives up independent deployment or scaling for those components
- Politically harder: it looks like undoing previous work
Choose when: The operation is genuinely one unit of work — money moving between two accounts, a booking and its payment authorisation — and the split had no strong independent justification.
Design for partial completion
Accept that steps complete separately, make each step idempotent, and define what happens when a later step fails.
- Preserves independence, availability and scaling
- Each service remains available when others are not
- Matches how the business already handles many of these cases
- Compensating actions to build, test and operate
- States that are visible to users and must be designed for
- Reconciliation and monitoring become permanent obligations
Choose when: The split is genuinely justified and the business can define what a partially completed operation means — which it usually can, once asked directly.
In practice
Asking the business what a half-finished order means
A retailer split ordering, inventory and payments. Roughly forty times a week, stock was reserved and payment failed, leaving reservations that expired silently and stock that appeared unavailable for twenty minutes.
Constraints
- Around 40 partial failures per week
- Reservations expire after 20 minutes
- Merging the services was rejected: they had genuinely different scaling profiles
- Support had no visibility into the state
Decision
Take the question to the product owner rather than solving it technically. The answer was that a failed payment should release the reservation immediately and notify the customer with a retry link, and that a payment succeeding without an order should be automatically refunded within an hour with an explanation.
Why
The team had been treating partial completion as a defect to be eliminated, which is not achievable once the state is split. Framed as a product question — "what should the customer experience when this happens?" — it had a clear answer that the business was well qualified to give and engineering was not. The technical work then followed the decided behaviour instead of guessing at it.
What it cost
Compensating actions, a reconciliation job and two new notification templates, plus an admin screen for the handful of cases the automation could not resolve. Around 0.3% of orders now go through a visible failure-and-retry path that would have been invisible in a monolith, and the business judged that acceptable in exchange for the scaling independence. Support gained a screen showing exactly which stage an order reached, which they valued more than the engineers expected.
A requirement says an operation "must be atomic"
As a developer
Looks for a technical mechanism that provides atomicity across the services involved, and finds distributed transaction protocols or a shared database.
As an architect
Asks what "must" means here and what the business actually needs, because "atomic" is usually shorthand for "the customer must never be charged without receiving an order". That requirement can be met with compensation and a refund within an hour, at far lower cost — or it may turn out to be a genuine, instantaneous, regulated invariant, in which case the honest answer is that these two things belong in one service. Both outcomes are better than simulating a transaction over a network.
Your team proposes using two-phase commit across three services because the operation must be atomic. What are the two things you would establish first?RevealHide
First, whether all three participants can actually take part: most services expose HTTP APIs and cannot enlist in a transaction manager, and if one of the three is an external provider the protocol is unavailable regardless of its merits. Second, what the availability of the resulting operation would be, because atomicity across participants means the operation succeeds only when all of them and the coordinator are up — three services at 99.9% give roughly 99.6% for the operation, which is worse than any component alone. If both checks pass and the business truly requires instantaneous atomicity, the more important question is why these three are separate services at all, since an operation that cannot tolerate partial completion is one unit of work and the boundary was drawn through it.
Key takeaways
- Splitting state removes atomicity, a property that made many operations correct without anyone designing them.
- Two-phase commit blocks on coordinator failure, multiplies availability downward, and needs participant support you usually do not have.
- An operation genuinely needing cross-service atomicity is evidence that the boundary cut through one unit of work.
- Partial completion is a product question before it is a technical one — ask what the customer should experience.
- Merging services is often cheaper and more reliable than any protocol that simulates a transaction.