Event-driven almost always sounds like the more modern architecture. One service publishes an event, others react to it, nobody knows anybody, everything is decoupled. On a diagram that works beautifully.
In a production system, though, that decoupling comes at a price: delayed consistency, retries, ordering problems, harder debugging and business processes whose state is suddenly spread across several systems.
That is why my default is not event-driven. If a service needs an answer from another service now, I call it synchronously. Events come in when the producer has already finished its work and other parts of the system should react to it independently. That sounds like a small difference. For the architecture it is a fairly large one.
Synchronous does not mean badly coupled
Take a checkout. The customer clicks "place order". Before I tell them the order has been accepted, I want to know: is the item available? Is the price still valid? Can the payment be authorized? Is this customer allowed to order?
If those answers are part of the decision whether the order may exist at all, I see little reason to turn it artificially into an asynchronous process:
Checkout
|
+--> Pricing Service
|
+--> Payment Service
|
+--> Order Service
|
v
Response to the customerREST or gRPC are perfectly legitimate tools for this. The caller waits for an answer. If the payment is declined, it finds out immediately. If the payment service is unreachable, the checkout can react. The control flow is visible.
This dependency is not an architectural mistake; it reflects the business reality. If process A cannot make a decision without the result of process B, A and B are coupled in the business sense. An event bus does not make that coupling disappear. It only makes it harder to see.
Which protocol suits the direct call is covered in REST or gRPC for backend services?
Decoupling that is not
"We need to decouple the services" is something I hear a lot with microservices. Then a direct call from the order service to the payment service turns into something supposedly more elegant:
Order Service
|
OrderRequested
|
v
Event Bus
|
v
Payment Service
|
PaymentProcessed
|
v
Event Bus
|
v
Order ServiceAt the network level the two services no longer know each other. In business terms, the order service knows the payment process exactly as well as before, because it cannot complete the order before it knows whether the payment went through. The coupling is still there. What has been added: two events, an intermediate state, event schemas, retry and timeout logic, correlating request and response, handling lost or late responses, a dead-letter queue and observability across several asynchronous steps.
I have not decoupled anything. I have turned a function call into a distributed protocol. That can be necessary. But I want a better reason for it than "event-driven scales better".
Question or fact
My first filter is therefore a single question: does the caller need the result now? If service A cannot sensibly continue without service B's answer, I start synchronously.
Can this customer pay?
What is the current price?
Does this record exist?
Is this account locked?
May this user perform this action?Those are questions, and a question expects an answer. It is different when something has already happened:
OrderCreated
PaymentCaptured
CustomerRegistered
InvoiceIssued
SubscriptionCancelledThose are not questions, they are facts, and facts are excellent material for events. I make the difference visible in the names too: CreateInvoice, SendWelcomeEmail or UpdateCRM are commands. An event is in the past tense, because it describes something that has already happened.
That has an important consequence: the producer does not need to know who reacts. As soon as an order has been created bindingly, the order service publishes OrderCreated. Email, analytics, CRM, fulfilment, the data warehouse and a loyalty service may react to it. If fraud detection is added tomorrow, I do not change the order service; the new consumer simply listens in.
This is where real decoupling happens. Not because HTTP is no longer used, but because the producer does not need to know anything about the reaction. For me it is the cleanest use case for event-driven architecture there is.
An order process contains both
Rules like "our services only communicate via events" strike me as a restriction nobody needs to impose on themselves. A realistic flow looks like this for me:
synchronous
Browser ----------------------------> Order API
|
| synchronous
v
Payment Service
|
v
Save order
|
OrderCreated
|
+---------------+---------------+
| | |
v v v
Email Analytics FulfilmentThe critical part stays synchronous, the customer gets an immediate answer. Everything that can happen afterwards without that answer depending on it runs asynchronously. So the more interesting question is not "synchronous or event-driven?" but: where does the operation that needs an answer now end? Everything after that is a candidate for asynchrony.
The simplest example is the welcome email. Does the response to POST /customers have to wait until the mail server has accepted the message? Of course not. I save the customer, the registration is complete, and then CustomerRegistered is emitted. If the mail provider is down for three minutes, the customer is still registered and the message is delivered later. Registration and sending the email have different availability requirements, so they may be different processes.
A queue and an event bus are not the same
This is where a lot gets lumped together under "event-driven" for no good reason. "Please process this image" is not the same as "this image has been processed". The first is work, the second is a fact.
For work I use a queue. The upload service wants something done, and exactly one worker should take it on:
Upload Service --> SQS Queue --> Image WorkerFor a business fact, publish/subscribe fits better, because several systems are independently interested in the same event:
OrderCreated
|
EventBridge
|
+----+------+
| | |
v v v
CRM Mail AnalyticsQueue: somebody should do some work. Event: something has happened, and whoever is interested may react. Both are asynchronous, but the meaning differs.
Two cases in which I move towards asynchrony quickly follow directly from this distinction. The first is load spikes: if 500,000 records arrive within three minutes and their processing may be spread over an hour, a synchronous design would have to size both sides for the same peak. With a queue the producer accepts quickly, the queue grows, and a worker pool works through it at its own capacity. That buys resilience, not architectural aesthetics. The second is fan-out: if the order service calls CRM, mail, analytics, the data warehouse and loyalty itself, it knows five systems and has to be changed for the sixth. With an event it only describes its own business truth, and what the organization does with it is no longer its responsibility.
How nightly runs become a queue with retries is covered in From cron jobs to queues.
The price: delivery, consistency, readability
Event-driven moves failures around; it does not remove them. With a synchronous call the failure is usually visible straight away: the payment service answers with a 500, and I can do something with that. A message, on the other hand, can be processed, later, several times or only after a retry. SQS and EventBridge guarantee that a message arrives at least once, not exactly once. If a consumer dies after doing the work but before acknowledging it, the message is delivered again once the visibility timeout has expired:
Message received
|
Create invoice
|
X process dies before acknowledging
|
Message delivered again
|
Create the invoice a second time?If the answer is "yes", I have a problem. Idempotency is therefore basic equipment for asynchronous processing. The robust question is not "will this message surely arrive only once?" but: what happens if it arrives twice?
Then there is consistency. The customer service stores customer 4711 and publishes CustomerCreated. The CRM may process it 50 milliseconds later, maybe two seconds, maybe five minutes because something is failing right now. For that long the customer exists in one system and not yet in the other. Technically that is normal; in business terms it has to be acceptable. For analytics almost always, for a search index often, for a welcome email of course. For a credit check whose result I need before I may pay out a loan, probably not. Eventual consistency is not a technical property I pick. It is a business decision.
And finally readability. A synchronous flow is pleasantly boring: A → B → C → response, and when something goes wrong I follow the request. Event-driven, the same business process can be spread across four events and five services. When support then asks "why was order 92815 not shipped?", I have to find out whether OrderCreated was published, whether the consumer received it, in which version, whether it arrived more than once, whether it sits in the dead-letter queue, which correlation ID belongs to it and whether the operation failed or simply has not finished yet. That is not an argument against events. It is their price.
So I do not build a serious event architecture without observability. For every event I want at least the event ID, type, timestamp, producer, correlation ID, business ID, consumer, processing status, number of attempts and the error. And I want not just queue depth and error rate, but the business timeline:
Order 92815
created: 14:02:11
paid: 14:02:12
event emitted: 14:02:12
fulfilment: 14:02:13
mail: 14:02:14
analytics: 14:02:18Otherwise the first major incident produces a strange state: every system works, only the order does not, and nobody knows where it is stuck.
How the trace context travels across a queue is covered in Introducing OpenTelemetry in PHP and Go.
Events are APIs
Almost everyone treats a REST API as a contract automatically. With events, surprisingly often nobody does. One person extends the JSON, another renames a field, a third reads a status differently, and suddenly three of five consumers still expect the old structure.
For me a published event is therefore an API, perhaps the more dangerous kind. With a synchronous API I usually know who calls it. With an event the producer deliberately does not know who is listening. So: version the schema, do not remove fields lightly, document the meaning, extend compatibly, keep consumers independently deployable. The decoupling only holds if the contract is stable. Otherwise the producer does not know its consumers, but breaks them with every change anyway.
Event chains and the hidden workflow
One event triggers another, that one triggers the next, and after a few months the business process looks like this:
A -> Event
|
B -> Event
|
C -> Event
|
DEach step looks cleanly decoupled. The process is still tightly coupled, just distributed. The dependencies have not disappeared, they have become invisible. At the latest when the order matters for the business, I ask: do I have independent reactions to events here, or actually a workflow?
Take creating an order, reserving the payment, reserving stock, triggering shipment. Now the stock reservation fails. Does the payment have to be reversed? What happens if the reversal fails too? Who knows what state the operation is in? All of that can be solved purely with events, as a choreography in which each service reacts to the previous one. That can be elegant, until nobody can find a place where the whole process is visible.
My tendency: independent reactions may be choreographed. A real business process needs a visible owner. That can be application code, a state machine or a workflow system such as AWS Step Functions. Which one depends on the problem. What matters more is that someone owns the state, including that of the compensating steps the saga pattern describes.
What such a chain looks like in operation when nobody planned it is covered in The distributed monolith.
The most dangerous mistake: save and publish
A classic event flow looks harmless at first: save the record, then publish the event. If the process dies in between, the order exists but nobody hears about it. The other way round is no better: the event is out, the database change fails, and the consumers react to something that never happened in business terms. That is not an exotic edge case but a system boundary that has to be solved explicitly.
The established pattern for it is the transactional outbox. Business state and event end up in the same local transaction:
BEGIN;
INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ('92815', '4711', 12900, 'created');
INSERT INTO outbox (id, type, business_id, payload, created_at)
VALUES ('evt-7f3a', 'OrderCreated', '92815', '{"total_cents": 12900}', now());
COMMIT;A separate process reads the outbox and publishes the entries, either by polling or via change data capture on the database log. If publishing fails, it tries again, which is exactly why the event may arrive twice, and the consumer absorbs that through its idempotency, for example with the event ID as the key. Database and message broker thus do not become a distributed transaction, but two steps of which the second may be repeated as often as needed.
Where events are business-critical, I consider this part more important than the question of which event bus to use.
The same underlying problem, two writes without a shared transaction, shows up in every data migration under load: Dual write and backfill.
My rules, and what a backend looks like with them
Synchronous when the caller needs the result now, the operation is part of the same business decision, an error has to go straight back to the caller, the flow is manageable and consistency matters more than temporal decoupling.
A queue when work can reliably be done later, load spikes need buffering, producer and worker have different capacities, a job is handled by exactly one consumer and retries are part of normal operation.
An event when a business fact has already occurred, several independent systems react to it, the producer should not know these consumers, new reactions should be added without changing the producer and eventual consistency is acceptable for the business.
These are not absolute rules. But they have helped me far more than the sentence "we want an event-driven architecture". A typical SaaS backend then usually ends up hybrid:
REST / gRPC
Client --------------------------------> API
|
| synchronous
v
Account Service
|
| synchronous
v
Payment Service
|
SubscriptionCreated
|
+----------+----------+
| | |
v v v
Mail Analytics CRM
Upload Service --> SQS --> Processing WorkerThere is no ideology in that. Three kinds of communication sit side by side: direct calls where answers are needed, a queue where work is distributed, events where state changes that have happened interest other systems. Built by the meaning of each kind of communication, not by a uniform technical dogma.
Event-driven is not a goal
I would never start an architecture project with the goal "we are making the system event-driven now", any more than with "we are building microservices now". Those are means. First I want to know which problem should disappear. Are synchronous dependencies dragging each other down? Do load spikes need absorbing? Do several systems need the same business changes? Are slow side processes blocking an important request? Should new consumers be able to join independently? Then events or queues can be exactly the right answer.
But if the problem is only "service A needs a piece of information from service B", the solution may be simple: Service A → Service B. A direct call is then not technical debt, but the most honest representation of the business dependency, with clear timeouts, proper error handling, observability and no bad conscience.
For me, event-driven is not the more advanced version of synchronous communication. It solves a different problem. I decouple where systems are allowed to be independent. Where a business dependency exists, the architecture may show it.
If you are facing this decision right now, the approach is described on the page for backend development.
