What is "Why Backend Engineering" about?

Backend systems can run for years while hiding critical gaps in how connections, protocols, and execution work. Here is why understanding the how matters.

What topics does "Why Backend Engineering" cover?

This article covers: backend engineering, backend fundamentals, tcp ip, http protocols, backend communication patterns, backend architecture.

Why Backend Engineering

There is a specific kind of discomfort that hits you around year two of building backends.

The app is live. Users are connecting. Requests are being served. And yet something gnaws at you — this quiet suspicion that you are not really in control. You wrote the code, you deployed it, you wired up the framework. But the why behind any of it? Blurry.

Why does the server sometimes fail to accept new connections even when it has CPU headroom? Why does WebSocket traffic behave so differently under a load balancer than HTTP does? Why does your Node.js process go quiet under load instead of crashing loudly?

You built the backend. You just don't understand it.

I know that feeling. And this series is my attempt to close that gap — the gap between making something work and knowing exactly why it works.

The Black Box Problem

ExpandThe backend stack — your app code sits at the top, the kernel at the bottom, and most engineers only ever touch the top layer

Most of us learn backend development the same way. Pick a framework. Follow a tutorial. Handle a request, query a database, return a response. It works. Ship it.

The framework handles the connection lifecycle. The OS handles the socket. The load balancer handles... something. You write the business logic and trust the layers below you.

That trust is fine — until it breaks.

When a production incident hits, the engineers who debug it fastest are not the ones who wrote the most code. They are the ones who can reason about what is happening below the code. They ask: is this a kernel queue problem? Is this a TCP backlog issue? Is this a thread starvation pattern? They have a mental model of the whole stack.

Knowing how the machine underneath you works is not academic. It is the skill that turns a two-hour outage into a twenty-minute fix.

There Are Only a Few Patterns

Here is the first thing that surprised me about backend communication: there are not that many ways a client can talk to a server.

It sounds wrong. Backend engineering feels infinitely complex. But strip away the framework names, the cloud vendors, the architectural acronyms — and you keep hitting the same handful of patterns:

Request-response
Pub/sub
Push
Polling
Long polling
Server-sent events
Stateful vs stateless

That is roughly it. Every backend you have ever worked with fits one of these. Kafka is pub/sub. A REST endpoint is request-response. ChatGPT's streaming output is server-sent events. A mobile app waiting for a notification is either polling or push.

The pattern is almost always one of these seven. What varies is the protocol used to implement it.

Once you see this, the complexity stops feeling arbitrary. There is a shape to it.

Patterns Then Protocols

Communication patterns are the "what" — the shape of the interaction. Protocols are the "how" — the specific wire format and rules used to carry it out.

The same pattern can be implemented with different protocols. WebSockets and SSE both support server-push, but they behave very differently under the hood, especially behind proxies. HTTP/2 and HTTP/1.1 can serve similar REST APIs but with fundamentally different multiplexing behavior.

The protocol you choose has real consequences:

How many connections does it require?
Does it maintain state, or is each request independent?
How does it handle packet loss?
What does it cost to parse?

These questions do not matter when traffic is light. They matter a lot when something breaks or when you are trying to serve a million concurrent users.

The stack goes: UDP and TCP at the bottom, everything else built on top. HTTP/1.1, HTTP/2, HTTP/3, gRPC, WebRTC, WebSockets — these are all concrete protocols solving specific problems on top of those two transport primitives. Knowing the layer each one lives at is what lets you reason about why a proxy works for one but not another.

Where the Real Complexity Lives

After the request reaches your process — after TCP delivers the bytes, after TLS decrypts them, after HTTP parses the headers — your application code runs. That part you know.

What most engineers have never thought about is everything that happened before that.

The kernel accepted the connection first. The OS placed it into a backlog queue. Your server process called accept() to pull it out. A thread (or an event loop tick) read the raw bytes off the socket. A parser turned those bytes into a request object. Only then did your onRequest handler fire.

Each of those steps has a cost. Each has limits. And each can fail independently.

If the kernel's backlog queue fills up before your server can drain it, new clients cannot connect — even if your application is idle. If you have a single-threaded acceptor and a burst of connections arrive, you will queue-stall. If your parser is slow, fast I/O does not help you.

Understanding this layer is what separates engineers who can tune and troubleshoot from engineers who can only reboot.

Who This Is For

This is not a beginner series. If you have not built a backend application that actually served real traffic, that handles things like auth, database queries, and error states — go build one first. Come back when it is live and you are staring at a log wondering why things happened the way they did.

If you have built that backend, this is for you. Specifically if you recognize yourself in any of these:

You are confident writing the code, but fuzzy on what the framework is doing for you
You have debugged connection errors by restarting the process and hoping for the best
You have avoided diving into proxies and load balancer config because it felt like "ops territory"
You have heard "Layer 4 vs Layer 7" and nodded without being entirely sure what it meant

This series walks from communication patterns all the way through to proxies and load balancers. Not a survey — a deep understanding of how each layer works and why the design choices were made.

The Map

Here is the terrain we are going to cover:

Communication design patterns — the seven patterns, why each exists, and when to use each
Protocols — OSI model, UDP, TCP, TLS, HTTP/1.1 through HTTP/3, gRPC, WebRTC, WebSockets
HTTPS configurations — TLS 1.2 vs 1.3, QUIC, 0-RTT, and the tradeoffs at each level
Backend execution patterns — processes vs threads, how connections are accepted, single vs multi-listener architectures
Proxies and load balancers — Layer 4 vs Layer 7, reverse proxies, WebSocket proxying, load balancing strategies

We will start with communication patterns — because before you can reason about any protocol or execution model, you need a clear picture of what kind of interaction you are even building. The first and most fundamental pattern is request-response. If you have built distributed systems before, you will recognize how all of this connects to the high-level design problems that emerge as traffic grows.

The Essentials

Working code is not the same as understood code. Backend systems can appear healthy while hiding critical gaps in how connections, protocols, and execution actually behave.
There are only a handful of communication patterns. Request-response, pub/sub, push, polling, long polling, SSE, stateful vs stateless. Every backend fits one or a combination.
The kernel does more work than you realize. Connection acceptance, socket management, and backlog queuing happen before your application code runs. Understanding this layer is what makes debugging production incidents fast.