Use TCP when Accuracy > Speed
- TCP: Guaranteed delivery, ordered packets. (Web, Email, File Transfer).
- UDP: Fire and forget. (Video Streaming, VoIP, Gaming, DNS).
The 3-Way Handshake Connection
Before sending data, Alice and Bob must agree on Sequence Numbers.
- SYN: "Let's talk. My sequence starts at ."
- SYN-ACK: "I hear you (). My sequence starts at ."
- ACK: "I hear you ()."
Connection Termination (4-Way Wave)
- Client:
FIN"I'm done." - Server:
ACK"Roger." - Server:
FIN"I'm done too." (Can happen later). - Client:
ACK"Bye."
Flow Control vs Congestion Control
These are often confused.
| Feature | Problem Solved | Mechanism |
|---|---|---|
| Flow Control | Receiver is too slow (Buffer overflow) | Receive Window (rwnd) sent in ACK headers. |
| Congestion Control | Network (Router) is congested | Congestion Window (cwnd) estimated by Sender. |
Congestion Algorithms
- Slow Start: Start fast, double speed every RTT. (1 packet -> 2 -> 4 -> 8...)
- Packet Loss: Oops, router dropped a packet.
- Reno/CUBIC (Loss-based): Cut speed in half. "Loss signals congestion."
- BBR (Model-based): Google's algo. Measures Bandwidth and RTT. "Loss isn't always congestion."
The "Bufferbloat" Problem
Routers have large queues (buffers).
- Packet Loss algo: Fills the buffer -> Latency increases -> Finally drops packet -> Sender slows down.
- Result: High latency (Lag) before speed adjusts.
- BBR Solution: Detects when RTT rises (buffer filling) and slows down before packet loss.
Code Example: Socket Programming (C-style)
This is what happens under the hood of http.Get().
import socket
# 1. Create Socket (IPv4, TCP)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# 2. Connect (Triggers 3-Way Handshake)
# Sends SYN
s.connect(("google.com", 80))
# Returns after ACK received (ESTABLISHED)
# 3. Send Data (Push)
msg = "GET / HTTP/1.1\r\nHost: google.com\r\n\r\n"
s.sendall(msg.encode())
# 4. Receive Data
response = b""
while True:
chunk = s.recv(4096)
if not chunk: break # Server sent FIN
response += chunk
s.close() # Sends FIN
print(response.decode())
The Real Cost: Handshakes Are Round Trips
The handshake isn't just protocol trivia — it's latency you pay before a single byte of useful data moves. Count the round trips for a fresh HTTPS request from London to a server in Virginia (~80ms RTT):
TCP handshake: 1 RTT (80ms)
TLS 1.3 handshake: 1 RTT (80ms)
HTTP request/response:1 RTT (80ms)
─────────────
First byte: 3 RTTs (240ms) — before any server processing
With older TLS 1.2, add another RTT. This math is why so much modern infrastructure exists:
- Connection pooling and keep-alive: pay the handshake once, reuse the connection for hundreds of requests. Every serious HTTP client, database driver, and load balancer does this — a service mesh making fresh connections per request would double or triple its latency.
- TLS session resumption and TCP Fast Open: resume prior cryptographic state to shave round trips on reconnection.
- QUIC (HTTP/3): merges transport and encryption handshakes into a single RTT — and 0-RTT on resumption, where the first packet already carries the HTTP request. This is the headline reason QUIC exists; see HTTP Evolution.
- Edge termination: a CDN terminates TCP+TLS a few milliseconds from the user, then proxies over pre-warmed, long-lived connections to origin. The user pays handshake costs on a 5ms path instead of an 80ms one.
Server-Side Tuning That Actually Matters
A few kernel-level realities show up in production incident reviews:
- The accept backlog: completed handshakes queue until the application calls
accept(). Under load spikes, an undersized backlog (somaxconn,listen()argument) silently drops connections that completed their handshake — clients see timeouts while your CPU looks idle. Symptoms live innetstat -s("listen queue overflows"), not application logs. - TIME_WAIT accumulation: a proxy churning through short-lived outbound connections can accumulate tens of thousands of TIME_WAIT sockets and exhaust ephemeral ports — capping you at roughly
port_range / (2 × MSL)new connections per second to one destination. Fixes: connection reuse (best),SO_REUSEADDR, wider port ranges, or more source IPs. - Half-open detection: a peer that vanishes (power loss, cable pull) leaves the other side believing the connection is ESTABLISHED forever — TCP sends nothing when idle. TCP keepalives default to two hours; production systems layer application-level heartbeats on top. See Heartbeat Protocol.
Congestion Control Choices You Can Make Today
CUBIC (loss-based) remains the Linux default; BBR is a sysctl away and widely used by Google and major CDNs. The practical difference appears on paths with random (non-congestion) loss, like cellular networks: loss-based algorithms halve throughput on every dropped packet even when the pipe is empty, while BBR keeps probing the true bandwidth. On lossy long-haul paths BBR can deliver several times CUBIC's throughput. The fairness debate is real — early BBR could starve CUBIC flows sharing a bottleneck — and BBRv2/v3 exist largely to address it.
Interview Tips 💡
- "Why 3-way handshake?" — To prevent old duplicate connection initiations from confusing the server. Both sides must confirm readiness.
- "SYN Flood Attack" — Attacker sends thousands of SYNs but never ACKs. Server memory fills up waiting.
- Defense: SYN Cookies (Stateless handshake until ACK).
- "Nagle's Algorithm" — Buffers small writes to send full packets. Bad for real-time games (disable with
TCP_NODELAY). - "TIME_WAIT" — Why can't I restart my server immediately? The OS holds the port for 2 mins to ensure stray packets from the old connection die.
Related Concepts
- UDP vs TCP
- HTTP Evolution (Why QUIC replaces TCP)
- OS Networking Stack
About ScaleWiki
ScaleWiki is an interactive educational platform dedicated to demystifying distributed systems, software architecture, and system design. Our mission is to provide high-quality, technically accurate resources for software engineers preparing for interviews or solving complex scaling challenges in production.
Read more about our Editorial Guidelines & Authorship.
Educational Disclaimer: The architectural patterns and system designs discussed in this article are based on common industry practices, technical whitepapers, and public engineering blogs. Actual implementations in enterprise environments may vary significantly based on specific product requirements, legacy constraints, and evolving technologies.
Related Articles
HTTP Evolution (H1 to H3)
From text-based HTTP/1.1 to binary HTTP/2 and UDP-based HTTP/3 (QUIC). Why we needed upgrades and how they solve Head-of-Line Blocking.
DNS Architecture
The phonebook of the internet. How Domain Name System works, the hierarchy of Route 53, and recursive vs iterative resolution strategies.
Load Balancing
Layer 4 vs Layer 7 Load Balancing. Algorithms (Round Robin, Least Connections, Consistent Hashing). Health checks and real-world implementation with Nginx.