What if a simple packet(s) could bring down the internet? In the world of TCP/IP, vulnerabilities like SYN flood attacks have threatened network stability for decades. This deep dive explores the ingenious mitigation known as TCP SYN cookies, from the basics of TCP state management to its pragmatic trade-offs and lessons for modern engineering.

Transmission Control Block

The Transmission Control Block (TCB) is a critical data structure created when a TCP entity opens a TCP connection. A TCB contains the whole state of the connection and must maintain all information required to send and receive segments.

The essential information contained within a TCB includes:

  1. Local IP address.
  2. Remote IP address.
  3. Local TCP port number.
  4. Remote TCP port number.
  5. Current state of the TCP Finite State Machine (FSM).
  6. Maximum Segment Size (MSS).

In practice, this structure requires significant memory space to maintain and can be utilized as an attack vector.

During the standard three-way handshake (SYN, SYN-ACK, ACK), when a server receives an initial SYN segment for a port in the LISTEN state, it transitions the connection state to SYN-RECEIVED. At this point, the server allocates state—by initializing or copying the TCB—to track the incoming connection.

TCP Backlog

The TCP backlog refers to the limit placed by operating system kernels on the amount of state (TCB structures) that can be retained for ‘half-open’ TCP connections (connections in the SYN-RECEIVED state). This limit is imposed to protect host memory from being exhausted by numerous connection requests.

When the backlog limit is reached, the server stops accepting any new SYN segments. This means:

  • Incoming SYNs may be ignored, resulting in the server being unresponsive to new requests.

  • Alternatively, uncompleted connections in the backlog may be replaced, which can disrupt TCP functionality if the rollover speed is faster than the completion of three-way handshakes.

This can prove problematic in either case.

SYN Flood DoS Attack

The problem stated above is called a SYN Flood Attack when it is done deliberately by bad actors. Effectively, the SYN Flood Attack is a denial-of-service (DoS) attack that specifically targets the limited resources allocated during the TCP connection establishment process. The attack exploits the fundamental behavior where a host running a TCP server retains state for some time after receiving a SYN segment.

There are three methods to this attack:

  1. Direct Attack: In this method, the attacker uses a single source device with its real IP address.
  2. Spoofed Attack: This type of SYN flood involves the attacker’s IP address being spoofed on each SYN packet.
  3. Distributed Attack: This attack is created using a botnet. The chances of tracing these attacks back to the ultimate source are extremely low. To make tracing more difficult, the attacker may also spoof the IP address of each distributed device within the botnet.

The Mitigation: Enter SYN Cookies

The Core Trick is to Forget, Not to Remember

As stated above, a SYN flood attack exploits standard TCP behavior perfectly. An attacker sends thousands of SYN packets, often from fake (spoofed) IP addresses, forcing the server to create thousands of TCB copies for bogus connections. The server’s backlog queue fills up, its memory is consumed, and it can no longer accept new, legitimate connections.

SYN cookies turn this process on its head. When a server using SYN cookies receives a SYN packet, it does not create a TCB copy or allocate any memory. It simply crafts and sends a reply (a SYN-ACK packet) and then immediately “forgets” that the request ever happened.

“SYN cookies allocate no state at all for connections in SYN-RECEIVED.”

Only if a valid final response arrives from a legitimate client does the server allocate resources and establish a full connection.

The Secret is Hidden in Plain Sight: The Sequence Number

If the server immediately forgets about the connection request, how can it possibly validate a response from a legitimate client? The genius of the SYN cookie is that it encodes all the necessary state and hides it in plain sight: inside the Initial Sequence Number (ISN) of the SYN-ACK packet it sends back to the client.

This specially crafted sequence number—the “cookie”—contains several key pieces of information:

  • A slowly incrementing timestamp (t): A 5-bit value derived from the system time. This prevents attackers from replaying old cookies, as the server can check if the timestamp is recent and valid.

  • An encoded Maximum Segment Size (m): A 3-bit value that represents the MSS requested by the client. This allows the server to remember the client’s preference for packet size without having to store it.

  • A cryptographic hash (s): A 24-bit value computed from the server and client IP addresses, their port numbers, the timestamp, and a secret key known only to the server. This acts as a digital signature, and early implementations commonly used MD5 for this purpose.

When a legitimate client receives the SYN-ACK, it responds with a final ACK packet. As per the TCP protocol, its Acknowledgement number will be the server’s sequence number (the cookie) plus one. The server receives this ACK, subtracts one to retrieve the original cookie, and can then reverse-engineer the components. It re-calculates the cryptographic hash and compares it to the value in the cookie. If they match and the timestamp is valid, the server knows the request is authentic and can build the full TCB to establish the connection.

Resilience Comes at a Price: The Performance Trade-Off

This clever hack is not a perfect solution. Securing the connection handshake in this way involves compromises, primarily because the server discards the initial SYN packet after creating the cookie.

The most significant compromise is the loss of advanced TCP options. Because the server no longer stores the initial SYN packet, it cannot retain information needed to negotiate features like large window scaling or Selective Acknowledgement (SACK). The loss of these options can hamper performance on high-bandwidth, high-latency networks.

By the early 2000s, a clever workaround emerged, most notably in FreeBSD 7.0, which leveraged the TCP Timestamp option to encode some of this state, but the fundamental limitation remains a trade-off.

A second, more subtle limitation involves the Maximum Segment Size (MSS). Because only 3 bits are available in the sequence number to encode this value, the cookie can only represent a small number of predefined MSS values—typically eight. This restricts the server’s ability to precisely honor the client’s requested segment size.

These trade-offs represent a pragmatic choice: sacrificing peak performance to ensure availability during an attack.

“The largest price to be paid for using SYN cookies is in the disabling of the window scaling option, which disables high performance.”

This Lifesaver Is Often Switched Off By Default

Surprisingly, despite being invented in 1996 and proving its effectiveness against a fundamental TCP vulnerability, SYN cookies are often not enabled by default. D.J. Bernstein, a primary inventor of the technique, has long expressed frustration with this, explicitly calling for users to enable it on his website. On a Linux system, for example, the process is as simple as writing a 1 to the /proc/sys/net/ipv4/tcp_syncookies file.

The deployment history across the industry highlights a cautious approach. While Linux has included the feature for years (typically off by default), Microsoft’s Windows operating systems began including a “TCP SYN attack protection” feature with Windows 2000, which also defaulted to off. It wasn’t until Windows Server 2003 SP1 that it was enabled by default.

The reason for this default-off configuration points back to the performance trade-offs. System designers often prioritize maximum performance over attack resilience in their default state. The assumption is that administrators who require heightened security for public-facing servers can easily enable the protection, while other systems can enjoy the full benefit of advanced TCP features without compromise.

The “Perfect” Successor That Never Was

The limitations of SYN cookies led to the design of a more formal and feature-complete successor: TCP Cookie Transactions (TCPCT). Defined in RFC 6013, TCPCT was engineered to provide stateless connection establishment while overcoming the option-incompatibility issues of the original SYN cookie hack.

There was, however, a critical difference. SYN cookies are a unilateral, server-side trick that works with any standard TCP client. TCPCT, on the other hand, was a formal protocol extension that required support from both the client and the server to function.

Ultimately, TCPCT failed to achieve widespread adoption. The RFC that defined it was eventually moved to Historic status, signaling that it was no longer a recommended standard. This provides a powerful lesson about a classic tension in internet engineering: the battle between a nimble, “good enough” unilateral patch and a comprehensive but burdensome multilateral standard. For internet protocols, a backwards-compatible hack that a single party can implement is often far more successful in the real world than a technically superior solution that requires coordinated change across millions of independent systems.

Conclusion: A Lesson in Pragmatic Engineering

The TCP SYN cookie is more than just a security feature; it is a brilliant example of pragmatic, defensive engineering. It is an imperfect but highly effective solution that works within the strict constraints of an existing, globally deployed protocol. It solved a problem that some considered insoluble by cleverly repurposing a field that was hiding in plain sight.

Further Reading

For more details on TCP SYN flooding attacks and common mitigations, including SYN cookies: