Protocol State Machines as Attack Surfaces

Protocol implementations are state machines. A server receiving network input transitions through defined states based on what it receives, in what sequence, and under what conditions. Each state represents a commitment: memory allocated, timers registered, CPU spent on processing. An attacker who understands the state machine can find states that are expensive to enter and cheap to maintain, transitions that are asymmetric in cost, and terminal conditions that are hard to reach from hostile inputs.

The SYN flood is the most widely known example, but only the most prominent instance of a broader pattern. TLS handshake exhaustion, BGP session reset attacks, TCP connection table overflow, application session exhaustion all follow the same logic. The protocol's own design, applied adversarially, becomes the attack vector.

TCP: The Canonical Case

The TCP state machine is defined in RFC 793. A connection transitions through CLOSED → LISTEN → SYN-RECEIVED → ESTABLISHED → various termination states, each with specific processing requirements and resource implications.

The SYN flood targets the SYN-RECEIVED state. When a server in LISTEN receives a SYN, it allocates a Transmission Control Block (TCB), sends a SYN-ACK, and transitions to SYN-RECEIVED. It then waits for the final ACK that would complete the three-way handshake. If the ACK never arrives, the connection remains in SYN-RECEIVED until a timeout expires, typically controlled by tcp_synack_retries, with each retry doubling the wait time and often yielding a total timeout of approximately 63 seconds with default settings.

The resource being exhausted is the SYN backlog queue, a finite table of SYN-RECEIVED connections. When the queue fills, new SYN packets are silently dropped. Legitimate clients receive no response.

RFC 793 was written in 1981 for a cooperative internet with a known, small population of hosts. The three-way handshake's design reflects an environment where source address spoofing was not an operational concern. The first party to commit resources, the server, does so before verification of the client's reachability, because in 1981 the scenario being designed for did not include hostile participants systematically spoofing source addresses.

SYN Cookies: The Stateless Handshake

SYN cookies, proposed by Dan Bernstein and Eric Schenk in the mid-1990s, address the SYN flood by deferring state allocation until the handshake completes. The design encodes connection parameters into the initial sequence number (ISN) of the SYN-ACK:

ISN = MSS_encoded(3 bits) | hash(src_ip, src_port, dst_ip, dst_port, timestamp, secret)(24 bits) | timestamp(5 bits)

The hash uses a server secret, making it computationally infeasible for an attacker to construct a valid ACK without having received the SYN-ACK. When the ACK arrives, the server verifies the ACK number equals ISN + 1, reconstructs connection parameters from the encoded fields, and allocates the TCB only then. No TCB exists between SYN and ACK. A flood of SYN packets from spoofed addresses consumes no persistent state on the server.

The TCP options problem with SYN cookies is worth understanding precisely. Options negotiated in the SYN/SYN-ACK exchange, SACK permission, window scaling, timestamps, must be conveyed in the ACK for the server to know what was agreed. Without state, the server cannot recover them. The 3-bit MSS field in the ISN can encode one of eight MSS values, allowing MSS to survive. Window scaling and SACK negotiation are handled by a separate mechanism: TSOPT-based SYN cookies encode option information in the timestamps field of the ACK when the client supports TCP timestamps. Without this, connections established through SYN cookies fall back to smaller windows and no selective acknowledgment, impacting throughput for high-bandwidth connections.

This degradation is acceptable under attack conditions and invisible under normal conditions, since the Linux implementation activates SYN cookies dynamically only when the SYN backlog overflows.

TCP Connection Table Overflow Beyond SYN Flooding

SYN flooding is not the only way to exhaust TCP connection state. A full ESTABLISHED connection also occupies a socket entry. An attacker controlling many IP addresses can exhaust the socket file descriptor table or ephemeral port range by holding connections open without sending data or sending data slowly enough to prevent the server from closing the connection.

This becomes CLOSE_WAIT exhaustion when the server has attempted to close connections the client refuses to acknowledge. A server that calls close() on a socket moves to FIN_WAIT_1 and eventually to FIN_WAIT_2 or TIME_WAIT. If the client never sends the final ACK, the connection lingers. The socket is not usable for a new connection while it occupies a TIME_WAIT entry, and TIME_WAIT has a default duration of 2*MSL (Maximum Segment Lifetime), typically 60 to 120 seconds. Under conditions of high connection churn to hostile clients, the TIME_WAIT table fills.

TLS Handshake Exhaustion

TLS introduces asymmetric computation cost at connection establishment. The server performs RSA or elliptic curve Diffie-Hellman operations during the handshake before client authentication. These operations are orders of magnitude more expensive than symmetric operations.

For RSA key exchange with a 2048-bit key, a server performs a private key operation, approximately 3-10 milliseconds on a general-purpose CPU, for each new connection. The attacker's cost: sending a ClientHello, one round trip, minimal computation. The server's cost: a private key operation per connection.

TLS 1.3 improves this through its 1-RTT and 0-RTT modes, but the fundamental asymmetry remains. Session resumption mechanisms (TLS session tickets, session IDs) address repeated connections from known clients by skipping the key exchange for resumed sessions. Against an attacker who sends a fresh ClientHello each time, resumption provides no benefit.

TLS ClientHello flooding is distinct from SYN flooding in that it requires completing the TCP handshake, which prevents source address spoofing. This makes the attacker population smaller and enables source-based rate limiting and reputation filtering. However, distributed attacks from botnets or cloud infrastructure circumvent source-based controls.

The amplification factor is more extreme at the TLS layer than at the TCP layer. A server that handles 10,000 new TCP connections per second may only handle a few hundred new TLS connections per second due to the asymmetric cryptographic cost, making TLS-level exhaustion accessible to attackers with modest traffic volumes.

BGP Session Reset

Border Gateway Protocol maintains persistent TCP sessions between peers. These sessions carry routing advertisements that determine how traffic flows across the internet. A BGP session that resets causes the peers to withdraw routes and reconverge, which may cause routing instability and traffic blackholing.

The BGP session reset attack targets the TCP state machine underlying the BGP session. BGP runs over TCP (port 179). If an attacker can cause a RST packet to arrive at a BGP peer connection, the TCP connection terminates, the BGP session fails, and the peers must re-establish the session and re-exchange full routing tables.

The classical form exploits predictable TCP sequence numbers. If an attacker can guess the sequence number of an in-window RST packet, they can inject it into an existing TCP session without having been party to the connection. The BGP session must use source address spoofing, but the TCP session is already established, so no handshake completion is needed.

RFC 4271 (BGP-4) and RFC 4272 (BGP Security Vulnerabilities Analysis) documented this. GTSM (Generalized TTL Security Mechanism, RFC 5082) mitigates this by requiring BGP packets to have a TTL of 255, which spoofed packets from non-directly-connected attackers cannot satisfy. MD5 authentication (RFC 2385) for BGP TCP sessions adds a per-packet authentication tag preventing injection. Both mechanisms are now standard practice.

A related attack is BGP route injection: an attacker advertising more-specific prefixes attracts traffic to their infrastructure, causing a blackhole or redirection. This blurs the line between routing manipulation and denial of service.

SCTP: Designed for Resilience, Not Invulnerable

Stream Control Transmission Protocol was designed in part to address TCP's denial of service vulnerabilities. SCTP's INIT/INIT-ACK/COOKIE-ECHO/COOKIE-ACK four-way handshake defers state allocation to the third message. The server includes a state cookie in the INIT-ACK that the client must echo back in the COOKIE-ECHO. Only upon receiving the valid COOKIE-ECHO does the server allocate the association state. This design eliminates the half-open connection vulnerability that plagues TCP.

SCTP has its own attack surfaces, however. Multi-homing support introduces a path validation mechanism that can be abused. The ASCONF mechanism for adding new addresses to an existing association had vulnerabilities related to address validation timing. SCTP chunk flooding using SACK and similar control chunks creates CPU exhaustion through protocol processing overhead. The protocol's complexity, partly a consequence of its resilience features, expands the attack surface in other dimensions.

Application Session State Exhaustion

The pattern repeats at the application layer. Any application protocol that allocates state before verifying the legitimacy of the requesting party is vulnerable to state exhaustion.

HTTP session management is the most common case. A web application that creates a database session, allocates application memory, or acquires a connection pool entry when a request arrives, before authentication or rate limiting, provides an attacker with the ability to exhaust these resources by flooding requests. The server allocates a session for each request; the attacker's requests never complete legitimately; the sessions accumulate until they time out.

Slowloris, examined in more detail in a later post, is a pure example of state exhaustion at the HTTP layer. An attacker opens many connections and sends HTTP headers slowly, one byte at a time, at intervals just below the server's header timeout. The server holds each connection open waiting for the complete request headers. The attacker accumulates connections until the server's connection limit is reached, blocking legitimate clients. The resource being exhausted is the server's thread pool (in threaded server models) or the event loop's file descriptor table (in event-driven models). Both are finite.

The General Pattern

Across all these cases the attack structure is the same. The target protocol has states that represent resource commitments. These states can be entered in response to received traffic. The conditions under which states are exited involve either successful protocol completion or timeout expiration. The attacker's strategy is to fill these states at a rate faster than timeouts expire them.

The defense, in the abstract, is to move resource commitment later in the protocol, requiring the remote party to demonstrate something, reachability at the claimed address, possession of a secret, ability to perform some computation, before the server commits significant resources. SYN cookies demonstrate reachability. DTLS cookies demonstrate reachability over UDP. Proof-of-work challenges in HTTP push computational cost to the client. Each mechanism shifts the resource commitment point later and raises the attacker's cost per unit of server resource consumed.

The trade-off is functionality. Moving commitment later eliminates some information about what was negotiated before commitment, or adds round trips, or excludes clients who cannot perform the required computation. These costs are acceptable for some applications and unacceptable for others.

Legacy protocols designed without this awareness still run the bulk of internet traffic.