Varol Cagdas Tok

Personal notes and articles.

Anycast and Scrubbing Infrastructure

The technical problem of DDoS defense is absorbing or discarding attack traffic while allowing legitimate traffic to reach its destination. At the volumetric scale of modern attacks, hundreds of gigabits to multiple terabits per second, this is fundamentally an infrastructure problem. No single defense node can absorb all attack traffic and remain functional. The architecture must distribute the load.

The standard architecture for large-scale DDoS mitigation is anycast routing with geographically distributed scrubbing capacity.

Anycast Routing

In unicast routing, a destination IP address maps to exactly one physical host at one location. Traffic destined for that address follows a path determined by routing tables to that specific destination. If the destination is overwhelmed with traffic, there is no routing mechanism to redistribute the load.

Anycast routing assigns the same IP address prefix to multiple physical locations simultaneously. Each location announces the same prefix via BGP. Routers throughout the internet learn multiple paths to the same address and, per standard routing policy, prefer the path with the best attributes, typically the shortest AS path or the highest local preference. A client connecting to an anycast address is routed to whichever of the multiple anycast nodes is topologically closest.

The property that makes anycast useful for DDoS defense is that attack traffic, like all traffic, is distributed across the anycast nodes according to routing policy. A terabit-scale attack directed at an anycast address arrives at many nodes simultaneously, each receiving a fraction of the total. If the total attack volume is 1 Tbps and it is distributed across 20 anycast nodes, each node receives approximately 50 Gbps, a volume within the handling capacity of a well-provisioned node.

Anycast does not prevent attack traffic from arriving, but it prevents it from arriving at one place. Distributed, the scrubbing problem becomes tractable.

BGP and Traffic Dispersion

How attack traffic distributes across anycast nodes depends on the BGP topology and the geographic distribution of attack sources. Sources from Europe route to European nodes; sources from North America route to North American nodes. Anycast effectively constrains traffic propagation to the topological neighborhood of each source cluster.

This geographic distribution matters for the effectiveness of carpet bombing attacks: distributed attack sources that anycast handles by distributing to nearby nodes are precisely what anycast is designed for. Attack sources concentrated in one region create uneven load, the node serving that region may be overwhelmed while other nodes are lightly loaded.

Operators of anycast networks tune prefix announcements to adjust how traffic distributes across nodes. A node that is at capacity can withdraw its prefix announcement, causing traffic to route to the next-best node. This mechanism allows dynamic load balancing under attack conditions, though withdrawal during an attack may cause transient routing instability as BGP reconverges.

Traffic Scrubbing

Scrubbing is the process of filtering attack traffic from the mix of traffic arriving at a scrubbing center. The clean traffic is then forwarded to the origin. Scrubbing happens after traffic has been diverted to the scrubbing infrastructure (via anycast or via BGP diversion) and before it reaches the destination.

The scrubbing process requires classifying traffic as attack or legitimate. This classification is never perfect, there is always some fraction of attack traffic that is indistinguishable from legitimate traffic, and some fraction of legitimate traffic that resembles attack traffic. The goal is to maximize the fraction of attack traffic discarded while minimizing the fraction of legitimate traffic discarded.

L3/L4 Filtering

At the network and transport layers, filtering operates on packet headers: source and destination addresses, protocol, port numbers, packet size, and header flags. This layer can address:

L3/L4 filtering is fast, implemented in hardware or kernel fast paths, and scales to line rate. The limitation is that it operates on information available in packet headers, which is minimal. An HTTP flood from widely distributed legitimate-looking IP addresses presents no distinguishing characteristics at L3/L4.

BGP Flowspec

Flowspec (RFC 5575) extends BGP to distribute traffic filtering rules. An operator can announce filter rules, specifying source/destination prefix, protocol, port, DSCP values, packet length, and similar L3/L4 attributes, that are propagated via BGP to upstream routers and applied at the network boundary. This allows filter rules to be pushed upstream, dropping attack traffic before it consumes bandwidth on the protected network's links.

The scope of Flowspec is L3/L4. It cannot express application-layer filtering rules. Its effectiveness against application-layer attacks is limited to cases where the attack traffic has distinguishing L3/L4 characteristics.

RTBH (Remotely Triggered Black Hole)

RTBH is the most blunt mitigation available through BGP: the operator announces the target's IP address with a blackhole community attribute. Upstream providers configured to accept this community route traffic to the announced address to a null interface, dropping it entirely.

The effect: the target's IP address becomes unreachable. All traffic, both attack and legitimate, is discarded. This is a denial of service achieved by the defender against their own service, with the intent of preserving the surrounding infrastructure from the collateral damage of the attack traffic. It is operationally correct when the alternative is the entire hosting provider's infrastructure becoming unavailable due to the attacking traffic saturating shared links.

RTBH does not restore service, it eliminates the target. Recovery requires restoring the legitimate announcements after the attack subsides or after mitigation infrastructure is deployed.

L7 Inspection

Application-layer inspection classifies traffic based on request semantics. This requires processing the full HTTP request (or other application protocol), examining URLs, headers, body content, and behavioral patterns. This is computationally expensive compared to L3/L4 filtering and cannot be implemented at the line-rate speeds of hardware filters.

L7 scrubbing infrastructure uses a combination of techniques:

Challenge-response: present potential clients with a challenge that legitimate browsers complete and automated attack tools do not. JavaScript execution, CAPTCHA, invisible behavioral challenges. The challenge is served from the scrubbing infrastructure; traffic that fails the challenge is discarded; traffic that passes is forwarded to the origin. This effectively filters attack traffic from HTTP floods while allowing legitimate browser traffic through.

Rate limiting by behavioral profile: track requests per IP, per user agent, per geographic region, per URL. Apply rate limits based on deviation from legitimate traffic distribution. This requires knowledge of what legitimate traffic looks like, which changes over time and varies by application.

TLS termination: the scrubbing center terminates TLS, allowing inspection of HTTP content. Without TLS termination, HTTPS traffic is opaque and can only be filtered at L4. With TLS termination, the scrubbing center is a man-in-the-middle between the client and origin, which has privacy and certificate management implications but enables L7 inspection.

The Origin Bypass Problem

The standard anycast/scrubbing architecture protects the announced IP addresses, the anycast addresses or the CDN edge nodes. The origin server, which processes actual application logic, is typically behind these layers. If the origin's IP address is known to the attacker, they can bypass the scrubbing layer by sending traffic directly to the origin.

Origin IP disclosure is a persistent problem. It can occur through:

An attacker who identifies the origin IP can send attack traffic directly to it, bypassing the anycast/scrubbing infrastructure entirely. The origin receives the attack traffic on its direct internet connection, which typically has far less capacity than the scrubbing infrastructure.

The mitigation is to restrict the origin to only accept traffic from the scrubbing infrastructure's known IP ranges. This is implemented via firewall rules, network ACLs, or origin-side IP allowlisting. The scrubbing provider publishes its IP ranges; the origin blocks all other sources.

This works until the scrubbing provider's IP ranges are used as attack sources. Some scrubbing providers' IP ranges are also used by legitimate users or by cloud services that share infrastructure. Blocking those ranges creates false positives. The attack technique of sending traffic that appears to originate from scrubbing infrastructure IP ranges is called "scrubber saturation", if the attacker can send traffic through the scrubbing provider's own network, the origin's allowlist cannot distinguish it from legitimate scrubbed traffic.

BGP Blackhole Propagation and IBGP Architectures

Large organizations with multiple peering points use Internal BGP (IBGP) to distribute routing information within their AS. During a DDoS attack, the routing decisions made internally affect how attack traffic is handled before reaching the scrubbing or diversion point.

Hot potato routing, where the AS egresses traffic at the first available exit point, has an equivalent for inbound traffic: traffic is carried internally to the destination after entering at the closest ingress point. For an organization with multiple ingress points, attack traffic entering through all of them must be carried to the scrubbing center or discarded internally before reaching the destination.

IBGP-based black hole triggers allow an organization to propagate drop instructions for specific prefixes to all internal routers via IBGP communities. A scrubbing center receiving traffic for a target prefix announces a null route for that prefix via IBGP, causing all internal routers to discard matching traffic at the ingress point, before it consumes internal backbone bandwidth.

This mechanism is essential for large distributed networks where carrying attack traffic across the backbone to a centralized scrubbing point would saturate internal links. The BGP community-based signaling allows fast, coordinated drop deployment across all network elements.

The Measurement Problem

The effectiveness of scrubbing depends on being able to distinguish attack traffic from legitimate traffic. This classification problem is easier for some attack types than others.

Amplification attacks are easily classified: UDP traffic from port 53 during a DNS amplification attack has a distinctive source port pattern. The fraction of legitimate traffic from port 53 at attack volumes is small. Classification accuracy is high.

HTTP floods from distributed botnets are harder. The traffic may appear to come from residential IP addresses worldwide, use realistic user agent strings, and request legitimate URLs. The distinguishing characteristics are statistical and behavioral, not syntactic. Classification accuracy is lower, and the false positive/false negative trade-off is harder to tune.

Scrubbing infrastructure maintains reputation databases, BGP origin information, geolocation data, and behavioral baselines for each customer to improve classification. The quality of these databases and the sophistication of the classification algorithms determine the practical effectiveness of the scrubbing service.

Fundamental Limits

The anycast/scrubbing architecture has physical limits. A scrubbing provider's total capacity, the sum of bandwidth across all scrubbing nodes, is finite. An attack that exceeds this total capacity will cause some scrubbing nodes to become overloaded, at which point they cannot maintain clean-traffic forwarding for all legitimate traffic.

The published capacity of major scrubbing providers has increased to hundreds of terabits per second. Documented attacks have not yet reached this ceiling for the largest providers, but the ceiling is real and the attacks are growing. The architectural response is continued capacity expansion and more efficient hardware-accelerated filtering.

The classification accuracy limit is less tractable. Attacks that generate traffic indistinguishable from legitimate traffic at all inspectable layers cannot be filtered without accepting significant false positives. The ultimate limit of scrubbing-based defense is the information content of traffic characteristics: if attack and legitimate traffic have the same distribution of observable features, they cannot be separated.

This is the motivation for challenge-response mechanisms that operate outside of traffic analysis: requiring clients to demonstrate browser behavior, CAPTCHA completion, or other proof-of-human properties that the attack traffic cannot produce. These mechanisms work precisely because they require information not present in the traffic itself, they change the observable characteristics of legitimate traffic to distinguish it from attack traffic.

Headless browsers execute JavaScript; CAPTCHA-solving services use human labor or machine learning; behavioral mimicry tools pass biometric challenges. Every challenge mechanism has a corresponding evasion.

Related Articles