One of the advantages of having a rotating graveyard schedule (two weeks of 10 PM shifts, followed by two months of normal living) is that quiet nights allow for copious amounts of time to muck around on the Internet. One topic that’s piqued my interest over the last few days is tarpitting. Purposefully delaying responses seems a little more interesting than strictly rate limiting; it’s a little more of a retaliatory attitude, without causing any damage at the other end of the connection. Most of the writing I’ve found related to the idea is focused on lower-layer implementation (e.g. the TARPIT iptables module) or SMTP, so I decided to roll my own for HTTP services.

It’s easy enough to build an artificial delay into requests using the OpenResty stack (the Lua Nginx API provides ngx.sleep(), which relays on timers, so there’s no additional CPU load); the trick was figuring out the logic of when to delay, and how long. I came up with a stepwise system that allows for a baseline request rate, and introduces a delay after a predefined threshold is met. This delay is then increased if requests continue, scaling linearly. Under the hood, the request state is stored using a shared dictionary, using the client address and the request URI as the key. This allows for a client to be tarpitted against a single resource (e.g. login pages, expensive dynamic resources, etc) without being bogged down elsewhere. Nginx’s event processing model means that we can stick a number of requests on hold without blocking incoming requests from different clients for the same resource. The module itself is simple enough, using the shm zone to store JSON objects tracking a client’s state:

A lot of this is hardcoded (like the response status, the dictionary key), and the possibility for a race condition does exist (shared dictionaries are locked during read/write operations, but the small amount of work done by the module could allow for a separate access from a different request to write to the shm before the tarpit function completes; I’m planning on using lua-resty-lock to fix that), but so far testing has looked solid. Implementation is very straightforward:

A shared zone called ‘tarpit’ must be setup, otherwise the module will silently bail (by exiting out of the current phase with ngx.OK). The public function of the module (tarpit()) takes three parameters:

1. Number of requests allowed before moving to the next state
2. Time to decrement the state counter
3. How long requests should be delayed initially

The module uses a state to determine if the request should be tarpitted; state 0 adds no artificial delay, while positive-integer states will tarpit the request n number of seconds, multiplied by the current state, where n is the third param provided to the function. So, in the example above, the first five requests to a resource will be handled as normal; the next five will have a delay of 1 second added, the next five will be delayed by 2 seconds, etc. If no requests are received within 5 seconds, the state will decrement, and the state counter will reset to 0.

While researching this I didn’t find much else in the way of HTTP tarpits. The GoLang http-tarpit project functions as a more traditional tarpit, by slowly feeding data back to the client to the hold the connection open; however, this prevents the daemon from forwarding or handling legitimate requests. My solution is an alternative from a true tarpit in the academic sense, but it functions as an effective deterrent to brute force solutions and integrates nicely into an existing, robust stack, rather than living solely as a honeypot. The module is available on GitHub; I’ll be making tweaks for a while (I’ve still got another week left on graveyard).

Update: I took another look at creating a more traditional tarpit with the OpenResty stack, and it seems pretty easy. From what I can tell, the best analogy to a TCP tarpit would be to send a response with a very large Content-Length header, and slowly dribble response characters to the client (this would essentially function as a reverse Slowloris attack). OpenResty’s ngx.sleep() and ngx.flush() make this trivial inside of any content phase handler:

This code block can replace the content in the existing _do_tarpit() function, since the state mechanism would still allow for a small number of legitimate requests to be passed through.