What’s a fast way to insert a large number of elements into a Lua table? What’s the fastest way? And what’s faster than that? There’s a lot of discussion and advice floating around regarding such a primitive topic, so it’s time to dig into some implementation details. We’ll look briefly at a few idiomatic approaches, and discuss the compilation, results, and drawbacks of each.
Working on a WAF solution for the Nginx ecosystem provides a lot of opportunities for discussion, given that such work is a meeting of crossroads. Mixing high-performance engineering, WAF technologies, ModSecurity DSL, and the OpenResty community puts lua-resty-waf in a unique context. I often get asked about other WAF solutions for Nginx, including Naxsi, and how these solutions compare to ModSecurity, lua-resty-waf (and other security OpenResty libs), and commercial solutions. I haven’t spent a lot of time working with the Naxsi project, but I’ve poked at it enough to at least start putting some thoughts on paper.
OpenResty’s biggest selling point is its performance. Embedding Lua allows for some pretty cool new features to pop into a simple Nginx proxy, and the synchronous but non-blocking paradigm introduced by hooking into the event loop (thanks, epoll!) is awesome, but the OpenResty stack as a whole really shines when everything is tied together under the umbrella of approaching, crossing, and then shattering C10k. Out of the box, Lua code embedded into any number of phase handlers runs at an impressive speed, and with the flick of a switch, we can really kick things into high gear.
In recent weeks I’ve found the need to configure and deploy a proper load balancing solution for an authoritative DNS cluster. Now for most solutions (up to a certain scale, and you’d know if you were there) a single-purpose authoritative DNS resolver doesn’t really need a balancing frontend; you can reasonably expect a decent-sized box running a modern kernel to handle several hundred thousand UDP packets per second, with a minimal amount of complimentary TCP traffic. Putting a frontend load balancing tier in front of an authoritative DNS cluster is really only necessary when either hardware redundancy or significant traffic shaping is a requirement, or the generation of authoritative data is expensive and needs to be horizontally scalable. I found myself needing to satisfy a few of these conditions, and have had a wonderful time playing and poking at a purpose-built FOSS DNS load balancing solution in dnsdist.
Perhaps one of the most powerful primatives that lua-nginx-module provides out of the box is a sane, simple wrapper for regular expression operations (via PCRE). String matching, replacing (and now splitting!) via regex allows for much greater flexibility in string processing than Lua’s native string library. Recently while cleaning up an OpenResty InfluxDB client I needed to do some simple string comparison. My knee-jerk reaction was to use a simple expression in
ngx.re.find, but I had a hunch that the overhead of using the PCRE lib would be a waste, and that native Lua pattern searches would be quicker. Time for a benchmark to figure out the most sane solution!
I’ve spent a good chunk of the last three months hacking away at ModSecurity compatibility with FreeWAF, and thanks to some employer sponsorship and noise from the community, there is enough feature-completeness and stability that it’s ready to sit as the latest tagged release. The project also wears a shiny new name tag: lua-resty-waf.
Some time ago I wrote a comparison of lua-nginx-module’s per-request context table and per-worker shared memory dictionaries. Silly me- our examination of the usage of hitting ngx.ctx is pretty naive. Of course, constantly doing the magic lookup to get to the table will be expensive. What happens when we localize the table, do our work, and shove it back into the ngx API at the end of our handler?
Last year I took a look at generating Mod_Security audit logs as JSON data, rather than the module’s native format (which is… err… difficult to parse). My initial approach was incomplete, needlessly introduced additional dependencies, and leaked like a sieve; I ended up abandoning this to work on FreeWAF. Some new use uses came up that would benefit from more structured Mod_Security audit logs, so I’ve revamped a patch to emit JSON data using a more sane approach.
I’ve made some quick notes about this before, but I actually managed to forget the correct flags to make everything go zoom last night while doing some testing, so I’m writing a quick walkthrough for properly building in JIT support into OpenResty’s