(Re)Introducing lua-resty-waf

I’ve spent a good chunk of the last three months hacking away at ModSecurity compatibility with FreeWAF, and thanks to some employer sponsorship and noise from the community, there is enough feature-completeness and stability that it’s ready to sit as the latest tagged release. The project also wears a shiny new name tag: lua-resty-waf.

The development stretch was by far the most prolific (over 100 commits between tags), and introducse a host of new functionality, chiefest being compatibility and translation tooling for existing ModSecurity rules. Historically I wanted to avoid this route- when starting development of FreeWAF I wanted to leverage familiar concepts in a WAF environment without blatantly ripping off an existing community solution. Trying to squeeze ModSecurity ideas into a rigorous mold was a wholly unappealing idea. I told myself there would be too many incompatibilities, that the fundamental differences between Nginx and Apache processing would make such work a waste of time. It took several hours of painstaking manual translation of an existing ModSecurity ruleset to realize that automated translation was a worthwhile task.

Translation tooling currently exists in the form of several hundred lines of Perl, whiskey, and tears. Architecture differences between ModSecurity and lua-resty-waf rule syntax means a bit of faffing about is needed to wrangle logic- herding cats is only slightly messier than bouncing skip logic and rule actions from the start to the end of a chain. There are quite a few (though not as many as I imagined) untranslated elements, mostly due to either no sensible lua-resty-waf equivalent (like SDBM_DELETE_ERROR or MODSEC_BUILD), or lack of lua-resty-waf support (like FILES and related collections- patches welcome!).

The script is straightforward to use: read ModSecurity rules from stdin and write translated rules to stdout:

The script will also warn on translation failures by default so you know what you’re missing:

ModSecurity operators that read from on-disk files (such as ipMatchFromFile) will have their data files read in and added directly to the lua-resty-waf rule. Additionally, the -P flag can be specified to pretty print the translated rules for debugging/verification:

At this point smattering of shell commands and some I/O redirection is all that’s needed to translate rulesets with no mess.

Other release highlights include:

  • Name change: But you figured that out already. FreeWAF started out as an IaaS platform, and continuing development of a software project under the same name didn’t make sense. Obviously it’s free, and obviously it’s a WAF- what else is new? Bringing the naming convention in line with community standards more clearly communicates the nature of the project.
  • Resty.core compatibility: Thanks to the great support from the upstream open sources, lua-resty-waf is now compatible with resty.core out of the box. Using this FFI-based API leads to significant performance bumps.
  • Significantly expanded number of collections: Supporting ModSecurity’s wide scope of input data types required us to handle a good chunk of new collections.
  • Refactored persistent storage mechanism: Bringing persistent storage in line with ModSecurity’s paradigm proved to be a bit of a challenge. It essentially involved flattening a 3-D array into a lua-shared-dict (which is backed by an rbtree). Ultimately we made a few performance sacrifices in order to maintain compatibility with ModSecurity’s design (particularly with respect to its definition of a persistent storage collection). We now handle ModSecurity actions like initcol and setvar in an identical fashion.
  • Refactored request body handling: Varying request content types are handled more sanely. We’re also able to examine the request body as a string for non form upload requests, as well as offer configurable options for dealing with unknown content types and ranging body sizes.
  • Expanded unit and acceptance test coverage: Hopefully now we know before things break on the development branch! :p

There’s certainly plenty of translation work to be done, but it seems that we’ve got enough coverage to start pushing this and encouraging adoption. I’ve had multiple conversations with community members (man that’s still weird to type) who were hesitant to push for adoption without the ability to seamlessly translate existing configurations. Ultimately I think this shift in philosophy is a good thing: I’m viewing this project as less of a (friendly) competitor with ModSecurity and more of a gap-filler: Nginx support in ModSecurity has been sorely lacking for years, and while SpiderLabs’ recent work is promising, it’s still far off from being a stable reality. lua-resty-waf is a stable, immediate solution to a community demand, and I’m excited to see where its development will go from here.

21 thoughts on “(Re)Introducing lua-resty-waf”

  1. Hey man, nice read.

    I was also looking into these sort of technologies. Then I was thinking of developing my own and of course I stumbled upon your work. I’ve also read some of the past blogs you’ve written about and if I can say that you’ve done a great job in establishing this project.

    Anyway, I was hoping to contribute and actively participate into these types of project.

    Reply

  4. modsecurity prevent web robots by user_agent,ip,cookies and son on ? i am puzzled by web robos very long time,because it have ip proxy ,it can disguise UA，so i want to make use of js to increase the web robots’s costing. but it is only an idea,not yet method. combine lua-resty-waf with modsecurity cant finsh it ?

    Reply

  5. Hello poprocks：
    Forgive my broken English which is not my native language. You said ： just about any rule that you can write for ModSecurity, can be used in lua-resty-waf.Is that the mean of I can invoking the ruleset from Modsecurity directly or translate it into lua-resty-waf form? I do not quite understand how to prvent DOS ,Brute-Force Attacks.Can you demonstrate the operation .
    In addition,the variables store in redis ， and how can I access them and useing them such as exporting to linux process.

    Thank you~！

    Reply

    1. >Is that the mean of I can invoking the ruleset from Modsecurity directly or translate it into lua-resty-waf form?
      I think it is ok, modsec2lua-resty-waf.pl wrote by poprocks can help to translate modsecurity rules to lua-resty-waf rules directly.

      >I do not quite understand how to prvent DOS ,Brute-Force Attacks.
      the lua-resty-waf only can do action by rule now, it can not do some sumarry according to client requests . In my oppion, now lua-resty-waf can’t help you to solve your question until now.

      about this attack, I try to solve by offline. I do sumarry by storm for all client requests. if it is too more in one minute, you can stop the request IP by lua-resty-waf.

      >In addition,the variables store in redis ， and how can I access them and useing them such as exporting to linux process.
      Forgive my broken English which is not my native language. I can’t understand your meaning.

      Reply

  6. cool work!
    I am doing test for lua-resty-waf based on nginx. I am puzzled that how to config the “nondisrupt” filed. I found even if it is none the rule can work. Can you give me some suggest about “nondisrupt”?

    Another one question, now the rule saved in disk file, if I want to add one new rule, I need to modify the rule file, can reload the nginx, maybe it is not easy work if there are hundreds of nginx in my environment. Do you have a plan to save the rule in redis or other cache?

    Thanks very much!

    Reply

    1. > I am doing test for lua-resty-waf based on nginx. I am puzzled that how to config the “nondisrupt” filed. I found even if it is none the rule can work. Can you give me some suggest about “nondisrupt”?

      Correct, by definition a rule does not need nondisruptive actions to be valid. BTW these actions are automatically created by the translation tooling, I’m not quite sure what you’re looking for here.

      > Another one question, now the rule saved in disk file, if I want to add one new rule, I need to modify the rule file, can reload the nginx, maybe it is not easy work if there are hundreds of nginx in my environment. Do you have a plan to save the rule in redis or other cache?

      This is solved by any config management system you’d like- chef, puppet, ansible, salt, perl & whiskey :p Managing configs for a large N number of nodes is a solved problem (or, if you don’t consider it solved, it’s far from our problem domain 😉 ). lua-resty-waf does have a (planned to be deprecated) interface to load rules via a string, so you could integrate this into whatever memory/network based system you like. Going forward, we may also choose to just read in SecRules strings directly as well.

      Reply

      1. >Correct, by definition a rule does not need nondisruptive actions to be valid. BTW these actions are automatically created by the translation tooling, I’m not quite sure what you’re looking for here.

        Thanks for you reply. “nondisrupt” is define in modsecurity csr , maybe i can understand the filed meaning there.

        >Managing configs for a large N number of nodes is a solved problem
        Yes, you are right.
        I am waiting for your new interface about loading rules.

        I am not sure if it is ok to response this question here. But i think it is question bypass our waf.
        the following api
        ngx.req.get_uri_args()
        ngx.req.get_post_args()
        ngx.req.get_headers()

        for example：
        ngx.req.get_uri_args()
        some note from https://github.com/openresty/lua-nginx-module#ngxreqget_uri_args
        “Note that a maximum of 100 request arguments are parsed by default (including those with the same name) and that additional request arguments are silently discarded to guard against potential denial of service attacks.”
        Testing by me:
        1) rule checking ”information_schema“
        2) request:
        /test.php?&a0=0&a1=1&a2=2&a3=3&a4=4&a5=5&a6=6&a7=7&a8=8&a9=9&a10=10&a11=11&a12=12&a13=13&a14=14&a15=15&a16=16&a17=17&a18=18&a19=19&a20=20&a21=21&a22=22&a23=23&a24=24&a25=25&a26=26&a27=27&a28=28&a29=29&a30=30&a31=31&a32=32&a33=33&a34=34&a35=35&a36=36&a37=37&a38=38&a39=39&a40=40&a41=41&a42=42&a43=43&a44=44&a45=45&a46=46&a47=47&a48=48&a49=49&a50=50&a51=51&a52=52&a53=53&a54=54&a55=55&a56=56&a57=57&a58=58&a59=59&a60=60&a61=61&a62=62&a63=63&a64=64&a65=65&a66=66&a67=67&a68=68&a69=69&a70=70&a71=71&a72=72&a73=73&a74=74&a75=75&a76=76&a77=77&a78=78&a79=79&a80=80&a81=81&a82=82&a83=83&a84=84&a85=85&a86=86&a87=87&a88=88&a89=89&a90=90&a91=91&a92=92&a93=93&a94=94&a95=95&a96=96&a97=97&a98=98&a=information_schemas

        3) result: the request can bypass the rule

        Reply

          1. is it correct for my test? because I modify some code in lua-resty-waf for testing.

            I will install your original code, if it is same, I will open a GitHub issue for helping the coding perfect.

  7. @poprocks lua-resty-waf stored the rules in memory of every nginx worker now. Do you have a plan to store them in ngx.shared.DICT in the future? Can it save some memory? Thanks for help.

    Reply

    1. I don’t think this makes sense. The rules definitions are small chunks of data- maybe a few KB at most. If anything, storing in a shared dict would be a performance blocker due to the access mutex and (de)serialization required.

      Reply

      1. @poprocks
        Did you do performance testing for this module? Can you share some testing data?

        I try to do this with apache Jmeter.
        the rate of nginx is about 2000 requests per second without this module,
        the rate of nginx is about 2000 requests per second with this module working INACTIVE mode.
        the rate of nginx is about 500 requests per second with this module working SIMULATE mode.

        But all of the request the request_time is the same.

        Reply

          1. Yes, I built with JITable PCRE, and I only have four rules with regexes.

            I think the following API waste more time when it tries to read the POST data. But I don’t know how to change it. Maybe it can’t be changed with ngx_lua.
            ngx.req.read_body()
            ngx.req.get_post_args()

  8. ngx.req.read_body()
    ngx.req.get_post_args()

    it is about with machine. I did test in VM it is bad. In Physics machine it is ok.

    Reply

  9. To get the request uri with ngx.var.uri in this project. But I found if there is rewrite command in nginx conf for example : rewrite ^(.*)$ /index.php break; the ngx.var.uri and ngx.var.request_uri are different.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *