Mod_Security JSON Audit Logs, Revisited

Last year I took a look at generating Mod_Security audit logs as JSON data, rather than the module’s native format (which is… err… difficult to parse). My initial approach was incomplete, needlessly introduced additional dependencies, and leaked like a sieve; I ended up abandoning this to work on FreeWAF. Some new use uses came up that would benefit from more structured Mod_Security audit logs, so I’ve revamped a patch to emit JSON data using a more sane approach.

My new approach uses the included YAJL library, which contains a fast, streaming JSON generator. The API is relatively simple, and with a few macros added for simplicity, it’s easy to stream in new key-value pairs and manipulate arrays and maps.

The generated JSON contains five primary top-level keys: transaction (corresponding to SecLogAuditPart ‘A’), request (containing maps for request headers and body content, as well as the sanitized request line), response (response status, headers, and body), audit_data (SecLogAuditPart ‘H’ data), and matched_rules (represented as a JSON array). Below is an beautified example of an audit log entry generated from a simple rule designed to mitigate an ongoing spoofed Googlebot WordPress login bruteforce botnet:

And the corresponding Mod_Security rules:

Of note, new functions were added for generating stopwatch and producer data in JSON format. Request/response sanitation is also properly handled, using the same memory overwrites that native logging employs.

YAJL’s generator allocator provides the ability for users to specify a struct containing function pointers to memory allocation routines to replace the default (malloc, realloc, and free). This patch does not leverage that or the available APR pools, largely to simplify the changeset, and because performance testing indicated no significant difference between the two. Indeed, benchmark tests for both patched JSON and natively audit log generation shows no significant difference; JSON logging maybe be slightly more performant because it contains only a single atomic write (native audit logs may make several dozen or hundred write syscalls). The choice to buffer the entire audit log entry and call a single write is based on the desire to avoid having broken JSON structures written to disk.

Building with this patch is straightforward; just configure with –enable-json-logging=yes and build Configuring this option is straightforward, with a new configuration option ‘SecAuditLogFormat’, which takes the string ‘Native’ or ‘JSON’:

Audit log data is not written in a beautified fashion (what a pointless endeavor would that have been!), and is newline-separated. I’ve submitted a pull request for this patch, though I’m not holding my breath, as the json_logging branch on upstream looks like the developers want to enable a runtime option for enabling JSON logging, an approach this patch now follows. I’m hoping people find this useful, and will happily entertain comments/requests regarding the structure and data points present within the generated entries.

Update: Since posting this, I’ve made a few updates to the json_logging_audit branch, including removing the compile-time flag, and updating the matched_rules portion of the structure (see this commit for more details). The example audit log entry shown above is the correct version of the data emitted.

7 thoughts on “Mod_Security JSON Audit Logs, Revisited

    1. Thanks! I hope people have been able to take advantage of this!

      This patch hasn’t yet been merged into the master branch at https://github.com/SpiderLabs/ModSecurity; the tags on the pull request indicate that it’s planned for merge into ModSecurity 2.9.1. Additionally, I’ve been in talks with members of the development community; their focus is making sure that the previously-designed JSON logging structure that’s planned for libmodsecurity (e.g. ModSecurity 3.0) and the structure I’ve presented are identical. I believe they are planning on a new minor release of ModSecurity in Q1 2016, so hopefully we will see this added into ModSecurity proper within the next few months!

  1. In Some Cases Mod-Security produce invalid json audit log, its happens when we enable request and response header.

    Can we encode request and response header with base64 encoding

    Thanks

    1. I think a saner approach will be to escape the data with log_escape_ex (or similar). It’s on my radar, but I haven’t gotten there yet. Patches welcome 🙂

    1. That question is relevant more to the underlying log writer mechanism, and not the data format in which logs are written. ModSecurity does provide mlogc to ship logs to a remote server, but I haven’t looked at compatibility with JSON formatting. I’ve seen a number of methods to transport audit logs, including lumberjack, filebeat, and hand-rolled transporters (think rsync wrappers 😉 ).

Leave a Reply

Your email address will not be published. Required fields are marked *