Most people don’t like flying, I think. No one likes standing in long lines or sitting around, but I don’t mind the extra free hours. It gives me a chance to hack around on fun things I normally wouldn’t have the time for. I’m on my way back home from San Francisco, so I took advantage of the time to start hacking around with building a simple RPC protocol in OpenResty. It’s been a good chance to work with binary data and exercise the Nginx stream module. Tossing some notes and snippets in here as an outlet; I’m hoping to have a more formal follow-up in a few weekends as life starts to settle back to normal.

First up, we need to compile OpenResty with the stream-lua-module. The stream module version bundled with the current release of OpenResty doesn’t provided the peek method of reqsocket, so the socket calls themselves will rely on append a newline to the data packets, but we can still design the interface to declare the packet length in the body itself (ohp, I lied – while writing this, the official RC1 was kicked out).

We need interfaces for serialization. The data packets themselves are simply MessagePack over TCP (protobufs would be another fun option, if there’s a concrete specification for the application data):

We calculate the length of the packet message and prepend it to the message itself when serializing the message. The message length header is comprised of two bytes, using a byte array to hold the data and coercing the bytes into a Lua string:

I offset the length byte values by a little bit to avoid clashing with control characters – trying to call string.char on a byte containing 0x0a wasn’t actually returning back the value ’10’. I’d like to try to find a better solution for this, but the now-relatively arbitrary length limitation isn’t restrictive for something like this, so it’s not the end of the world.

From here, we can write a dispatch mechanism to take action based on the command sent in the packet:

Standard stuff here, thrown together in a few minutes. The play between cmds and handlers lets the client send the command as an integer constant, rather than a string, to save space (which, yes, is minimal, but this is just for funsies :D). We can use a Lua content handler to read a packet and response appropriately:

And a simple little client:

Again, remember that because we can’t peak into the stream to grab the header length bytes, we just need to read the whole packet directly. Giving it a bit of exercise, along with a packet dump showing the communications:

There’s a lot of fun potential with this, and I’m really enjoying playing around with some binary math and the streaming subsystem. I’m looking forward to fleshing this out in the upcoming weeks. The growth of microservice and service mesh patterns will mean that RESTful HTTP APIs will lose the massive foothold they have in modern architectures – gRPC, GraphQL, Thrift, and other protocols are continuing to grow and in adoption and will further shape the landscape of service communication in the coming years. OpenResty is positioning itself well to be a high-performance, flexible, and stable solution for binary and custom network protocols in the near future. It’s very exciting to see the developments being made and the possibilities opening up with existing and future releases.