The basic premise of fuzzing is throwing intentionally malformed inputs at target software to see if something fails.
The “see if something fails” part is a little trickier than you might think. Software failures come in many forms, including process crashes, memory corruption, excessive consumption of resources, and more.
In the world of Defensics, instrumentation is how Defensics detects failure in the target. Defensics supports a half dozen instrumentation methods:
- Connection instrumentation indicates failure if Defensics cannot make a connection to a network port.
- Valid case instrumentation checks if the target can still respond to valid messaging.
- SNMP instrumentation can be used (for targets that support SNMP) to check on resource consumption or other runtime data.
- Syslog instrumentation can be used (for targets that support Syslog) to check for error messages in an application log file.
- External instrumentation is the most flexible and powerful. Defensics can call out to scripts you have written to check on the health of the target in whatever way you wish.
- Agent instrumentation allows you to check on the health of the target using an agent that runs alongside the target software.
Meet the Defensics Agent Instrumentation Framework 1.1
Defensics 2020.06, coupled with the 1.1 release of the Agent Instrumentation Framework, streamlines and improves the experience of agent instrumentation.
The agent server, which runs on the target device or system, is distributed as a single executable file. In Defensics, connecting to the agent server and configuring agents is simple.
One innovation in version 1.1 of the instrumentation framework is the SanitizerProcessMonitorAgent, a new type of agent that recognizes the output of Address Sanitizer (ASan). ASan is special monitoring that is compiled with your target software and examines memory management. Among other things, it can detect memory leaks, in which the application asks for a certain amount of memory, then does not release all of it. Memory leaks are a concern because an attacker can repeatedly trigger the leak until memory is exhausted, which results in a denial of service.
Let’s see how you could use the agent framework to easily detect a memory leak vulnerability, CVE-2017-7654.
First, time travel
Begin by building a vulnerable version of the target software. The target software is mosquitto, an open source implementation of the MQTT protocol.
The easiest way to build mosquitto is on a Linux system. For this article, I am running Defensics, mosquitto, and the agent instrumentation server all on the same virtual machine, but you could just as easily have separate systems for Defensics and the target.
Get the source code like this:
git clone https://github.com/eclipse/mosquitto.git
cd mosquitto
Now get the 1.4.13 version like this:
git checkout tags/v1.4.13
Git will warn you about “detached HEAD” state, but don’t worry about it.
Install dependencies if necessary:
sudo apt install -y libssl-dev libc-ares-dev uuid-dev
Next, build the project. Include the appropriate flags to build the binary with Address Sanitizer (ASan) support, and don’t bother building the documentation:
make CFLAGS=-fsanitize=address LDFLAGS=-fsanitize=address WITH_DOCS=no
If all goes well, you should be able to check on the version of mosquitto you just built using the following command:
$ src/mosquitto -h
mosquitto version 1.4.13 (build date 2020-06-10 17:37:46+0000)
mosquitto is an MQTT v3.1.1/v3.1 broker.
Usage: mosquitto [-c config_file] [-d] [-h] [-p port]
-c : specify the broker config file.
-d : put the broker into the background after starting.
-h : display this help.
-p : start the broker listening on the specified port.
Not recommended in conjunction with the -c option.
-v : verbose mode - enable all logging types. This overrides
any logging options given in the config file.
See http://mosquitto.org/ for more information.
The target software is now ready to go.
If you were to run the Defensics MQTT Server test suite against this target, it would use connection instrumentation by default. As long as it can keep connecting to the mosquitto listening network port, Defensics assumes the target is healthy and continues testing. However, memory leaks are happening; you can detect them using a different kind of instrumentation.
You don’t need to run the target software yourself. Instead, the agent will take care of running the target and monitoring it for failure.
All you need is the agent server, which is a single executable file. For a 64-bit Linux environment, for example, download agent_linux_amd64.tar.gz and decompress it.
Run the agent server the quick and dangerous way
If you’re impatient, you can run the agent server without any authentication. This is quick but dangerous. The agent server is capable of running arbitrary software, so anyone who can connect to your agent server can run whatever software they please.
If you are feeling reckless, go ahead and run the agent server like this:
$ agent_linux_amd64 server --insecure
2020-06-10T19:24:41.939Z [INFO] Starting Defensics Agent Instrumentation server
2020-06-10T19:24:41.940Z [INFO] api: Server starting at :12345
2020-06-10T19:24:41.940Z [INFO] api: API started in insecure mode. Auth disabled.
In Defensics, all you have to do is load the MQTT Server test suite. Navigate to 4) Instrumentation > Agent.
Check Enable agents and fill in the hostname and port for the agent server. The default port number is 12345.
Now click on Fetch available agents. The dropdown box will list the available types of agents.
If you would rather play it safe and have Defensics and the agent server authenticate each other, keep reading. Otherwise, skip to the following section to see how to start fuzzing.
Run the agent server the right way
The agent server listens for incoming commands and executes them. Obviously, this could be a security hazard. A cryptographic certificate allows the agent server to prove its identity to Defensics, and an authentication token allows Defensics to prove itself to the agent server.
To make all this happen, first instruct the agent server to generate a set of keys. In the following examples, assume that the agent server agent_linux_amd64 is already on the path. The current user is siguser.
The agent server listens for incoming commands and executes them. Obviously, this could be a security hazard. A cryptographic certificate allows the agent server to prove its identity to Defensics, and an authentication token allows Defensics to prove itself to the agent server.
To make all this happen, first instruct the agent server to generate a set of keys. In the following examples, assume that the agent server agent_linux_amd64 is already on the path. The current user is siguser.
$ agent_linux_amd64 generateCerts
2020-06-10T19:06:14.494Z [INFO] Wrote CA cert to /home/siguser/synopsys/agent-instrumentation/certs/ca.cert
2020-06-10T19:06:15.055Z [INFO] Wrote cert to /home/siguser/synopsys/agent-instrumentation/certs/server.cert
2020-06-10T19:06:15.056Z [INFO] Wrote key to /home/siguser/synopsys/agent-instrumentation/certs/server.key
The agent server generates a key pair that it will use to authenticate to Defensics, and also generates a corresponding certificate authority (CA) root certificate. These are stored in the home directory in a subdirectory named synopsys/agent-instrumentation/certs.
Next, start the agent server running. It will start listening for incoming connections and print an API token that Defensics can use to identify itself.
$ agent_linux_amd64 server
2020-06-10T19:10:25.321Z [INFO] Starting Defensics Agent Instrumentation server
2020-06-10T19:10:25.322Z [INFO] api: Server starting at :12345
2020-06-10T19:10:25.322Z [INFO] api: TLS enabled
2020-06-10T19:10:25.322Z [INFO] api: API token: 1d724a41-ddca-4e91-9e10-1542e943a451
In Defensics, load the MQTT Server test suite. Navigate to 4) Instrumentation > Agent.
Check Enable agents and fill in the hostname and port for the agent server. The default port number is 12345.
Next, tell Defensics about the CA certificate so that the agent server will be able to authenticate itself. Click Browse… and navigate to the CA root certificate file that was the output of the generateCerts step. If you have separate systems for Defensics and the target, you will have to find a way to copy the CA root certificate file to a place Defensics can access.
Finally, copy the API token and paste it into the Defensics Authentication Token field. Click Fetch available agents, and you should get a list of available agents in the dropdown list.
Fuzz to locate CVE-2017-7654
Now that Defensics is hooked up to the agent server, you can take advantage of the ASan-enabled agent SanitizerProcessMonitorAgent. Choose it from the dropdown list, then click Add Agent.
For the Target process, fill in the absolute path of the mosquitto executable. Memory leaks are reported only when the process terminates, so check Stop process for instrumentation.
To make the testing go a little faster, you can try lowering the Startup delay, which is the amount of time the agent waits for the process to initialize before delivering a test case.
That’s it!
Start the test run, and you will see failures reported by the agent, having to do with memory leaks reported by ASan. Here, for example, is one of the memory leaks detected by Defensics using the ASan agent:
This particular vulnerability gets triggered by a handful of test cases in the default configuration of the MQTT Server test suite. The vulnerability was discovered using this technique and reported as CVE-2017-7654—and, of course, has been subsequently fixed.
The path to remediation is straightforward. The stack trace provided from ASan (via Defensics) might be enough information for a developer to fix the vulnerability. In addition, the developer could use Defensics to reproduce the failure to debug and fix the problem. Defensics has a streamlined remediation workflow to support not only finding the vulnerabilities but also getting them fixed.
X-ray vision for Defensics
If you have ever seen the TV series “Smallville,” you will remember Tom Welling (as Superman) squinting at walls to show he was using his X-ray vision to see through solid objects. Optimizing your instrumentation in Defensics is the same kind of thing. You can use a variety of techniques to gain better visibility into what is happening inside your target.
In this case, if we had run the MQTT Server test suite against mosquitto 1.4.13 with only connection instrumentation or valid case instrumentation, it would have appeared to be perfectly fine. While connection instrumentation and valid case instrumentation would certainly have detected problems like a process crash or an infinite loop, they were unable to see memory leakage. By adding SanitizerProcessMonitorAgent into the mix, we allowed Defensics to use its X-ray vision to detect that certain test cases caused memory leaks.
Expanding and focusing instrumentation is one way to mature your deployment of fuzzing. When you first run a fuzzer on a piece of software, you’re likely to find an initial bump of vulnerabilities. Once you fix those, you will locate vulnerabilities more slowly. By helping your fuzzer detect failure better, you can maximize the value you are getting from fuzz testing in your secure development life cycle.
Defensics already provides world-class generational fuzzing capabilities and the ability to deliver highly targeted test cases at target software at any point in a complex protocol conversation. Combined with the sophisticated capabilities of the Agent Instrumentation Framework to detect software failure, Defensics enables development teams to locate and fix dangerous zero-day vulnerabilities, stopping trouble before it begins.
Continue Reading
