Using Static Analysis to Harden Open Source Intrusion Detection Systems (IDS)

by Jeff Sass
Oct. 9, 2017 1 comment 9 minute read Apps & Hardening ids Intrusion Detec... opensource staicanalysis
Download PDF


Intrusion analysts use the principles of network security monitoring (NSM) to help secure computer systems. NSM is “the collection, analysis and escalation of indications and warnings to detect and respond to intrusions” (Bejtlich, 2013). NSM core functions include intrusiondetection systems (IDS), network based IDS (NIDS), host intrusion detection systems (HIDS), and physical intrusion detection systems (Physical IDS) (Berge, n.d). Analysts should evaluate software packages such as IDS and HIDS before deploying them.

There are many different ways to determine how secure a given software package is. One way is to use Aberlarde's security systems engineering approach (Abelarde, 2016). This approach details how commercial and open source software packages are evaluated at each phase of the software development life cycle (SDLC) to determine their security profiles. An advantage of examining open source software is direct access to the code. With direct access, developers can use techniques such as code inspection and static code analysis.

Static code analysis (SCA) is a way of finding issues in software without executing it. The SCA tool accomplishes this by emulating the execution of the different branches of the code by using possible input data. The SCA tool reveals both quality issues (e.g. COPY_PASTE_ERROR, FORWARD_NULL, INCOMPATIBLE_CAST) and security issues (e.g. UNINIT, BUFFER_SIZE, and USE_AFTER_FREE). The SCA tool also shows specific fixes the developer can apply to the source code to reduce the software's defect density. It calculates defect density by dividing the number of defects by the size of the component (usually specified in lines of code). In 2014, the average defect density for open source software was 0.61 per thousand lines of code or KLOC. In contrast, commercial software's defect density was 0.76 per KLOC (Coverity, 2014).

There are many static analysis tools to choose from (OWASP, 2016). Coverity has been gaining popularity in the last ten years after the Coverity Scan service was made available (Coverity, 2016). Coverity Scan allows open source developers a way to submit their code to Coverity’s cloud-based service for analysis and examine the results free of charge. Coverity also offers the same analysis tools in a commercial product that can be deployed locally in their customer’s environment. This paper demonstrates how Coverity’s static code analysis can be used in both deployment scenarios to scan some of the software packages that make up the Security Onion distribution.

Security Onion

Security Onion is a Linux distribution maintained by Doug Burks that includes full packet capture, NIDS, HIDS, and a set of analysis tools (Burks, n.d.). Those tools include:

    • netsniff-ng for full-packet capture
    • Snort, Suricata and Bro for NIDS
    • OSSEC for HIDS
    • Sguil, Squert, Snorby, and ELSA, for data analysis

Using the Security Onion distribution saves time when compared with configuring each of the tools separately. Before starting development with this distribution, step-by-step instructions for installing, configuring, and updating Security Onion should be followed (Burks, 2016). Once that is complete, the developer can examine the source code of the Security Onion software packages to look for security vulnerabilities.

Coverity Scan

The first deployment option for Coverity is Coverity Scan. Coverity Scan is a cloud service where registered open source developers upload their source code for analysis. The Coverity static analysis engine then executes against that source code. Developers review the reported issues, follow the advice to fix the issues, and then resubmit the source code. Coverity Scan is free to the open source community.

Coverity Scan Overview

Coverity Scan started in 2006 as a project funded by the Department of Homeland Security. The main mission was to improve the quality of open source software that the nation was beginning to use. The funding lasted three years until 2009 when Coverity took full ownership of the project (J. Croall, personal interview, January 21, 2016).

The Linux operating system was one of the original Coverity Scan users. Currently, there just under 7,000 projects with over 15,000 individual users using Coverity Scan. Some of the projects include Python, OpenSSL, PHP as well as packages found in Security Onion like Snort, Bro, and Wireshark. To prevent overloading the servers, Coverity Scan limits the number of uploads on projects with large codebases. Developers are permitted to submit up to three builds a day and twelve builds per week if their software package has less than 100,000 lines of code (Frequently Asked Questions, 2016). To help increase the security profile of one of the projects, developers request contributor access from the maintainers.

Coverity Scan Example: Wireshark

Developers follow a four-step process when using Coverity Scan: build; analyze; commit defects; and review results. For the build step, pass the native build command as an argument to Coverity’s command line cov-build tool. Cov-build instruments the native build and stores the information in the intermediate directory specified with the --dir flag. Using Wireshark as an example, the Coverity compile command would be:

    $ cov-build --encoding UTF-8 \
    --dir ~/cov-inter-wireshark make

For the analysis step, upload the intermediate directory to Coverity Scan manually or with a continuous integration system (i.e. Travis-CI). Code analysis is performed on the Coverity servers as opposed to locally on the developer's system. For the commit defects step, Coverity handles this automatically. To review the results, log on to the Coverity Connect web interface where the defects are shown inline with the source code. Section 5 below details this process

The Wireshark project is an active user of Coverity Scan. They have fixed thousands of defects since 2006 and have a very low defect density of 0.26 per KLOC as shown in Figure 1.

Figure 1: Coverity Scan: Wireshark

Coverity Local Analysis

In contrast to Coverity Scan's cloud service, developers can choose to purchase Coverity's commercial offering. The commercial offering runs locally on their network. A standard Coverity deployment uses two machines in a client/server architecture. Security Onion is the local development machine and acts as the client which sends the results to the Coverity database server. By default, Security Onion's software packages are installed as executables. Developers must compile and analyze the corresponding source code by downloading it first. Appendix A lists each of the commands to install Coverity, the GCC compiler, and the source code of the software under investigation. Developers execute the code analysis on the client machine rather than using Coverity Scan's servers. The database that stores the results is on a local network instead of on a Coverity Scan server. Browsing the results is done by logging into the Coverity web server and selecting the appropriate project (i.e. Wireshark) as shown in Figure 2.

Figure 2: Coverity Project Menu

Once a project is selected, select the Coverity Menu (the three-line icon) and choose “Outstanding Security Risks” as shown in Figure 3.

Figure 3: Outstanding Security Risks Filter

This view filters all of the Coverity defects into a smaller list that only includes the security issues. The examples in the next section use this filter.

Fixing Security Vulnerabilities

Before fixing the code, let's examine how the “do no harm rule” can be applied to software as well as how compiler warnings fit into the static analysis picture.

Do No Harm

“Learning to write clean code is hard work” (Martin 2009). In the beginning, source code can be elegant, but as time passes it can become “increasingly sucky” (Skorkin, 2010). Reading source code that one did not write is a critical part of being a good developer. For intrusion analysts who might not be as familiar with reading and writing code, it can be a daunting task.

“The Boy Scouts of America have a simple rule that we can apply to our profession. Leave the campground cleaner than you found it. If we all checked-in our code a little cleaner than when we checked it out, the code simply could not rot” (Martin, 2009).

There are two benefits to the “do no harm rule”: developers improve their coding skills; and the original authors will appreciate the responsible disclosure (Hughes, 2015).

Compiler Warnings

Another aspect of static code analysis is compiler warnings. Getting code to compile is a mini-celebration in itself, so compiler warnings are often ignored. Taking an example from the daq-2.0.6 package, line 859 of daq_afpacket.c declares the variable rc:

        int rc;

Line 866 contains:

        rc = send(instance->peer->fd, NULL, 0, 0);

The compiler warning is:

    daq_afpacket.c:859:25: warning: variable ‘rc’ set but not used 
    [-Wunused-but-set-variable]                      int rc;

The compiler is informing the developer that the return value from the call to send() is set in the variable rc, but rc is not used later in the function. One fix would be to delete line 859 and change line 866 to:

    (void) send(instance->peer->fd, NULL, 0, 0);

This change silences the compiler warning and keeps the code change as close to the original as possible. By assigning the return value from the send() call to (void), the code is ignoring it as it does currently. Another possible fix is to add additional code after line 866, to check rc against all of the return values. That fix changes the program execution and should be reviewed by the maintainers.

Compilers also have the ability to “treat warnings as errors”. Turning on this feature, is a good way to introduce a level of coding discipline in a phased approach to the project. Developers can turn on one warning at a time, fix each one, and then turn on additional warnings when time permits. On Adobe Photoshop, the compilers have the option “treat warnings as errors” turned on which forces a higher level of awareness amongst the team. The continuous build system fails the build with newly introduced compiler warnings. If the build fails, the team fixes the errors quickly. Another reason to turn on “treat warnings as errors”, is to minimize static analysis defects. It is better to eliminate them from the code with the compiler’s help before adding another tool.

Coverity Security Checkers

Coverity 7.7 has over seventy checkers that apply to C and C++ and of these, eighteen focus on security issues. This section focuses on UNINIT, BUFFER_SIZE, and USE_AFTER_FREE.


In ANSI C, the “initial contents of a variable are undefined” (Roberts, 1997). Because the language allows the definition of variables without initialization, there is often a large amount of C code that doesn’t explicitly initialize variables. Some of that code immediately fill the variable after its declaration, so initialization does happen. Sometimes the compiler sets it to zero automatically. Because developers have to remember these rules, there is room for security issues to enter the software. Although there have been proposals to fix this for the C language, for now, developers need to remember the rules (Myers, 2015).

One way to eliminate these issues is to use Coverity’s security checker UNINIT. UNINIT looks for uninitialized stack variables and dynamically allocated memory on the heap that could lead to crashes or security issues. Line 222 of sf_bpf_filter.c in the daq-

2.0.6 package declares an array of int32’s called mem.

Figure 4: mem declaration

Line 406 uses mem in a condition where it was not initialized.

Figure 5: mem assignment

Coverity simulated running through the loop exercising all code paths, as shown in the green text. The simulation found that in at least one condition, the variable mem was assigned to variable A before initialization. To fix this issue, explicitly zero-initialize the array in line 222 as follows.

        int32 mem[BPF_MEMWORDS] = {0};


According to Michael Howard and David LeBlanc’s Writing Secure Code, “buffer overruns that lead to a security patch can cost up to $100,000” (Howard, 2003). Coverity’s security checker BUFFER_SIZE helps developers find and fix defects that involve buffers in their C/C++ code. Taking an example from the snort- package, line 962 of encode.c initializes the variable next of type PROTO_ID to PROTO_MAX. PROTO_MAX is the last element of the PROTO_ID enum defined as:

    typedef enum { 

Line 960 defines the function UDP_Encode as shown in Figure 6.

Figure 6: out-of-bounds read example

The green text shows which execution path Coverity used. The value returned from the NextEncoder function is stored in next which is of type PROTO_ID. There is a case where the returned value could be PROTO_MAX, or 22, which is the last element of the enum. Line 992 indexes into the encoders array at the position specified in next which is one past the end of the array because array indexing starts at 0 instead of 1. To prevent this possible buffer overrun, wrap line 992 in an if/else statement to check that next is less than PROTO_MAX before it is used to index into the encoders array:


Defining variables reserves a place in memory for them. When the program explicitly frees the memory, developers need to ensure there are no cases where the memory is used after it is freed. Using memory in this way can lead to unpredictable results and possible exploitation.

One way to eliminate these issues is to use Coverity's security checker USE_AFTER_FREE. Taking an example from the netsniff-ng-0.6.0 package, line 304 of curvetun_client.c declares a pointer to a structure called ahead as shown in Figure 7.

Figure 7: netsniff-ng - ahead declaration

Line 339 assigns the ahead pointer to ai as shown in Figure 8.

Coverity found in line 358 the ahead pointer was freed. The goto statement at line 367 jumps program execution back to line 311. The next time through the loop at line 339, the pointer was assigned to ai without first checking the pointer for NULL. To fix the issue, add the following line of code after line 358 to set the pointer to NULL.

        ahead = NULL;

Responsible Disclosure

After fixing vulnerabilities, developers have an ethical responsibility to disclose the issues back to the maintainers of the code. For projects like Wireshark that use GitHub, fixes are submitted with a “git push” command by following the project documentation (Wireshark Developer's Guide, 2014). Other projects have mailing lists or defect tracking systems to submit the fixes.

Future Work

In January 2016, Coverity released version 8.0 of its static analysis tools. One of the major new features was the ability to analyze Python code. Security Onion contains a packet manipulation tool called Scapy. Scapy is gaining in popularity especially as intrusion analysts investigate devices that make up the Internet of Things (The 2015 SANS Holiday Hack Challenge, 2015). A future project could examine the static code analysis results of Scapy.


Hardening computer networks with an open source IDS requires the intrusion analyst understand the security profile of the software packages on the system. By utilizing static code analysis on the software that makes up the IDS, the analyst has a better understanding of the security profile the open source software provides. John Carmack, the co-founder of id Software, stated:

The most important thing I have done as a programmer in recent years is to aggressively pursue static code analysis” (Carmack, 2011).

Taking the advice from one the most famous software developers allows the intrusion analyst to utilize some of the best practices from software developers.

Published with the express permission of the author.

Irina Alexandra Negrii 5 months, 1 week ago

Looking for attacks isn't the only use case for IDS, you can also use it to find violations of network policy. IDS will tell you an employee was using Gtalk, uploading to Box, or spending all their time watching Hulu instead of working.