Memory Leak Testing in Continuous Integration Systems
Continuous Integration (CI) is the process of frequently integrating changes from development into the existing code repository, which automatically triggers a build and test of the changes. In CI systems, the code testing process is automated. It typically includes unit tests, functional tests, and, sometimes, performance tests. You will note that none of these tests analyze the impact that new snippet of code will have on the application’s memory usage, i.e. memory leak testing.
Memory testing involves validating a C or C++ application's use of memory and looking for memory leaks or illegal uses of memory, such as buffer overwrites. Memory leaks can be catastrophic to an application, resulting in hangs, buffering, or crashes. In the worst-case scenario, they are reported by a customer.
While developers may run their own memory tests, this step is often done too late, or overlooked entirely in the rush to deliver a new release. Let’s take a closer look at the benefits of adding memory leak testing and memory usage analysis to the CI process.
Advantages of Integrating Memory Leak Detection Into Your CI System
Continuous Integration systems enable constant feedback of the impact of changes on a codebase. Using the automated capabilities of a CI system to conduct routine memory-use analysis delivers the same advantages as it does for regular functional testing:
- New code changes are immediately tested, making them easier to fix.
- Any new illegal uses of memory and memory leaks are detected and reported through the CI system.
- The CI system’s test reports provide transparency to a problem that is often undetected until it is too late.
- Generation of memory result files allows for follow-on detailed analysis of the application's use of memory.
Where Memory Leak Testing Fits In
1. Build the application
Changes to the codebase are gathered and the CI system kicks off a job to build the application. If the application is built successfully, another job is triggered to regression test the application. If a build failure occurs, the CI system records information about the failure in the job build report and notifies the team. Testing of the application will not be done if a build failure occurs.
2. Regression test the application
With a successful build of the application, the CI system starts a job to test the functionality by running unit, integration, and system-level tests. Again, test results are gathered by the CI system and presented as part of the results of the test job and reported to the team.
3. Memory test the application
Using TotalView to test the memory usage of an application under the CI system is not much different than regression testing the application. It usually involves running the test suite again but, this time, running each application test under the control of the TotalView memory debugging technology. Depending on the test suite, this might be as simple as adding an argument.
4. Generate Leak Detection Report
The CI system gathers the test results of each job run and presents them as part of a report for the team. When TotalView memory tests are run, TotalView generates a test report log and, if instructed, memory result files. The CI system can be configured to pick up the log file as part of a post-build action and sent to team members. Application run logs also provide details on the location of the log files and memory result files so that they can be examined
Memory Debugging Technology
The TotalView memory debugging solution provides C and C++ leak detection, memory write overrun detection, and a variety of other memory validation checks. Developers typically use the TotalView user interface to find memory problems in their program, but may not know about a script-driven interface for use in automated environments, such as CI deployments.
Our underlying memory-tracking technology does not require any recompilation or special build of the target application. Instead, it dynamically proxies between your program and the underlying system memory function calls to track all memory-related activity. This solution enables developers to check memory usage on target applications “on the fly,” with minimal runtime overhead.
TotalView script interface, memscript, enables applications to test for memory leaks run within an automated test environment. The script generates output logs and memory recording files, including leak detection results and notifications of any illegal uses of memory.
If memory leaks or memory use errors are detected, developers can load the generated memory recording files into the TotalView user interface and analyze the program’s use of memory. The script-based memory debugging functionality provides the basis for integration into your CI system.
How to Test for Memory Leaks in Jenkins
Now we'll walk step-by-step through how to set up TotalView to find memory leaks in a Jenkins CI system. Integration with other CI systems that understand JUnit or xUnit XML files would follow a similar approach.
Jenkins uses plugins to manage its automated testing process. The JUnit plugin provides capabilities to consume XML test reports that are generated as part of building the product. The XML reports contain information about the tests run during a Jenkins job. JUnit consumes the XML test information and provides graphical visualizations, using the JUnit graph plugin, of the historical test results. (The XML files are also compatible with the xUnit XML file format, allowing them to be easily incorporated into xUnit testing results, too.)
To incorporate the leak detection results from the TotalView HPC debugging platform into Jenkins, we’ll use JUnit to transform leak detection log information into JUnit XML consumable files. Let's start at the beginning.
1. Install the JUnit Plugin for Jenkins
Typically, the JUnit plugin is installed as part of the Jenkins installation. To double check, select Manage Jenkins from the dashboard, then select Manage Plugins. Click on the Installed tab and type “junit” in the filter box. JUnit Plugin should show up in the list. If it does not, click on the Available tab, search for “junit” again and install the plugin.
2. Create a Job to Find Leaks in Jenkins
It’s time to test for memory leaks. In this example, we are assuming that a prior Jenkins job built our target application. From your Jenkins dashboard, click on New Item.
For this session, we will create a Freestyle project to check the leaks on our application. Enter an item name for the project, click on Freestyle project, and then click the OK button.
Jenkins creates our new Freestyle project and presents a page with various options. For this example. we’re only concerned with the Build section. Click on the Build tab at the top of the page and then select Execute shell from the Add build step menu.
Jenkins will present the following form for providing a command to execute within a shell. It's here that we are going to leak detect our application by running it under the control of TotalView’s memory script interface, memscript. Enter the following command into the Command field for Execute shell.
/opt/toolworks/totalview/bin/memscript \ -event_action "termination_notification=>list_leaks" \ tx_local_leak
Note: The path to memscript may need to be adjusted depending on where TotalView was installed.
In this scenario, we are running the target application under the control of memscript directly within Jenkins. This same technique can be applied to your test harness and the test harness would be run by Jenkins.
TotalView’s memory debugging script, memscript, and the underlying memory debugging technology is bundled under the MemoryScape name. It provides many capabilities beyond leak detection, including checking for buffer overwrites, reporting double free events, accessing uninitialized memory, and other common memory problems. For any event that is surfaced, different actions can be done, such as listing the leaks found so far, listing all of the current memory allocations, saving all the memory state information into a file that can be loaded into the memory debugging UI.
Our job is now configured to run our test application under memscript and detect leaks just before it exits. When Jenkins runs our job, memscript will generate a human-readable log file with textual information about any leaks found in the target application. Next, we need to extract the leak detection information from the log file and prepare it for Jenkins through the JUnit plugin.
3. Convert memscript .log Files Into JUnit XML Files
Currently, memscript generates text log files that contains human-readable information about detected leaks, memory errors, and detailed information about all the memory issues. In order to bring this information into Jenkins, we are going to leverage JUnit’s capabilities to read test result information from XML files.
Using a handy Python script, we are going to convert memscript log file information into JUnit’s XML file format. When run, the script takes the name of the JUnit XML file to produce and one or more of the memscript generated log files to convert.
python memscript2junit.py memscript.xml *.log
For a single log file, the script would produce an XML file something like the following:
<!--[CDATA[ … Details from the memscript log file … ]]-->
To perform this conversion within Jenkins, simply add another Execute shell build step to the project and enter the memscript2junit.py command. I’ve placed my memscript2junit.py script in the Jenkins workspace directory so it is easy to locate. You may need to adjust paths based on where your memscript2junit.py script is located.
4. Publish Jenkins Unit Test Report With Leak Detection Results
In order for JUnit to generate test reports, it needs to be told where to gather up the XML files with test result data. To do this, click on Add post-build action at the bottom of our project configuration and then select Publish JUnit test result report. As you can see, there is also an option to Publish xUnit test result report.
The following form allows you to specify where to pick up JUnit XML files. We can simply enter memscript.xml in the Test report XMLs field, since we are generating the memscript.xml files into the Jenkins workspace area.
Finally, click Save to save our project.
5. Generate Leak Detection Results Reports in Jenkins
With our project configured and saved, we click Build Now from the project dashboard to run it.
Jenkins will run our project and show the most recent build in the Build History on the dashboard.
The build is marked as yellow, indicating there is a problem. Let’s explore the leak detection results.
6. Explore Leak Detection Results in Jenkins
Clicking on the build #1 link in the Build History section brings up a summary of the status.
There is one test failure in our tx_local_leak test application. Clicking on the link gives us details about the failure.
The Error Message reports that the test failed due to 9 leaks totaling 450 bytes of leaked memory. The “Stacktrace” area provides the full contents of the log file generated by memscript, including details about all the leaked memory.
The development team now has a way to automatically check their applications for memory leaks, with any failed test results automatically reported in Jenkins.
Find and Eliminate Memory Leaks
CI systems provide many benefits for simplifying the build and test process, shortening development cycles, and improving code quality. Integrating TotalView leak detection and memory usage capabilities into the CI process helps teams find and eliminate memory problems as a routine part of the CI process – and before they get noticed by your customers.