How to Improve Print Statement Debugging

In high-performance computing, debugging can quickly turn into finding a needle in a haystack — especially when using Print statement style debugging. But that doesn’t mean teams should completely remove Print statements from their debugging arsenal. In this blog, we’ll talk about the ways in which debugging Print statements can be problematic in high-performance computing (HPC), and how using a debuggers aggregation, dynamic analysis, and features such as evaluation points can make debugging and refactoring code a much more efficient process.

What Is Print Statement Debugging?

Print statement debugging is a process in which a developer instruments their application with “printf” statements to generate output while the program is executing. The output generated helps diagnose issues within the program. Print statements is typically the first way developers begin to debug their code because they are easy to add. They often will begin by simply adding printf statements to their code and may progress to more advanced logging solutions provided by open source libraries such as log4c, log4cplus, Boost.Log or glog. At certain levels, print output and logging can give clear insight into what is going on while their program is executing.

When Are Print Statements Typically Used?

Developers will often use print debugging to get insight into what's happening in the processes that they're running. They will print out when certain functions are called, the values of function arguments, values of variables used in algorithms and when messages are exchanged with other processes. But, at scale, and in complex systems with concurrent processes, Print debugging quickly becomes unwieldy to use as the sole debugging method.

Pros and Cons of Print Statement Debugging

How TotalView Can Augment and Replace Print Statement Debugging

So, what should a developer do when they need to debug a program and not end up adding large amounts of Print statement output?

For HPC and complex applications, TotalView can help to make using Print statements in the debugging process a much easier task.

TotalView does this by allowing users to:

1. Print output through evaluation points
2. Print aggregated data
3. Reverse debug processes

Print Output through Evaluation Points

One of the pain points for Print statement debugging is in understanding concurrent or otherwise complex processes. Because Print statements can provide so much data, correlating which processes are interacting where and when can be an exercise in futility.

Even when a developer has pinpointed the process point they want to inspect further, they still need to instrument code with a new print statements, recompile, and rerun the application to get the desired Print output. With

TotalView, developers can use evaluation points to print additional output without changing code, recompiling, and rerunning the program. Check out the following screenshot for an example of where output was generated on the fly through the evaluation point created on line 75.

Additionally, through TotalView’s tvscript functionality, users configure scripts that react to debugging events and when triggered can perform a variety of operations including generating rich log output containing information such as stack traces, variables values, aggregation of data values across parallel processes and even memory leaks.

Print Aggregated Data

Say for instance you have a parallel job of 1024 processes, all generating print output of variable values during their execution. It’s now up to the unlucky developer to comb through all of output from those processes to find any anomalies in the data.

With a debugging tool, like TotalView, the values across all the processes is aggregated before it is printed out. Instead of a developer combing through 1000 lines of Print output to find a single anomalous value, TotalView will show them a simple aggregated display of the information, allowing the developer to quickly pick out errant data. Check out the following output example where the values from variable random_int are aggregated across a series of processes. Aggregation can also be done for arrays and structured data as well.

random_int: Focus: 15:15[0-14,1] 
0x00000000 (0) : 3:3[2.1, 12-13.1] 
0x00000001 (1) : 2:2[4.1, 6.1] 
0x00000003 (3) : 4:4[0-1.1, 11.1, 14.1] 
0x00000005 (5) : 3:3[5.1, 9-10.1] 
0x00000006 (6) : 2:2[3.1, 8.1] 
0x00000007 (7) : 1:1[7.1] 

Now, if you apply the scale of HPC to the situation above — where a program is handling upward of 100,000 processes and threads at once — manually debugging Print statements goes from painful to pointless. The aggregation provided by TotalView in HPC applications can make that process not only feasible, but relatively easy.

Reverse Debugging Print Statements

Print statements can tell you that an error or event has occurred, but they can’t necessarily tell you why that event occurred.

Typically a developer would need to determine where in the code issues may be, add in new Print statements, recompile, and rerun the program to further analyze the code for refactoring, debugging, or otherwise improving that piece of code.

With TotalView’s advanced reverse debugging feature, the developer can quickly go back and forth with the execution to determine how that process works, or doesn’t work without changing code, recompiling, or rerunning the program. Any print statements encountered while debugging will generate output for the user to examine and users can dynamically add additional print statements using evaluation points if needed.

Closing Thoughts

Print statements, while useful, can also become problematic — especially at HPC scale and in complex programs. Leveraging the capabilities of TotalView with Print statement debugging can provide advanced ways for generating print output and can also reduce the reliance on Print style debugging, making those statements more manageable, and more impactful for developers.

Additional Resources

Blogs

• Dynamic Analysis in HPC
• Improved Debugging in C/C++ HPC
• Debugging Mixed-Language Applications

Videos

• Reverse Debugging With TotalView
• Multithreaded Analysis With TotalView

Streamline Print Statement Debugging With TotalView

TotalView can help developers to quickly find pertinent processes within Print statements, set evaluation points, and reverse execution of processes for on the spot refactoring and debugging.

Want to see how TotalView works with your C, C++, or Fortran HPC program? 

Try TotalView Free