What's New

TotalView 2024.3 builds on its powerful debugging capabilities with the following new features, platform updates, and enhancements: 

Process and Thread Navigation Improvements

TotalView 2024.3 adds the ability to cycle through processes with a group of processes and threads within a process with new Next Process/Thread, Previous Process/Thread options in the Process and Thread menu.

The Next Process action will cycle to the next process in the control group and Previous Process will cycle to the previous process in the group. Next and Previous Thread work similarly but within the process in focus.

To focus on a specific process or thread use the new Jump to Process/Thread (Px)… menu
option to bring up the Jump To Process/Thread dialog. Enter a debugger ID or rank of a
process and thread you wish to focus on and press OK to have TotalView change your
process/thread focus.

Process and thread cycling and Jump to Process/Thread can be conveniently triggered using the P+, P-, Px, T+, and T- buttons in the Process and Thread view’s toolbar and also through keyboard accelerators.

Array View Filtering

TotalView 2024.3 adds the ability to filter array contents within the Array View using new Filter options in Array Configuration Options dialog. Simply select a filter operation from the first drop down (such as '==', '<', or '<=', etc.). Then add a value to filter by or choose a special IEEE floating point value from the dropdown. If you are adding multiple filter expressions, you can also choose to && or || the expressions. Click OK or Apply to use the filter on the selected array data.

TotalView Downloads SHA256 Checksums

Beginning with the TotalView 2024.3 release TotalView file releases will use SHA256 instead of MD5 checksums to verify the downloaded files are the same as the files prepared for release.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed with TotalView 2024.3.

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

View Release Notes    Get Latest Release

TotalView 2023.4

Registers View

TotalView 2023.4 adds the ability to easily examine the CPU and GPU registers for target programs as they are debugged. Display the Registers View by selecting Window > Views > Registers.

To edit the value of a register or cast it to a different type simply right click on it and select Add to Data View or drag-and-drop it into the Data View. Through the Data View the register value can be changed by double clicking on the value and entering a new one. The type can similarly be changed by double clicking on the existing type and entering a new one.

Assembly View Early Access

The ability to debug low-level machine instructions has been added to TotalView 2023.4. The new Assembly View displays machine instructions for the current function in PC scope. Instruction level stepping commands enable fine-grained control stepping across machine instructions. Instruction level breakpoints allow precise program execution control and enable debugging tough situations where it is important to understand individual program instructions and data move operations. To display the Assembler View right click in the source area and select Show Assembler.

New C++ Type Transformations

TotalView 2023.4 adds additional type transformation support for the following C++ collection classes and iterators for traversing through the data elements on the collections.

• deque
• forward_list
• list
• queue
• priority_queue
• stack
• array
• pair
• tuple

CUDA 12 Support

TotalView 2023.4 adds support for CUDA 12. See the release notes for full details and recommended configurations to successfully debug on CUDA 12 systems with TotalView.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed with TotalView 2023.4

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

View Release NotesGet Latest Release

TotalView 2023.3

Array Slicing and Striding

TotalView 2023.3 adds the ability to define a slice and stride for arrays in the Array View.  Simply click on the configuration icon in the Array View toolbar to display the Array Configuration Options dialog.

Define the start and end dimensions for the array and an optional stride for the data in the dialog.

Use the apply button to live update the display in the Array View or Ok to close the dialog and use the defined settings.  The type of the array can also be changed through the Options dialog.

Memory Debugging Block Overwrite / Guard Block Report 

Building on the memory block overwrite detection capabilities added in TotalView 2022.2, version 2023.3 adds the ability to generate an on-demand Corrupt Guard Block Report that lists all memory blocks that have had their bounds overwritten.  

TotalView detects memory block overwrites by padding a pattern filled guard block on either side of the allocated memory block.  If the pattern is disturbed an overwrite has occurred.  To turn on overwrite detection, open the Memory Debugging Options dialog, and toggle the Guard allocated memory checkbox.

At any point during program execution, all allocated memory blocks can be checked for overwrites by opening the Corrupt Guard Block Report.  This is easily done by pressing the icon in the Memory toolbar or selecting Debug > Corrupt Memory Block Report from the main menu.

The Corrupt Guard Block Report shows all the memory blocks found with overwrite violations.  Locations are broken down by process, file, function, line number, and the list of individual blocks that have been overwritten.  TotalView also reports if the block was overwritten at the start (pre) or end (post) of the block.

Python 3.11 Debugging Support

Debugging Python 3.11 applications along with C/C++ Python modules is supported in the TotalView 2023.3 release.  Python 3.11 significantly changed how the debug and program information was stored.  TotalView has been updated to find the proper information and provide a unified Python and C/C++ debugging view of the mixed language debugging session.

Apple M1/M2 Debugging Support

TotalView 2023.3 adds platform support for Apple’s ARM based M1 and M2 architectures.  Only debugging with TotalView’s new UI and CLI are supported.  Both macOS Monterey and Ventura are supported on M1/M2 based systems.

Preliminary CUDA 12 Support

TotalView 2023.3 adds preliminary support for CUDA 12.  See the CUDA specific platform notes later in this document for full details and recommended configurations to successfully debug on CUDA 12 systems with TotalView.

Bug Fixes and Performance Improvements 

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations.

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

RELEASE NOTES  LATEST RELEASE

TotalView 2023.1

TotalView Performance Enhancements

For TotalView 2023.1, user interface updates and debug event processing have been optimized, resulting in faster debugging session launches, especially for large scale parallel jobs.  These changes will also improve user interface updates during debugging operations, such as when quickly stepping through multiple lines of source code.

CUDA 11.8 Support

TotalView 2023.1 adds support for debugging applications using CUDA 11.8 SDK using the legacy CUDA debugger backend.  CUDA 11.8 introduced a new unified CUDA debugger backend, but several serious problems were identified while testing TotalView against the new implementation. The only viable workaround is to force CUDA to fall back to the legacy CUDA debug implementation by setting the CUDBG_USE_LEGACY_DEBUGGER environment variable to 1, e.g., export CUDBG_USE_LEGACY_DEBUGGER=1. The CUDA 12.0 r525 driver release should address the issues so that the new unified CUDA debugger backend will work correctly with TotalView. Note, because every debugger process andapplication process in the debug session must use the same setting, we recommend that the environment variable be set in your shell startup file (e.g., .cshrc or .bashrc), which is the simplest way to satisfy this requirement.

TotalView AMD GPU ROCm Initialization and ROCm Support Update

For the 2023.1 release, TotalView automatically discovers if the target program is utilizing the ROCm toolkit for running code on AMD GPUs.  As a result, developers no longer need to add the -rocm command line switch to turn on AMD GPU debugging.  This version of TotalView also supports ROCm 5.4.

Memory Debugging Painting

Finding memory access problems for uninitialized memory or when memory has been freed is a real challenge often resulting in a lot of time debugging sporadic data issues and crashes.  To make it more predictable in finding these issues, TotalView 2023.1 adds the ability to turn on memory painting for newly allocated memory blocks and freed memory blocks.  In either case, a hexadecimal paint pattern is written into the memory block giving you a consistent memory value for cases when your program tries to read from the memory blocks.  Turn on painting by simply checking the option in the Session dialog from the Start Page.

Optionally, enable the option within the memory options dialog from the memory debugging toolbar.

TotalView Working Directory

Users can now specify a working directory where TotalView will executing your target program. If not provided, the default is the directory from which you invoked TotalView. 

This value can be entered into the UI or on the command line when starting TotalView. It can then be modified during a debug session using the Process > Modify Arguments menu or simply as part of setting up a new debugging session.         

Specify the working directory from the command line using the new -working_directory command line argument:

totalview -working_directory /tmp 

Python 3.10 Mixed Language Debugging Support

TotalView 2023.1 adds support for Python 3.10 when performing mixed language C/C++ and Python debugging.

AIX 7.2 Support

TotalView 2023.1 adds support for AIX 7.2.  In our testing though we found that targets launched through a fork/exec on AIX 7.2 occasionally hang in low-level ptrace() system calls as TotalView tries to write to the target processes address space.  This has been traced to a kernel bug in the underlying AIX operating system.  This issue has been fixed by IBM and is will soon be available in a software fix from IBM.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations.

View Full Release Notes

For more details, please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

RELEASE NOTES  Get Latest Release

TotalView 2022.4

AMD GPU Debugging Support 

TotalView 2022.4 adds support for debugging AMD GPUs in addition to its ability to debug NVIDIA GPUs. TotalView supports versions 5.1 to 5.2 of the ROCm software stack for GPU programming, including support for debugging HIP (Heterogeneous Interface for Portability) and MPI code running on AMD MI50, MI100, and MI200 series of GPUs.  TotalView supports the following debugging features on AMD GPUs: 

• Process launch, attach, and detach 
• Viewing scalar, vector, general, and special AMD GPU registers 
• Instruction disassembly 
• Breakpoint creation and deletion on AMD GPU code 
• Single-stepping and fast smart-stepping 
• Stack unwinding, including inlined functions 
• Navigation controls for changing the logical workgroup / work-item focus or physical agent, queue, dispatch, wave, and lane focus 
• Variable display with the ROCm 5.1+ compilers 
• Data watchpoints on global memory variables 

Preparing your program for debugging is simple. Use the “-O0 -g” flags with the ROCm 5.1+ compilers to generate the necessary debug symbols for TotalView. 

To debug a program on AMD GPUs, use the “-rocm” flag when you start TotalView, for example: 

totalview -rocm a.out

Once started, simply set breakpoints in the HIP kernel code by selecting the line number where you would like to stop, then begin running your program.  TotalView will automatically plant breakpoints in the HIP code once it is loaded onto the GPU.  Use TotalView’s stepping commands to control execution, and dive on variables to view their values in your GPU code.  Reverse debugging will not be supported for code running on the AMD GPUs but is planned to work with reverse debug code running on the CPU in a combined CPU/GPU debugging session.  There currently is an issue while reverse debugging when the -rocm switch is used and debugging on the GPU.  We are working to resolve this issue for an upcoming release of TotalView. 

To learn more about specific AMD GPU debugging characteristics, see help for the drocm command in TotalView’s Command Line Interface (dhelp drocm). 

Array View for Array Debugging and Visualization 

A new Array View is available in TotalView 2022.4 which enables focused debugging of array data.  Through the Array View, you can examine statistical information about an array and visualize array data. 

To bring up the Array View, either activate it from the Window > Views > Array View menu item or select Add to Array View from the context menu in the Local Variables view or Data View when selecting an array variable. 

Figure 1 - Add to Array View menu item 

The toolbar area of the Array View displays the process and thread for the array data being displayed.  The array expression is shown along with the slice of the array to debug.  In Figure 2, array expression floatArray[:] is shown and the [:] specifies to debug the whole array.  A portion of the array can be specified by providing the beginning and ending indices, such as [10:15]. 

The Array View displays array statistics information initially.  In previous versions of TotalView this was separate functionality available in the Array Statistics View.   

Figure 2 - Array Statistics 

A histogram plot of the array data is shown when the bar plot icon is clicked in the toolbar.  Adjust the number of bins used in the plot using the field in the plot. 

Figure 3 - Visualize Array in Histogram Plot 

A line plot of the array data is also available when the line plot icon is selected in the toolbar. 

Figure 4 - Visualize Array in Line Plot 

Watch for more array debugging capabilities to be added to the Array View in coming versions of TotalView! 

TotalView Startup Performance Enhancements 

For TotalView 2022.4, further startup performance improvements optimize how the debugger looks up symbols and entry point values in the runtime linker.  This can have a big impact on debugger startup performance for applications built with many shared libraries. 

DEBUGGING TIP:  A further startup performance optimization customers should consider using is to instruct TotalView to defer processing of shared libraries until later in the debug session.  By adding the TotalView command line option -no_dlopen_always_recalculate the debugger will delay processing shared library symbols until they are needed later in the debug session.  Learn more about tuning TotalView’s parallel debugging performance in the “Scalability in HPC Computing Environments” section of the TotalView User Guide, Part III Parallel Debugging chapter. 

TotalView Base Compilation System and Dependency Changes 

TotalView 2022.4 changes the base compilation for the following systems: 

Linux x86-64 

• Base build system was changed from CentOS 7.0 to CentOS 7.9, resulting in a base OS of CentOS/RHEL 7.9 and above.  Ubuntu 18.04 and 20.04 are also supported.  
   
  Note that TotalView’s user interface requires have libxkbcommon and libXcomposite libraries, typically installed as part of a desktop graphical environment. 

Linux PowerLE 

• Base build system was changed from Ubuntu 14.04 to CentOS 7.5.  The new base supported system changes to CentOS/RHEL 7.5 and above.  Ubuntu 18.04 and 20.04 are also supported.  

Note that TotalView’s user interface requires have libxkbcommon and libXcomposite libraries, typically installed as part of a desktop graphical environment. 

macOS 

• Base build system was changed from macOS High Sierra to macOS Big Sur.  TotalView’s base compilation support stays the same with support for macOS Big Sur and Monterey. 

Bug Fixes and Performance Improvements 

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations. 

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Release Notes  Latest Release

TotalView 2022.3 

AMD GPU Debugging Preliminary Support 

TotalView 2022.3 expands its GPU debugging capabilities with support for debugging AMD GPUs, in addition to its ability to debug NVIDIA GPUs.  TotalView supports versions 5.1 and 5.2 of the ROCm software stack for GPU programming, including support for debugging HIP (Heterogeneous Interface for Portability) and MPI code running on AMD MI50, MI100, and MI200 series of GPUs.  With this release, TotalView supports the following debugging features on AMD GPUs: 

• Process launch, attach, and detach 
• Viewing scalar, vector, general, and special AMD GPU registers 
• Instruction disassembly 
• Breakpoint creation and deletion on AMD GPU code 
• Single-stepping and fast smart-stepping 
• Stack unwinding, including inlined functions 
• Navigation controls for changing the logical workgroup / work-item focus or physical agent, queue, dispatch, wave, and lane focus 
• Variable display with the ROCm 5.1+ compilers 
• Data watchpoints on global memory variables 

Preparing your program for debugging is simple. Use the “-O0 -g” flags with the ROCm 5.1+ compilers to generate the necessary debug symbols for TotalView. 

To debug a program on AMD GPUs, use the “-rocm” flag when you start TotalView, for example: 

totalview -rocm a.out 

Once started, simply set breakpoints in the HIP kernel code by selecting the line number where you would like to stop, then begin running your program.  TotalView will automatically plant breakpoints in the HIP code once it is loaded onto the GPU.  Use TotalView’s stepping commands to control execution, and dive on variables to view their values in your GPU code.  Reverse debugging will not be supported for code running on the AMD GPUs but is planned to work with reverse debug code running on the CPU in a combined CPU/GPU debugging session.  There currently is an issue while reverse debugging when the -rocm switch is used and debugging on the GPU.  We are working to resolve this issue for the next release of TotalView. 

To learn more about specific AMD GPU debugging characteristics, see help for the drocm command in TotalView’s Command Line Interface (dhelp drocm). 

Watch for more AMD GPU debugging features in our 2022.4 release in November, including watchpoint support!  If you are attending Supercomputing 2022 this year, make sure to attend the “Debugging the Toughest Challenges with NVIDIA and AMD GPUs” Exhibitor Forum talk for more details about debugging AMD and NVIDIA GPUs with TotalView. 

Dive Stack Expansion State 

TotalView 2022.3 expands on the Dive Stack functionality introduced in version 2022.2.  The Dive Stack enables developers to easily explore data by traversing/diving into data structures and back out.  Version 2022.3 adds functionality to maintain the expansion and scroll state before the dive so that an undive operation restores the Data View as it was. 

NVIDIA OpenACC Support 

TotalView 2022.3 adds official support for debugging NVIDIA OpenACC compiled applications. 

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been addressed, including performance improvements for data transformations. 

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Watch the What's New Webinar

Bill Burns, Senior Director of Product Engineering will show you all the new features. Watch now >>

 release notes   Get Latest Release

TotalView 2022.2

Data Debugging Workflow Improvements With Dive Stacks 

The latest release incorporates a popular feature from TotalView’s Classic UI, allowing users to navigate a Dive Stack of expressions when using data structures in Data View. Navigating up and down the stack allows you to move more easily through the data and streamlines workflows.

Fedora 35 and Ubuntu 22.04 Preliminary Support

There have been many Fedora 35 and Ubuntu 22.04 system level changes that impact debuggers. Our latest release offers enhancements and fixes to support debugging on these platforms. Testing for full support in the 2022.3 release is ongoing.

Solaris Base System Change and Query About System Support

Due to infrastructure changes, we’ve moved from Solaris 11u3 to 11u4.  We are interested in hearing if customers still actively use Solaris for development and debugging. Please send an email to Bill Burns, TotalView product manager, at bburns@perforce.com if Solaris is still an important platform for you.

TotalView CLI dprint Command Timeout Option

The latest release supports the ability to define a timeout via CLI dprint commands. Simply add “-timeout nSeconds” to define the number of maximum number of seconds the command should run.

TotalView's Security Updates

TotalView 2022.2 upgrades its use of zlib to the latest version to eliminate a CVE security vulnerability.

TotalView Startup Performance Improvements

TotalView’s processing of debug symbols during startup have resulted in a 42x performance improvement and reduced processing time from minutes to seconds, in one use case, as compared to the 2021.4 version. In cases where an executable or shared library contains thousands of subroutines, this new processing logic greatly improves efficiency.

Utilize Library Build ID's to Improve TotalView Processing Performance

TotalView 2022.2 incorporates advancements to processing of libraries in parallel jobs.  Checksums are used to find similar libraries loaded by parallel processes across a cluster of nodes to determine which libraries are identical. Checksumming can incur a performance penalty as each library is checked. TotalView 2022.2 attempts to extract the Build ID from shared library image files, thereby improving overall processing.

New User Interface Improvements

TotalView’s new user interface is available by default for new users and is activated through the Display Preferences panel or using the -newUI command line option for existing TotalView users.

Share Your Feedback

If you have any feedback about new features, wish to make requests for new or missing features, or want to report any problems please send an email to tv-beta@perforce.com.

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

View Release Notes   Get Latest Release

TotalView 2022.1

Memory Debugging Block Notify

A particularly challenging memory debugging situation occurs when a specific memory block is free but a program accesses the memory later through a dangling pointer reference. Because the memory may or may not still contain valid data, sporadic crashes and data problems can occur. TotalView 2022.1 adds the ability to watch a specific allocated memory block and raise an event when it gets freed or deleted, effectively mitigating this debugging challenge.

MacOS Monterey Support

TotalView 2022.1 adds support for Apple’s latest version of macOS 12 Monterey.

CUDA 11.4 and 11.5 Support

TotalView 2022.1 adds official debugging support for CUDA 11.4 and 11.5.

Bug Fixes and Performance Improvements

Numerous bug fixes and performance improvements have been made to TotalView.  See the release notes for the full list of bug fixes.

Share Your Feedback

If you have any feedback about new features, wish to make requests for new or missing features, or want to report any problems please send an email to tv-beta@perforce.com.

View Full Release Notes

For more details please refer to the TotalView release notes, where you can see more details and specs on how to enable various features, what to expect from updates, and info about any features on hold.

Watch the What's New Webinar

Bill Burns, Senior Director of Product Engineering will show you all the new features. Watch now >>

View Release NotesGet Latest Release