September 21, 2022

Debugging Embedded Techniques in C/C++ Applications with TotalView

Debugging Best Practices
Back to top

What is an Embedded System?

An embedded system is a computer system—a combination of a computer processor, computer memory, and input/output peripheral devices that has a dedicated function within a larger mechanical or electronic system. It is embedded as part of a complete device often including electrical or electronic hardware and mechanical parts (Embedded System - Wikipedia).

Embedded systems present their own particular set of challenges for developers who need to debug applications running on them. Often embedded systems may not have a graphical user interface, or a full set of system libraries available, instead running a lightweight or stripped-down OS, which can make debugging challenging.

Back to top

About TotalView

TotalView is a high performance debugging tool for Linux, UNIX, and macOS platforms. It provides debugging support for multi-threaded and multi-process applications. TotalView can be used from the user interface or from the command line interface, and supports local as well as remote debugging.

Back to top

How TotalView Helps with Embedded Application Debugging

TotalView has several features that can help when debugging C/C++ applications running on embedded systems. These include:

  • Support for Remote Debugging
  • Advanced Thread Control
  • Mixed language debugging support
  • Command Line Interface
  • Scripting capabilities
  • Multiple platform support

We will look at each of these features in turn and examine how they assist in debugging applications in an embedded system environment.

Back to top

Remote Debugging

TotalView’s Remote User Interface (UI) feature allows you to run the debugger from your local system and efficiently and securely conduct your debugging on a remote system. For example, you could run the TotalView UI on your local Linux or macOS workstation or laptop and debug a remote application running on an ARM server. Currently the TotalView UI can be run natively on Linux and macOS, and a Windows version is due to be released soon.

Debugging Embedded TV Blog 1

For more information on remote debugging see the blog post Guide To Remote Debugging and Development

Back to top

Advanced Thread Control

TotalView has advanced support for debugging threads, allowing you to step individual threads or groups of threads, and to set thread-level breakpoints. You can also examine thread specific data and easily compare values across threads.

This provides extremely fine grained control over the application being debugged.

More information about debugging multi-threaded application is available in this blog post 

threading and Multithreaded Applications

Back to top

Mixed Language Debugging

TotalView has support for debugging mixed language C or C++ applications with Python. TotalView allows you to view the C/C++ and Python variables, examine the data exchange between the language barriers, and to debug your C/C++ code.

Currently TotalView “understands” three Python frameworks: SWIG, Ctypes,  and Pybind. But, it will work just fine with other frameworks that facilitate calling from Python to C/C++ code.

For more information on mixed language debugging see the relevant Python blog posts

How to Debug Python With SWIG

How to Debug Python in Mixed-Language Applications With Ctypes and TotalView

Debugging Python in Mixed-Language Apps With Pybind and TotalView

Back to top

Command Line Interface

TotalView has a full featured user interface, but there are occasions when the remote system may not have the required libraries to support a graphical user interface. TotalView’s Command Line Interface (CLI) presents an alternative that can be used in these circumstances to debug an application. The CLI can be launched by using the command totalviewcli and contains a comprehensive set of debugging commands.

Debugging Techniques in C & C++ Application With TotalView 2

More information about using the CLI is available in the Reference Guide

PART I Using the CLI

Back to top

Scripting Capabilities

TotalView has a powerful scripting engine called tvscript which allows you to create your own scripts to run a debugging session in batch or unattended mode. This is particularly useful if you have a long running program, or want to include debugging as part of your CI/CD processes, for example as a Jenkins job. You could for instance create a tvscript that checks for memory leaks whenever code is checked in.

More information on using tvscript is available in the Reference Guide

Batch Debugging Using tvscript

Using TotalView to test for memory leaks with Jenkins is described in this blog post

How to Test for Memory Leaks in Jenkins

Back to top

Multiple Platform Support

As well as Linux on Intel, AMD and ARM platforms, TotalView also supports GPU debugging on both NVIDIA and AMD hardware.

The following processors are supported :

  1. ARM (aarch64)
  2. Intel/AMD (x86-64, x86)
  3. NVIDIA GPUs on Linux x86-64, Linux ARM64, and Linux PowerLE (Power9)
  4. NVIDIA Jetson AGX Xavier
  5. OpenPOWER (ppc64le)
  6. Power (ppc64, ppc32)
  7. SPARC

The following operating systems are supported:

  • Linux including Cray Linux Environment (CLE) and PowerLE
  • UNIX including AIX and Solaris
  • macOS 

The full platform support matrix is available here TotalView_Platforms_Guide

Back to top

Putting It All Together

Here we look at the steps required to establish a remote connection with TotalView from a Mac or Linux machine to an NVIDIA Jetson AGX Xavier and debug a CUDA application on the Jetson machine.

Jetson AGX Xavier offers amazing compute capabilities at the fraction of the size and cost of a workstation, making it a perfect fit for a variety of autonomous machine applications.

The same version of TotalView must be installed on both the local (MacOS or Linux) machine as well as the remote (Jetson) machine in order to use remote debugging.

  1. Create a new Remote UI connection for the Jetson machine
Debugging Embedded TV Blog
  1. Select the jetson-avx connection from the Launch Remote Debugger drop down list and enter the required password at the command prompt
Debugging Embedded TV Blog 2
  1. Once a connection has been established you can now debug code running on the ARM CPU or debug CUDA code running on the NVIDIA GPU
  2. Start debugging a CUDA application by selecting the Debug a Program option and choosing a CUDA application. The example shown below is the CUDA matrix multiply example
Debugging Embedded TV Blog 4
  1. Debugging other embedded applications follows a similar workflow to that outlined above
Back to top

Next Steps & Additional Resources

In this blog you learned about TotalView's debugging capabilities and steps required to debug a CUDA application. Try TotalView for free and test our debugging capabilities for yourself, today.


For additional reading, here is a blog on everything you need to know on embedded system software development. 

Read Blog

Back to top