Debugging Python with Pybind and TotalView

February 22, 2021

Debugging Python in Mixed-Language Apps With Pybind and TotalView

Technologies & Languages

Python is one of the most widely used programming languages. It can be used to call other high level languages such as C, C++ and Fortran in order to provide access to high performance routines without having to rewrite existing code.

In this article we will see how to debug a mixed language Python/C++ and Python/C++/Fortran example using the Pybind framework and TotalView.

What Is Pybind?
Python/C++ Example Using Pybind
Python/C++/Fortran Example Using Pybind
Further Reading

What Is Pybind?

Pybind is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. - Source

Python/C++ Example Using Pybind

Requirements

This example uses Ubuntu 19 and Python 3.7.

sudo apt-get install python-dev
sudo apt-get install python-dbg
sudo apt-get install pybind11
pip install pybind

Step by Step

1. Start with an example.cpp file.

#include <pybind11/pybind11.h>
int add(int i, int j) {
    return i + j;
}
PYBIND11_MODULE(example, m) {
    m.doc() = "pybind11 example plugin"; // optional module docstring
    m.def("add", &add, "A function which adds two numbers");
}

2. Pybind11 is a header-only library, hence it is not necessary to link against any special libraries and there are no intermediate (magic) translation steps. On Linux, the above example can be compiled using the following command:

g++ -g -O3 -Wall -shared -std=c++11 -fPIC -I/usr/include/python3.7dm example.cpp -o example$(python3.7dm-config --extension-suffix)

This creates a shared library named (similar to) example.cpython-37dm-x86_64-linux-gnu.so.

3. Test the Python module by importing example and calling the add function.

4. Write a Python test wrapper test_example.py

import example
result = example.add(1,2)
print("Result = ", result)

5. Start TotalView debugger to debug the Python / C++ code:

$totalview -args python3-dbg test_wrapper.py

6. Set a pending breakpoint on the C++ add function.

7. TotalView shows the C++ code when you press GO.

Enable stack frame filtering by clicking on the orange filter button in the Call Stack window.

8. And the Python code.

Python/C++/Fortran Example Using Pybind

This uses the same requirements noted in the above example.

Step by Step

1. Start with a Fortran module fortmodule.f90 that prints “Hello from Fortran!” and assigns integer values to an array

! fortmodule.f90

module fortmodule

use iso_c_binding

implicit none

integer(C_INT), bind(C), dimension(5) :: numbers

contains

subroutine fortransub() bind(C)
   print *, "Hello from Fortran!"
   numbers(1) = 1
   numbers(2) = 2
   numbers(3) = 3
   numbers(4) = 4
   numbers(5) = 5
end subroutine

end module

2. Compile the Fortran module to a shared library:

gfortran -g -fPIC -c fortmodule.f90

3. Create a CPP file TestFortran.cpp which calls the Fortran module fortransub() subroutine and contains a Pybind reference (TestFortran).

// TestFortran.cpp
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <stdio.h>

extern "C" void fortransub();
extern "C" int numbers[5];

int makeMe()
{
   int i;
   printf("Hello from C!\n");
   fortransub();
   for (i=0; i<5; i=i+1)
      printf("%d ", numbers[i]);
   printf("\n");

return 0;
}


PYBIND11_MODULE(TestFortran, m) {
    
    m.def("makeMe", &makeMe, "Test fortran/C/python");
}

4. Compile the TestFortran.cpp code into a shared library linking the fortmodule.o library:

g++ -g -O3 -Wall -shared -std=c++11 -fPIC -I/usr/include/python3.7dm TestFortran.cpp fortmodule.o -lgfortran -o TestFortran$(python3.7dm-config --extension-suffix)

The output should be a shared library names similar to TestPython.cypthon-37dm-x86_64-linux-gnu.so.

5. Test the Python module.