Working gdb config for Zephyr multi-thread aware debugging?

Good evening Jared,

Regarding thread aware gdb support for nRF9160, I would be thrilled to learn the change(s) I need make to the gdb server, and run everything command line. As my debugging tools stand today I’m just one step removed from this. At a command prompt I issue west -v debug, and by some Python and config magic west starts Segger’s JLinkGDBServer. Some options must be passed because the server knows to connect to an ARM Cortex-M33, and establishes a connection at once.

From this point I can examine a number of things statically, like variables, and the starting addresses of named functions. I have been able to run some of the monitor commands which generally affect run time entities of the target microcontroller and firmware. But though I am using a recent version of Segger gdb server, I am not seeing an equivalent list of firmware app threads that Zephyr reports. I see just two differently named threads which don’t match Zephyr’s default threads, nor my application threads.

For some reason I’m not able to connect to my targeted board when I invoke JLinkGDBServer directly.

CircuitDojo looks to be a large forum. Some other developers here must have need to debug Zephyr based applications, and so require thread awareness. I don’t know how one would get much debugging done without thread awareness, since Zephyr itself appears to set up at least four of its own threads by the time app main() starts to execute.

Any news you hear yes please let me know. I’ve already enabled a couple of recommended / required Kconfig symbols in prj.conf. I wonder whether there is a command line option which west is failing to pass to JLinkGDBServer?

• Ted

Hello @jaredwolff ,

I thought some here might be interested to hear a discovery I made regarding JLInk and remote debugging of Zephyr based applications. Reading through the messages generated when invoking west -v debug in an nRF9160 + Zephyr + ncs_v1.6.1 app I was able to extract the following two command line calls:

JLinkGDBServer -select usb -port 2331 -if swd -speed 4000 -device cortex-m33 -silent -singlerun -nogui -rtos /opt/SEGGER/JLink_V760g/GDBServer/RTOSPlugin_Zephyr

and

/opt/gcc-arm-none-eabi-10.3-2021.10/bin/arm-none-eabi-gdb /home/ted/projects/zephyr-based/aws-iot-stand-alone/build/zephyr/zephyr.elf -ex 'target remote :2331' -ex 'monitor halt' -ex 'monitor reset' -ex load

When I run the first command in one terminal window, I see a typical successful start up of gdb server, ending with a message about “Connecting . . .”. In a second Linux terminal I can then invoke the second command, and this brings me to a full remote debugging session with the app running on a Sparkfun Thing Plus nRF9160 (or similar nRF9160 based) board.

At this time I am still not seeing my expected listing of app threads, when at gdb’s prompt I issue thread apply all bt. The fact however that I now have insight into all the options passed to both the gdb server and gdb client, I am more confident that the server is running with the Zephry RTOS plug-in from Segger.

My more refined question now is: when connected to and debugging a remote Zephyr application in gdb, what command steps are needed to put the remote application in a state where gdb correctly shows Zephyr and the app’s threads?

One thing I observe is that as gdb start up it halts the firmware app. I can reset the firmware’s processor and restart it with the two gdb commands:

(gdb) monitor reset
(gdb) monitor go

I’ve even been able to “pause” the Zephyr app by alternately issuing via gdb it’s mon halt, mon go commands. But whether halted or running, gdb reports only the threads:

(gdb) thread find a
Thread 1 has target id 'Thread 57005'
Thread 2 has target id 'Remote target'

whereas Zephyr’s thread analyzer API reports from the firmware itself:

1) 'at_cmd_socket_thread' stack size 1472 bytes, stack used 432 bytes, 29%
2) 'thread_simple_cli' stack size 4096 bytes, stack used 1768 bytes, 43%
3) 'time_thread' stack size 1024 bytes, stack used 240 bytes, 23%
4) 'sysworkq' stack size 2048 bytes, stack used 168 bytes, 8%
5) 'idle 00' stack size 320 bytes, stack used 56 bytes, 17%

Am I misunderstanding how gdb is able to interact with, or supposed to interact with a multi-threaded application?

• Ted

An update to my search for Zephyr thread aware debugging at the command line: the following Devzone post includes detailed instruction from Nordic Semi engineer Håkon, and these instructions are the first to allow me debugging view of the correct expected threads running in my Zephyr + ncs 1.6.1 based app:

• https://devzone.nordicsemi.com/f/nordic-q-a/86265/gdb-commands-to-list-zephyr-app-threads

While I still have gdb learning curve to scale this development is exciting. I was not able to get the Zephyr RTOS plug-in for VSCode configured to produce useful results. The command line is simpler to document and reproduce. There really wasn’t any configuration. I just needed to download Segger’s JLink utilities for 64-bit Linux, which I did a few weeks ago. Download file at that time was named JLink_Linux_V760g_x86_64.deb.
As I make further progress debugging I’ll amend this post.

• Ted