Debug nRF9151Feather

cptel

is there a video/document on how to use RPi debug probe to debug nRF9151 Feather board?

cptel

I’ll try this out first: https://docs.circuitdojo.com/nrf9160-feather/programming-and-debugging.html#debugging-in-visual-code

cptel

The above instructions require j-link and I do not see a place to connect J-Link.

Look forward to instructions/video/document on how to debug nRF9151 Feather

jaredwolff

@cptel there’s no video or content on debugging using the onboard RP2040. I’ll see what I can do this week.

cptel

reason for requesting debug info is because I am running into file-system issue. And, it is always good to have debug capability

I am running into SPI stability issue when writing to the external Flash. In my app, after I create x (anywhere from 10 to 199) files, it gives me a USAGE FAULT… all the while the application ran through the same code numerous times. The Faulting instruction address points to:
0x000000002000cbd0 0x78 zephyr/drivers/spi/libdrivers__spi.a(spi_nrfx_spim.c.obj)
I know I am not out of memory based on this info after a reset:
[00:00:03.858,123] <inf> PWL: bsize = 16 ; frsize = 4096 ; blocks = 4096 ; bfree = 4075

Usage Fault error:

[02:37:54.849,243] <inf> PWL: MO Build path: /lfs/mo/mo0103
[02:37:58.065,032] <err> os: ***** USAGE FAULT *****
[02:37:58.070,678] <err> os:   Stack overflow (context area not valid)
[02:37:58.077,972] <err> os: r0/a1:  0x20010cc0  r1/a2:  0x00000001  r2/a3:  0x00020b2a
[02:37:58.086,700] <err> os: r3/a4:  0x81003800 r12/ip:  0x0fc232c3 r14/lr:  0x00000001
[02:37:58.095,428] <err> os:  xpsr:  0x00037000
[02:37:58.100,708] <err> os: s[ 0]:  0x00000000  s[ 1]:  0x00037c00  s[ 2]:  0x20010cc8  s[ 3]:  0x00020747
[02:37:58.111,206] <err> os: s[ 4]:  0x20010cc0  s[ 5]:  0x00000001  s[ 6]:  0x00000000  s[ 7]:  0x00000000
[02:37:58.121,704] <err> os: s[ 8]:  0x2000cbd0  s[ 9]:  0x00000000  s[10]:  0x00037088  s[11]:  0x2000e8a8
[02:37:58.132,202] <err> os: s[12]:  0x00037b14  s[13]:  0x00020b0f  s[14]:  0x2000f174  s[15]:  0x00000000
[02:37:58.142,669] <err> os: fpscr:  0x00000001
[02:37:58.147,918] <err> os: Faulting instruction address (r15/pc): 0x2000cbd0
[02:37:58.155,883] <err> os: >>> ZEPHYR FATAL ERROR 2: Stack overflow on CPU 0
[02:37:58.163,848] <err> os: Current thread: 0x2000d1f0 (unknown)
[02:37:58.170,654] <err> os: Halting system

After I reset the board, it hangs at fs_open when I am trying to create a new file.

jaredwolff

The most important clue is that Stack overflow text. That means whatever thread you’re doing the work it is not sufficient in size (or you’re using all the allocated memory accidentally)

You can run addr2line to see where in the code it’s being caused

~/.zephyrtools/toolchain/zephyr-sdk-0.16.4/arm-zephyr-eabi/bin/arm-zephyr-eabi-addr2line -e build/zephyr/zephyr.elf 0x2000cbd0

This will help find the offending instruction which is causing the overflow. Note, make sure you point to the correct .elf file.

I would start with using the 0x2000cbd0 address. Then sometimes I’ll drill in and use the r14 address if it’s useful. In your case 0x00000001 is not very useful.

It’s likely a custom thread that you’re running as indicated by [02:37:58.163,848] <err> os: Current thread: 0x2000d1f0 (unknown)

You can also turn on CONFIG_THREAD_NAME=y which will show in plain text which thread is borking.

cptel

and sometimes it just hangs and doesn’t go anywhere… i think its stuck in an OS while(1) loop somewhere…

[01:41:24.342,346] <inf> PWL: Message: 00.02.2E.00.01.10.30.35.44.39.44.38.39.44.46.45.30.43.31.41.41.34.02.01.02.03.08.FF.E7.30.2A.99.01.00.00.04.02.20.14.06.01.1B.
[01:41:24.357,818] <inf> PWL: f2 fs_write: bytes written - 42 / 42
[01:41:24.364,471] <inf> PWL: f1 fs_write: err - 0: 8

Normal operation is as below…

[01:39:03.269,134] <inf> PWL: Message: 00.02.2D.00.01.10.30.35.44.39.44.38.39.44.46.45.30.43.31.41.41.34.02.01.02.03.08.EE.C0.2E.2A.99.01.00.00.04.02.20.14.06.01.1B.
[01:39:03.284,637] <inf> PWL: f2 fs_write: bytes written - 42 / 42
[01:39:03.291,290] <inf> PWL: f1 fs_write: err - 0: 8
[01:39:03.297,729] <inf> PWL: f1 fs_close: err - 0: 0
[01:39:03.304,626] <inf> PWL: f2 fs_close: err - 0: 0
[01:39:03.310,119] <inf> PWL: MMOB() - EVENT: 19

Related question - how many files can be open at any given time? I have 2 files open and being operated on.

jaredwolff

I think you can open several files at once, you need to allocate the right amount of memory to do that though. By default with LittleFS it’s 4.

config FS_LITTLEFS_NUM_FILES
	int "Maximum number of opened files"
	default 4
	help
	  This is a global maximum across all mounted littlefs filesystems.

Additionally I would highly recommend make sure that you’re only accessing the filesystem from one thread. Same goes for accessing peripherals (I2C ,etc)

cptel

yes, I am accessing files from one thread only. Since everything is driven through a message queue, there are no possibilities of same file being opened twice by a re-entrant condition.

I am also confident that stack & heap memory allocated is adequate.

Look forward to debug info 🙂

jaredwolff

I’ll be keeping the up-to-date version here: https://docs.circuitdojo.com/nrf9151-feather/debugging.html

Debugging in VSCode

Debugging the nRF9151 uses probe-rs as the gdb server. You can easily step through your application in a few steps.

Install Debugger for probe-rs via the VSCode Marketplace.

Debugger Extension

The extension helps facilitate the integration between probe-rs and the built-in debugging facilities of VSCode.

Add a .vscode/launch.json file. Here’s an example which is using Circuit Dojo Zephyr Tools:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "nRF9151 • App (Non-Secure)",
            "type": "probe-rs-debug",
            "request": "attach",
            "chip": "nRF9151_xxAA",
            "cwd": "${workspaceFolder}",
            "speed": 4000,
            "coreConfigs": [
                {
                    "coreIndex": 0,
                    "programBinary": "${workspaceFolder}/app/app/build/circuitdojo_feather_nrf9151/app/zephyr/zephyr.elf",
                }
            ],
            "consoleLogLevel": "Console"
        }
    ]
}

Program merged.hex as normal to your device. In Circuit Dojo Zephyr Tools, you can run the Flash option:

Flash button

This will flash and start running your code. At any point in the process you can view the console output (via the Monitor button).

Then, in the debug menu you should see the new launch option that you created:

Launch Debug

If the connection is successful, you will see a few indications that debugging is running:

Debugging working

This includes a Running indication in the left debug column. You should also see a red disconnect button on the top next to the debugger controls.

From here you can set halt points as normal and debug your firmware!

cptel

having challenges with Circuit Dojo initializing project:

open visual studio
click on Circuit Dojo and I see this screen:
I click on ‘Init Repo’ and select local folder + provide URL to my github project

my Terminal output captured here:

 *  Executing task: west init -m https://github.com/chirag-cptel-io/PWL --mf west.yml 

FATAL ERROR: already initialized in /Users/summer, aborting.
  Hint: if you do not want a workspace there, 
  remove this directory and re-run this command:

  /Users/summer/.west

 *  The terminal process "/bin/zsh '-l', '-c', 'west init -m https://github.com/chirag-cptel-io/PWL --mf west.yml'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: west update 

=== updating nrf (nrf):
M       samples/cellular/gnss/src/main.c
HEAD is now at 9eb5615da manifest: Tag v3.0.1
=== updating zephyr (zephyr):
HEAD is now at 77f865b8f8d [nrf fromlist] bluetooth: host: Fix param_len for LE CS Test command
=== updating bsim (tools/bsim):
HEAD is now at a88d335 main manifest: Update to v2.5
=== updating pcf85063a (pcf85063a):
HEAD is now at 81c1c28 APIs for Offset Register and CAP_SEL setting (#6)
=== updating hostap (modules/lib/hostap):
HEAD is now at 8cd83053a [nrf noup] zephyr: Disable WNM pre-scan check
=== updating wfa-qt-control-app (modules/lib/wfa-qt-control-app):
HEAD is now at af011c8 [nrf fromlist]: Add missing null checks for malloc calls
=== updating mcuboot (bootloader/mcuboot):
HEAD is now at ae1ee57f [nrf fromtree] zephyr: Disable SPI_NOR by default for nrf7002dk/nrf5340/cpuapp
=== updating qcbor (modules/tee/tf-m/qcbor):
HEAD is now at 751d365 restore inc/qcbor.h for backwards compatibility (#181)
=== updating mbedtls (modules/crypto/mbedtls):
HEAD is now at 44ce293d1 [nrf noup]: Allow for using legacy when MBEDTLS_FORCE_LEGACY_MD/CIPHER
=== updating oberon-psa-crypto (modules/crypto/oberon-psa-crypto):
HEAD is now at de2e3e0 [nrf toup] fix unused variable warning
=== updating nrfxlib (nrfxlib):
HEAD is now at f953fbcc0 softdevice_controller: rev 899a508a95019c58fc39d2c11004ee0264ce25be
=== updating trusted-firmware-m (modules/tee/tf-m/trusted-firmware-m):
HEAD is now at 17765832a [nrf noup] Remove version mismatch CMake warning
=== updating psa-arch-tests (modules/tee/tf-m/psa-arch-tests):
HEAD is now at 3da9313 [nrf fromtree] Add support for the nrf54l15dk
=== updating cjson (modules/lib/cjson):
HEAD is now at c6af068 [nrf fromtree] fix: potential memory leak in merge_patch() (#611)
=== updating azure-sdk-for-c (modules/lib/azure-sdk-for-c):
HEAD is now at 308c171c [nrf fromlist] Make repository a Zephyr module
=== updating cirrus (modules/hal/cirrus-logic):
HEAD is now at 3873a08 Added missing curly brace for #ifdef __cplusplus
=== updating suit-generator (modules/lib/suit-generator):
HEAD is now at d6afcd8 Add Error message when suit sequence number larger than uint32
=== updating suit-processor (modules/lib/suit-processor):
HEAD is now at 94effb7 Introduce compression info struct type.
=== updating memfault-firmware-sdk (modules/lib/memfault-firmware-sdk):
HEAD is now at 0bd360f Memfault Firmware SDK 1.18.0 (Build 11548)
=== updating coremark (modules/benchmark/coremark):
HEAD is now at d5fad6b Merge pull request #51 from volodymyr-bondarchuk/zephyr
=== updating canopennode (modules/lib/canopennode):
HEAD is now at dec12fa stack: map CANopenNode crc16_ccitt() to Zephyr crc16_itu_t()
=== updating chre (modules/lib/chre):
HEAD is now at 3b32c76e zephyr: link the .unstable_id data in ROM
=== updating nanopb (modules/lib/nanopb):
HEAD is now at 7307ce3 Update README.md (#1042)
=== updating tf-m-tests (modules/tee/tf-m/tf-m-tests):
HEAD is now at 502ea90 Merge pull request #11 from tomi-font/update_2.1.1
=== updating zscilib (modules/lib/zscilib):
HEAD is now at ee1b287 samples/tests: Update for hwmv2 board names
=== updating cmsis (modules/hal/cmsis):
HEAD is now at 4b96cbb doc: Update documentations for CMSIS 5.9.0
=== updating cmsis-dsp (modules/lib/cmsis-dsp):
HEAD is now at d80a49b2 do not generate warnings when scalar functions are used with MVE
=== updating cmsis-nn (modules/lib/cmsis-nn):
HEAD is now at ea987c1 manifest: zephyr module file
=== updating edtt (tools/edtt):
HEAD is now at b9ca3c7 Fix for for python >= 3.11
=== updating fatfs (modules/fs/fatfs):
HEAD is now at 427159b zephyr: fix ffconf.h override use of STRINGIZE
=== updating hal_nordic (modules/hal/nordic):
HEAD is now at 8cdcc33 nrfx: nrfx_twim: allow change and restore event_handler
=== updating hal_st (modules/hal/st):
HEAD is now at 05fd453 sensor/stmemsc: Align stmemsc i/f to v2.8
=== updating hal_tdk (modules/hal/tdk):
HEAD is now at e0ade95 Merge pull request #5 from tdk-invn-oss/main
=== updating hal_wurthelektronik (modules/hal/wurthelektronik):
HEAD is now at e3e2797 Remove preprocessor warnings for not enabling float
=== updating liblc3 (modules/lib/liblc3):
HEAD is now at bb85f7d math: Remove usage of Infinity that leads to undefined beahior using fast-math
=== updating libmetal (modules/hal/libmetal):
HEAD is now at 14f5195 lib: update libmetal to SHA 9a21915a5f8f
=== updating littlefs (modules/fs/littlefs):
HEAD is now at 009bcff Merge pull request #16 from de-nordic/update_to_v2.9.3
=== updating net-tools (tools/net-tools):
HEAD is now at 93acc8b net-setup: Add config for setting up multiple network interfaces
=== updating nrf_hw_models (modules/bsim_hw_models/nrf_hw_models):
HEAD is now at b84bd73 UART PTY backend: Correct a few comments
=== updating nrf_wifi (modules/lib/nrf_wifi):
HEAD is now at eeceee7 nrf_wifi: Use correct memory pool in HAL allocations
=== updating open-amp (modules/lib/open-amp):
HEAD is now at f7f4d08 lib: update open-amp lib  to SHA 9a21915a5f8f
=== updating percepio (modules/debug/percepio):
HEAD is now at 0d44033 Merge branch 'main' into zephyr
=== updating picolibc (modules/lib/picolibc):
HEAD is now at 82d62ed1a zephyr: Disable LTO when building the library
=== updating segger (modules/debug/segger):
HEAD is now at cf56b1d readme: Add readme
=== updating tinycrypt (modules/crypto/tinycrypt):
HEAD is now at 1012a3e Fix warnings reported by UBSAN
=== updating uoscore-uedhoc (modules/lib/uoscore-uedhoc):
HEAD is now at 84ef879 Merge pull request #4 from rlubos/upmerge-06-06-24
=== updating zcbor (modules/lib/zcbor):
HEAD is now at 9b07780 Update version to 0.9.1
 *  Terminal will be reused by tasks, press any key to close it. 

 *  Executing task: "/Users/summer/.zephyrtools/env/bin/python" -m pip install -r zephyr/scripts/requirements.txt 


[notice] A new release of pip is available: 25.0.1 -> 25.2
[notice] To update, run: python -m pip install --upgrade pip
ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'zephyr/scripts/requirements.txt'

 *  The terminal process "/bin/zsh '-l', '-c', '"/Users/summer/.zephyrtools/env/bin/python" -m pip install -r zephyr/scripts/requirements.txt'" terminated with exit code: 1. 
 *  Terminal will be reused by tasks, press any key to close it.

querying pip on my terminal, it gives me 25.2. A bit different than above.

pip install --upgrade pip                     
Requirement already satisfied: pip in /Library/Frameworks/Python.framework/Versions/3.13/lib/python3.13/site-packages (25.2)

jaredwolff

You don’t need the CD to debug. If you’ve programmed via Nordic’s extension that should be fine too. The .elf file will be in a similar spot.

More info on the probe-rs extension is here: https://probe.rs/docs/tools/debugger/

cptel

debugger works!
is static variable the best way to check for heap/stack overflow?
my current assumption is that the issue is related to fs_close() requiring additional time. Still chasing this issue…

jaredwolff

cptel debugger works!

Great

cptel is static variable the best way to check for heap/stack overflow?

I would try to figure out which thread it’s happening in. Then that would give a clue as to what operation you’re doing that is causing it.

cptel

I know the thread, function, location where it happens, and when it happens. The deterministic part varies based on build.
In my application, I log every message using file system to external flash. It is a circular buffer of 50 files, roughly 50 - 60 bytes message per file. Messages are generated in-frequently in the end solution but I am generating them once every 2 mins or faster to speed up testing.
In the most recent build, I would get the system-fault every 47th message whenever it tries to close a file. I would erase the external Flash completely and it would save up to message #47 and crash - very odd on the deterministic behavior. Once it crashes, upon reset, it would read the file as expected but would crash on first attempt to write the file.
I used the debugger to step through on the 47th attempt and it worked fine - hence assuming there’s a timing issue.
Crash occurs at 0×0001bcce. Looking at the attached stack info, that’s beginning of code (assuming)?

In the attached version, the crash didn’t occur as I was stepping through the code.