Bricked nrf9160? Plus not finding led0...

mikaelj · Mar 1, 2024

Hello,

I finally got a working setup for building out-of-tree using ncs v2.5.1, until I actually started running the Blinky sample. It’s unmodified from the official Zephyr tree. Note: the Blinky sample did not have CONFIG_BOOTLOADER_MCUBOOT set.

ZEPHYR_BASE=/path/to/ncs/v2.5.1/zephyr
west build -b circuitdojo_feather_nrf9160_ns -p

No problems there. Transferring using cortex-debug targetting a BlackMagicProbe and the TC connector.

Less important. The first few times, I got it to work, but the global “led” variable was set to all 0xFF (port, pin), as if the dts alias wasn’t found. So it failed in gpio_pin_configure_dt() because the “led” struct was invalid. build/zephyr.dts had an alias led0. VSCode show squiggly lines about identifier “__device_dts_ord_7” is undefined.
What could be wrong?

More importantly. After a few times, the debugger said “warning: ARM M in lockup state, stack unwinding terminated”, and the green led keeps on blinking. Even an empty int main() { return 0; } gives the same result. I’ve tried power cycling everything to no avail.

After this, no applications work :-/ After applying USB power, I only get the high-frequency green blinking.
Is it bricked? What can I do?

I also have a GDB session which confirms it’s stuck in internal boot code or similar:

/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb
GNU gdb (Zephyr SDK 0.16.1) 12.1
(gdb) target extended-remote /dev/cu.usbmodem97B742171
Remote debugging using /dev/cu.usbmodem97B742171
(gdb) monitor swdp_scan
Target voltage: 3.3V
Available Targets:
No. Att Driver
1 Nordic nRF9160 M33
(gdb) attach 1
Attaching to Remote target
warning: No executable has been specified and target does not support
determining executable automatically. Try using the “file” command.
0xeffffffe in ?? ()
(gdb) file build/zephyr/zephyr.elf
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from build/zephyr/zephyr.elf…
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xeffffffe in ?? ()
(gdb)

Thanks in advance,
Mikael

jaredwolff · Mar 1, 2024

Are you building your samples with mcuboot enabled?

# Enable Zephyr application to be booted by MCUboot
CONFIG_BOOTLOADER_MCUBOOT=y

It seems like you may have overwritten the bootloder. You may also want to check and make sure that the BMP supports the nRF9160’s architecture. I have one here but do not use it. I recommend a nRF53DK for programming and debugging since it’s well supported by Nordic.

Hope that helps!

mikaelj · Mar 1, 2024

Tried using “launch” instead of “attach” for cortex-debug, and got some additional information. Not really helpful either, though.

-> ~“ received signal SIGSEGV, Segmentation fault.\n”
received signal SIGSEGV, Segmentation fault.
-> ~“0×00009028 in hard_fault (esf=0×3339, recoverable=0×200123c0) at /opt/nordic/ncs/v2.5.1/zephyr/arch/arm/core/aarch32/cortex_m/fault.c:798\n”
0×00009028 in hard_fault (esf=0×3339, recoverable=0×200123c0) at /opt/nordic/ncs/v2.5.1/zephyr/arch/arm/core/aarch32/cortex_m/fault.c:798
-> ~“798\t\t\t\treason = esf->basic.r0;\n”
798 reason = esf->basic.r0;
-> *stopped,reason=“signal-received”,signal-name=“SIGSEGV”,signal-meaning=“Segmentation fault”,frame={addr=“0×00009028”,func=“hard_fault”,args=[{name=“esf”,value=“0×3339”},{name=“recoverable”,value=“0×200123c0”}],file=“/opt/nordic/ncs/v2.5.1/zephyr/arch/arm/core/aarch32/cortex_m/fault.c”,fullname=“/opt/nordic/ncs/v2.5.1/zephyr/arch/arm/core/aarch32/cortex_m/fault.c”,line=“798”,arch=“armv8-m.main”},thread-id=“1”,stopped-threads=“all”

reason=422369556

mikaelj · Mar 1, 2024

jaredwolff

The BMP supports the nRF9160. I have used it with the nRF9160 Feather. It’s set to flash build/zephyr/zephyr.elf.

Yes, I built a sample from plain Zephyr without enabling MCUboot support first. That probably overwrote the bootloader.

I’m fine with using only the BMP for development, without the bootloader. From my understanding, if I don’t use the USB loading (i.e. app_update.bin), and instead flash zephyr.elf, there’s no need to have a bootloader.

What is the next step for making the nRF9160 Feather work again - bootloader or not. Just need to continue working.
I’d rather not buy a new Feather nor a nrf53DK. (They’re both rather expensive, and they take a very long time to ship.)
Especially if it’s a software problem.

Please help.
Mikael

jaredwolff · Mar 1, 2024

mikaelj Yes, I built a sample from plain Zephyr without enabling MCUboot support first. That probably overwrote the bootloader.

You need to use NCS not plain vanilla Zephyr.

mikaelj What is the next step for making the nRF9160 Feather work again - bootloader or not. Just need to continue working.

Understood. I can’t support use of the BMP but if you have another programmer like a JLinkMini you can write one of the precompiled hex files to the device. Since I’m not sure what you did to the device it’s also possible you may have corrupted the flash for the MFW core. You’d need a JLink to re-flash that firmware as well.

I’d rather not buy a new Feather nor a nrf53DK. (They’re both rather expensive, and they take a very long time to ship.)
Especially if it’s a software problem.

I’m surprised about this comment. I ship same day usually. Where did you purchase your board and where are you located?

jaredwolff · Mar 1, 2024

Latest pre-compiled binaries/hex files are here: https://github.com/circuitdojo/nrf9160-feather-examples-and-drivers/actions/runs/7269640835/artifacts/1125505217 under Artifacts. The latest is nfed_v2.4.x_7dfaec1

mikaelj · Mar 1, 2024

I’m surprised about this comment. I ship same day usually. Where did you purchase your board and where are you located?

Sweden, ordered from DigiKey. Which perhaps is shipped from you? Don’t know - they act as a marketplace, AFAICT.

You need to use NCS not plain vanilla Zephyr.

I’m only referring to the sample project. In this case, the blinky sample is identical in NCS-Zephyr and vanilla-Zephyr - the only difference to be found is in the nrf9160-examples-and-drivers, where (as you’ve pointed out) CONFIG_BOOTLOADER_MCUBOOT=y, which I now have learned the hard way. :-)

JLink/JLinkMini

Understood - thanks. I’ll dig into the BMP some more and see if I can use it to save my precious board.

Precompiled binaries

Oh, convenient! I’m guessing this is the code/data fw file put into the Feather in the production rig, i.e. it’ll be like performing a “factory restore”?

Thanks for the quick replies!
Mikael

jaredwolff · Mar 1, 2024

mikaelj Sweden, ordered from DigiKey. Which perhaps is shipped from you? Don’t know - they act as a marketplace, AFAICT.

If you did it recently, it’s directly fulfilled by them. I know it can take a bit longer to get to Sweeden. Sorry about that delay!

mikaelj Oh, convenient! I’m guessing this is the code/data fw file put into the Feather in the production rig, i.e. it’ll be like performing a “factory restore”?

That one isn’t in there but these are all the precompiled samples that you can load and use immediately. It can also help confirm if you have issues with your SDK/toolchain.

mikaelj · Mar 2, 2024

Hey, the board is alive again - thanks! Now the problem is my toolchain…

The merged hex file from nfed_v2 restores the board. Building my own, however, is no-work. A fault is happening from what I can tell from the map file, but I have no clue on why/what/where. The output from “west build” looks perfectly normal.

NCS v2.5.1. Same results with NCS v2.4.0.

blinky sample:

prj.conf

CONFIG_GPIO=y
CONFIG_NO_OPTIMIZATIONS=y
CONFIG_BOOTLOADER_MCUBOOT=y

west build

$ rm -rf build
$ west build -b circuitdojo_feather_nrf9160_ns

-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky
-- CMake version: 3.28.3
-- Found Python3: /opt/homebrew/opt/python@3.11/bin/python3.11 (found suitable version "3.11.8", minimum required is "3.8") found components: Interpreter 
-- Cache files will be written to: /Users/mikaelj/Library/Caches/zephyr
-- Zephyr version: 3.4.99 (/opt/nordic/ncs/v2.5.1/zephyr)
-- Found west (found suitable version "1.2.0", minimum required is "0.14.0")
-- Board: circuitdojo_feather_nrf9160_ns
-- Found host-tools: zephyr 0.16.1 (/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk)
-- Found toolchain: zephyr 0.16.1 (/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk)
-- Found Dtc: /opt/homebrew/bin/dtc (found suitable version "1.7.0", minimum required is "1.4.6") 
-- Found BOARD.dts: /opt/nordic/ncs/v2.5.1/zephyr/boards/arm/circuitdojo_feather_nrf9160/circuitdojo_feather_nrf9160_ns.dts
-- Generated zephyr.dts: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/zephyr.dts
-- Generated devicetree_generated.h: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/include/generated/devicetree_generated.h
-- Including generated dts.cmake file: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/dts.cmake
Parsing /opt/nordic/ncs/v2.5.1/zephyr/Kconfig
Loaded configuration '/opt/nordic/ncs/v2.5.1/zephyr/boards/arm/circuitdojo_feather_nrf9160/circuitdojo_feather_nrf9160_ns_defconfig'
Merged configuration '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/prj.conf'
Configuration saved to '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/.config'
Kconfig header saved to '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/include/generated/autoconf.h'
-- Found GnuLd: /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd (found version "2.38") 
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc
-- Found Python3: /opt/homebrew/opt/python@3.11/bin/python3.11 (found version "3.11.8") found components: Interpreter 
Changed board to secure circuitdojo_feather_nrf9160 (NOT NS)

=== child image mcuboot -  begin ===
loading initial cache file /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/child_image_preload.cmake
Loading Zephyr default modules (Zephyr base).
-- Application: /opt/nordic/ncs/v2.5.1/bootloader/mcuboot/boot/zephyr
-- CMake version: 3.28.3
-- Found Python3: /opt/homebrew/opt/python@3.11/bin/python3.11 (found suitable version "3.11.8", minimum required is "3.8") found components: Interpreter 
-- Cache files will be written to: /Users/mikaelj/Library/Caches/zephyr
-- Zephyr version: 3.4.99 (/opt/nordic/ncs/v2.5.1/zephyr)
-- Found west (found suitable version "1.2.0", minimum required is "0.14.0")
-- Board: circuitdojo_feather_nrf9160
-- Found host-tools: zephyr 0.16.1 (/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk)
-- Found toolchain: zephyr 0.16.1 (/opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk)
-- Found Dtc: /opt/homebrew/bin/dtc (found suitable version "1.7.0", minimum required is "1.4.6") 
-- Found BOARD.dts: /opt/nordic/ncs/v2.5.1/zephyr/boards/arm/circuitdojo_feather_nrf9160/circuitdojo_feather_nrf9160.dts
-- Found devicetree overlay: /opt/nordic/ncs/v2.5.1/bootloader/mcuboot/boot/zephyr/app.overlay
-- Generated zephyr.dts: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/zephyr.dts
-- Generated devicetree_generated.h: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/include/generated/devicetree_generated.h
-- Including generated dts.cmake file: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/dts.cmake

warning: UART_CONSOLE (defined at drivers/console/Kconfig:43) was assigned the value 'y' but got the
value 'n'. Check these unsatisfied dependencies: CONSOLE (=n). See
http://docs.zephyrproject.org/latest/kconfig.html#CONFIG_UART_CONSOLE and/or look up UART_CONSOLE in
the menuconfig/guiconfig interface. The Application Development Primer, Setting Configuration
Values, and Kconfig - Tips and Best Practices sections of the manual might be helpful too.

Parsing /opt/nordic/ncs/v2.5.1/bootloader/mcuboot/boot/zephyr/Kconfig
Loaded configuration '/opt/nordic/ncs/v2.5.1/zephyr/boards/arm/circuitdojo_feather_nrf9160/circuitdojo_feather_nrf9160_defconfig'
Merged configuration '/opt/nordic/ncs/v2.5.1/bootloader/mcuboot/boot/zephyr/prj.conf'
Merged configuration '/opt/nordic/ncs/v2.5.1/bootloader/mcuboot/boot/zephyr/boards/circuitdojo_feather_nrf9160.conf'
Merged configuration '/opt/nordic/ncs/v2.5.1/nrf/modules/mcuboot/tfm.conf'
Merged configuration '/opt/nordic/ncs/v2.5.1/nrf/modules/mcuboot/fih_low_enable.conf'
Merged configuration '/opt/nordic/ncs/v2.5.1/nrf/subsys/partition_manager/partition_manager_enabled.conf'
Merged configuration '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/misc/generated/extra_kconfig_options.conf'
Configuration saved to '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/.config'
Kconfig header saved to '/Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot/zephyr/include/generated/autoconf.h'
-- Found GnuLd: /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/../lib/gcc/arm-zephyr-eabi/12.2.0/../../../../arm-zephyr-eabi/bin/ld.bfd (found version "2.38") 
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc
MCUBoot bootloader key file: /opt/nordic/ncs/v2.5.1/bootloader/mcuboot/root-ec-p256.pem
-- Configuring done (2.7s)
-- Generating done (0.1s)
-- Build files have been written to: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/mcuboot
=== child image mcuboot -  end ===

CMake Warning at /opt/nordic/ncs/v2.5.1/nrf/modules/mcuboot/CMakeLists.txt:320 (message):
  

          ---------------------------------------------------------
          --- WARNING: Using default MCUBoot key, it should not ---
          --- be used for production.                           ---
          ---------------------------------------------------------
          



CMake Warning at /opt/nordic/ncs/v2.5.1/nrf/cmake/partition_manager.cmake:79 (message):
  

          ---------------------------------------------------------------------
          --- WARNING: Using a bootloader without pm_static.yml.            ---
          --- There are cases where a deployed product can consist of       ---
          --- multiple images, and only a subset of these images can be     ---
          --- upgraded through a firmware update mechanism. In such cases,  ---
          --- the upgradable images must have partitions that are static    ---
          --- and are matching the partition map used by the bootloader     ---
          --- programmed onto the device.                                   ---
          ---------------------------------------------------------------------
          

Call Stack (most recent call first):
  /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/kernel.cmake:247 (include)
  /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/zephyr_default.cmake:138 (include)
  /opt/nordic/ncs/v2.5.1/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /opt/nordic/ncs/v2.5.1/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:92 (include_boilerplate)
  CMakeLists.txt:6 (find_package)


Dropping partition 'nrf_modem_lib_trace' since its size is 0.
Dropping partition 'nonsecure_storage' since it is empty.
-- Configuring done (7.1s)
-- Generating done (0.1s)
-- Build files have been written to: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build
-- west build: building application
[1/241] Preparing syscall dependency handling

[8/241] Performing build step for 'mcuboot_subimage'
[1/211] Preparing syscall dependency handling

[5/211] Generating include/generated/version.h
-- Zephyr version: 3.4.99 (/opt/nordic/ncs/v2.5.1/zephyr), build: v3.4.99-ncs1-1
[211/211] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:       48484 B        48 KB     98.64%
             RAM:       24584 B        32 KB     75.02%
        IDT_LIST:          0 GB         2 KB      0.00%
[9/241] Generating include/generated/version.h
-- Zephyr version: 3.4.99 (/opt/nordic/ncs/v2.5.1/zephyr), build: v3.4.99-ncs1-1
[14/241] Generating ../../tfm/CMakeCache.txt
CMake Warning at cmake/version.cmake:22 (message):
  Actual TF-M version is not available from Git repository.  Settled to
  v1.8.0
Call Stack (most recent call first):
  CMakeLists.txt:22 (include)


-- Found Git: /opt/homebrew/bin/git (found version "2.42.0") 
-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gcc
-- Found Python3: /opt/homebrew/opt/python@3.11/bin/python3.11 (found version "3.11.8") found components: Interpreter 
CMake Deprecation Warning at /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/FindDeprecated.cmake:121 (message):
  'PYTHON_PREFER' variable is deprecated.  Please use Python3_EXECUTABLE
  instead.
Call Stack (most recent call first):
  /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/python.cmake:16 (find_package)
  /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/user_cache.cmake:30 (include)
  /opt/nordic/ncs/v2.5.1/zephyr/cmake/modules/extensions.cmake:5 (include)
  /opt/nordic/ncs/v2.5.1/nrf/subsys/nrf_security/tfm/CMakeLists.txt:36 (include)


-- Found Python3: /opt/homebrew/opt/python@3.11/bin/python3.11 (found suitable version "3.11.8", minimum required is "3.8") found components: Interpreter 
-- Cache files will be written to: /Users/mikaelj/Library/Caches/zephyr
-- Configuring done (1.3s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/tfm
[20/241] Performing build step for 'tfm'
[153/157] Linking C executable bin/tfm_s.axf
Memory region         Used Size  Region Size  %age Used
           FLASH:       31544 B      32256 B     97.79%
             RAM:       18832 B        32 KB     57.47%
[157/157] Linking C static library secure_fw/libtfm_s_veneers.a
[24/241] Performing install step for 'tfm'
-- Install configuration: "RelWithDebInfo"
[232/241] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:       61080 B       448 KB     13.31%
             RAM:        4772 B     211608 B      2.26%
        IDT_LIST:          0 GB         2 KB      0.00%
[236/241] Generating ../../zephyr/app_update.bin
image.py: sign the payload
[237/241] Generating ../../zephyr/app_signed.hex
image.py: sign the payload
[239/241] Generating ../../zephyr/app_test_update.hex
image.py: sign the payload
[241/241] Generating zephyr/merged.hex

$

debug zephyr.elf

$ /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb

GNU gdb (Zephyr SDK 0.16.1) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-host_apple-darwin --target=arm-zephyr-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://github.com/zephyrproject-rtos/sdk-ng/issues>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target extended-remote /opt/bmp
Remote debugging using /opt/bmp
(gdb) monitor swd_scan
Target voltage: 3.3V
Available Targets:
No. Att Driver
 1      Nordic nRF9160 M33
(gdb) att 1
Attaching to Remote target
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x00007b90 in ?? ()
(gdb) file build/zephyr/zephyr.elf 
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from build/zephyr/zephyr.elf...
(gdb) load build/zephyr/zephyr.elf 
Loading section rom_start, size 0x144 lma 0x18000
Loading section text, size 0xe6a0 lma 0x18144
Loading section .ARM.exidx, size 0x8 lma 0x267e4
Loading section initlevel, size 0x60 lma 0x267ec
Loading section device_area, size 0x64 lma 0x2684c
Loading section sw_isr_table, size 0x208 lma 0x268b0
Loading section log_const_area, size 0x60 lma 0x26ab8
Loading section rodata, size 0x278 lma 0x26b20
Loading section datas, size 0xb0 lma 0x26db0
Loading section device_states, size 0xa lma 0x26e60
Loading section k_heap_area, size 0x14 lma 0x26e6c
Loading section k_mutex_area, size 0x14 lma 0x26e80
Loading section .last_section, size 0x4 lma 0x26e94
Start address 0x000197d8, load size 61046
Transfer rate: 29 KB/sec, 836 bytes/write.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: .../blinky/build/zephyr/zephyr.elf 
^C
Program received signal SIGINT, Interrupt.
0x00007b90 in ?? ()
(gdb) bt
#0  0x00007b90 in ?? ()
#1  0x00000530 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Checking up the address of the crash. Looks like it’s in fatal.c (see below)

relevant parts of zephyr.map

.debug_info    0x0000000000005e57     0x2200 zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a(fatal.c.obj)
.debug_info    0x0000000000008057      0x995 zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a(irq_manage.c.obj)

debug merged.hex

(gdb) file build/zephyr/merged.hex 
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from build/zephyr/merged.hex...
(No debugging symbols found in build/zephyr/merged.hex)
(gdb) load build/zephyr/merged.hex 
Loading section .sec1, size 0xbd64 lma 0x0
Loading section .sec2, size 0x10000 lma 0x10000
Loading section .sec3, size 0x6f2f lma 0x20000
Start address 0x00000000, load size 142483
Transfer rate: 29 KB/sec, 962 bytes/write.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/merged.hex 

Program received signal SIGSEGV, Segmentation fault.
0x00000002 in ?? ()
(gdb)

pre-built blinky (nfed_v2)

$ /opt/nordic/ncs/toolchains/20d68df7e5/opt/zephyr-sdk/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb

GNU gdb (Zephyr SDK 0.16.1) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "--host=x86_64-host_apple-darwin --target=arm-zephyr-eabi".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://github.com/zephyrproject-rtos/sdk-ng/issues>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target extended-remote /opt/bmp
Remote debugging using /opt/bmp
(gdb) monitor swd_scan
Target voltage: 3.3V
Available Targets:
No. Att Driver
 1      Nordic nRF9160 M33
(gdb) att 1
Attaching to Remote target
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x00007b90 in ?? ()
(gdb) file ~/Downloads/nfed_v2/blinky_circuitdojo_feather_nrf9160_ns_merged.hex 
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from ~/Downloads/nfed_v2/blinky_circuitdojo_feather_nrf9160_ns_merged.hex...
(No debugging symbols found in ~/Downloads/nfed_v2/blinky_circuitdojo_feather_nrf9160_ns_merged.hex)
(gdb) load ~/Downloads/nfed_v2/blinky_circuitdojo_feather_nrf9160_ns_merged.hex 
Loading section .sec1, size 0xad44 lma 0x0
Loading section .sec2, size 0xd4bf lma 0x10000
Start address 0x00000000, load size 98819
Transfer rate: 28 KB/sec, 968 bytes/write.
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: .../Downloads/nfed_v2/blinky_circuitdojo_feather_nrf9160_ns_merged.hex 
^C
Program received signal SIGINT, Interrupt.
0x00019192 in ?? ()
(gdb)

….and this is blinking happily.

I’m scratching my head.

jaredwolff · Mar 3, 2024

What’s the output on the USB console?

mikaelj · Mar 3, 2024

jaredwolff
Nothing. minicom -b 115200 -D /dev/cu.SLAB_USBtoUART then RST button.

mikaelj · Mar 3, 2024

The stacktrace says 530 is somewhere in heap.c and 7b90 is somewhere in nrfx_clock.c

Makes no sense at all.

mikaelj · Mar 3, 2024

Keeps getting weirder… I can successfully run parts of programs by loading the merged.hex but using the zephyr.elf for debug symbols. i.e.

(gdb) file build/zephyr/zephyr.elf
(gdb) load build/zephyr/merged.hex

This way I get all the way to k_msleep(….) in the standard Blinky sample.

Still built as before, i.e.
west build -b circuitdojo_feather_nrf9160_ns

Digging down, I get a crash in z_tick_sleep():
z_add_thread_timeout(_current, timeout) <- bang
kernel/timeout.c: z_add_timeout()
timeout.c:139: sys_clock_set_timeout() <- bang
nrf_rtc_timer.c:689: compare_set() <- bang
nrf_rtc_timer.c:386: compare_set_nolocks() <- bang
nrf_rtc_timer.c:348: z_nrf_rtc_timer_read() <- bang
call counter() <- bang
nrf_rtc_timer.c:100: nrf_rtc_counter_get(RTC) <- bang
step into nrf_rtc_counter_get(RTC)
(gdb) p p_reg
$3 = (const NRF_RTC_Type *) 0×40015000
(gdb) p p_reg->COUNTER
Cannot access memory at address 0×40015504
(gdb)

According to the data sheet (https://infocenter.nordicsemi.com/index.jsp?topic=%2Fps_nrf9160%2Fmemory.html), the RTC1 (NS version) is indeed located at 0×40015000, and counter at offset 0×504.

However, I get memory inaccessible.
Just for fun (?) I tried building the S version, but get the same issue “inaccessible memory” but instead at 0×50015504.

mikaelj · Mar 4, 2024

Attempt #2 using v2.4.2 which is the same as the github CI uses, I get a crash in the very first parts of the Zephyr kernel initialization process:

(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y
Starting program: zephyr.elf

Breakpoint 4, regulator_is_enabled (dev=0x27dac <__device_dts_ord_8>) at /opt/nordic/ncs/v2.4.2/zephyr/drivers/regulator/regulator_common.c:109
109		const struct regulator_common_config *config = dev->config;
(gdb) c
Continuing.

Breakpoint 7, k_mutex_lock (mutex=0x2000c84c <data0>, timeout=...) at /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/include/generated/syscalls/kernel.h:955
955		return z_impl_k_mutex_lock(mutex, timeout);
(gdb) bt
#0  k_mutex_lock (mutex=0x2000c84c <data0>, timeout=...) at /Users/mikaelj/code/goride/centralstationen/nrf9160/vscode-nrf-sdk-test-2024-02-28/blinky/build/zephyr/include/generated/syscalls/kernel.h:955
#1  0x00024ff8 in regulator_is_enabled (dev=0x27dac <__device_dts_ord_8>) at /opt/nordic/ncs/v2.4.2/zephyr/drivers/regulator/regulator_common.c:116
#2  0x0002548c in regulator_fixed_init (dev=0x27dac <__device_dts_ord_8>) at /opt/nordic/ncs/v2.4.2/zephyr/drivers/regulator/regulator_fixed.c:80
#3  0x0001f1fe in z_sys_init_run_level (level=INIT_LEVEL_POST_KERNEL) at /opt/nordic/ncs/v2.4.2/zephyr/kernel/init.c:255
#4   0x0001f284 in bg_thread_main (unused1=0x0, unused2=0x0, unused3=0x0) at /opt/nordic/ncs/v2.4.2/zephyr/kernel/init.c:304
#5  0x00021ff0 in z_thread_entry (entry=0x1f26d <bg_thread_main>, p1=0x0, p2=0x0, p3=0x0) at /opt/nordic/ncs/v2.4.2/zephyr/lib/os/thread_entry.c:36
#6  0x0001ace2 in arch_switch_to_main_thread (main_thread=0x2000c680 <z_main_thread>, stack_ptr=0x2000ccd0 <z_idle_stacks> "\276o\347\357\064\070yp\257:\345]\364\377I]\337\067\377\273\231", _main=0x1f26d <bg_thread_main>)
        at /opt/nordic/ncs/v2.4.2/zephyr/arch/arm/core/aarch32/thread.c:576
#7  0x0001f422 in switch_to_main_thread (stack_ptr=0x2000ccd0 <z_idle_stacks> "\276o\347\357\064\070yp\257:\345]\364\377I]\337\067\377\273\231") at /opt/nordic/ncs/v2.4.2/zephyr/kernel/init.c:442
#8  0x0001f50c in z_cstart () at /opt/nordic/ncs/v2.4.2/zephyr/kernel/init.c:551
#9  0x0001ad12 in _isr_wrapper () at /opt/nordic/ncs/v2.4.2/zephyr/arch/arm/core/aarch32/isr_wrapper.S:259
#10 <signal handler called>
#11 0x1f00cb00 in ?? ()
#12 0x00024ff8 in regulator_is_enabled (dev=0x1f26d <bg_thread_main>) at /opt/nordic/ncs/v2.4.2/zephyr/drivers/regulator/regulator_common.c:116
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) b z_impl_k_mutex_lock
Breakpoint 8 at 0x1fc30: file /opt/nordic/ncs/v2.4.2/zephyr/kernel/mutex.c, line 99.
(gdb) c
Continuing.

Program received signal SIGINT, Interrupt.
0x00002dd8 in ?? ()
(gdb)

Corresponding to:

 .debug_abbrev  0x0000000000002c00      0x1df zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a(swap.c.obj)
 .debug_abbrev  0x0000000000002ddf       0x12 zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a(swap_helper.S.obj)

i.e. https://github.com/zephyrproject-rtos/zephyr/blob/v2.3-branch/arch/arm/core/aarch32/swap.c or its sibling swap_helper.S

Looking at swap.c, it seems like it’s something related to the task switcher. But I really don’t have knowledge about the Zephyr kernel. Clearly, there are many things happening here:

mutexes don’t work
memory-mapped peripherals give memory violations

Also, the green LED keeps blinking in a very high frequency, while connected to USB. I don’t think it used to do that. Is the board itself broken? What does “high frequency green LED blinking” mean?

mikaelj · Mar 5, 2024

I thought maybe v2.5.2 on macOS is broken….

Clean install of nRF v2.5.2 on Ubuntu 22.04.
west build -b circuitdojo_feather_nrf9160_ns -p
copied zephyr.elf and merged.hex to my mac.
zephyr-gdb load zephyr.elf => hangs not doing anything
zephyr-gdb load zephyr.hex => hangs doing nothing
zephyr-gdb load merged.hex => SIGSEGV
github-CI-built blinky_circuitdojo_feather_nrf9160_ns_merged.hex OK.

…but it’s just as broken on Ubuntu. Since the github-CI-built binary works when I transfer it over the BMP, I’m ruling out debugger error.

Same thing on nRF NCS v2.4.3 on Ubuntu.

src/main.c:

#include <zephyr/kernel.h>
#include <zephyr/drivers/gpio.h>
#define SLEEP_TIME_MS   1000
#define LED0_NODE DT_ALIAS(led0)
static const struct gpio_dt_spec led = GPIO_DT_SPEC_GET(LED0_NODE, gpios);
int main(void)
{
    int ret;

    if (!gpio_is_ready_dt(&led)) {
        return 0;
    }

    ret = gpio_pin_configure_dt(&led, GPIO_OUTPUT_ACTIVE);
    if (ret < 0) {
        return 0;
    }

    while (1) {
        ret = gpio_pin_toggle_dt(&led);
        if (ret < 0) {
            return 0;
        }
        k_msleep(SLEEP_TIME_MS);
    }
    return 0;
}

prj.conf:

CONFIG_GPIO=y
CONFIG_NO_OPTIMIZATIONS=y
CONFIG_BOOTLOADER_MCUBOOT=y

CMakeLists.txt:

# SPDX-License-Identifier: Apache-2.0

set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

cmake_minimum_required(VERSION 3.20.0)
find_package(Zephyr REQUIRED HINTS $ENV{ZEPHYR_BASE})
project(blinky)

target_sources(app PRIVATE src/main.c)

build output: https://gist.github.com/mikaelj/068c50620a5b9569e677a88cb6617471

mikaelj · Mar 6, 2024

Good news everyone! /Prof Fansworth

When debugging, I prefer -O0, so I read up on Zephyr and it said CONFIG_NO_OPTIMIZATIONS=y,

CONFIG_NO_OPTIMIZATIONS=y is incompatible with the nrf9160!

Also, beware BlackMagicProbe users: zephyr.elf can be set as gdb:s “file”, but loading zephyr.elf does not work!

Working GDB commands:

~/.zephyrtools/toolchains/zephyr-sdk-0.16.4/arm-zephyr-eabi/bin/arm-zephyr-eabi-gdb \
   -ex "target extended-remote /dev/path-to-bmp" \
   -ex "monitor swd_scan" -ex "att 1" \
   -ex "file build/zephyr/zephyr.elf" \
   -ex "load build/zephyr/merged.hex"

I can now successfully build out-of-tree with nRF Conect for Desktop-installed SDK and with Jared’s Zehyr Tools.

Phew. What a strange debugging journey.

jaredwolff · Mar 6, 2024

Wow good work @mikaelj. I would have never arrived there so well done