AWS_fota: download successful but do not load with new binary file

zirunhong · Apr 14, 2021

Hi,

I was trying with the aws_fota sample, I follow the instruction here and download the app_update.bin with no problem. After it shows completed in 100%, device reboot automatically but wasn’t running the downloaded binary (the version number printed wasn’t changed). I checked the memory using Programmer tool in nRF Connect, seems it have the firmware storage in the memory location. I was new to nrf9160 and fota, can anyone helps me or with any suggestion?

Here is the terminal print:

*** Booting Zephyr OS build v2.4.99-ncs1 ***
Flash regions Domain Permissions
00 02 0×00000 0×18000 Secure rwxl
03 31 0×18000 0×100000 Non-Secure rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region Domain Permissions
00 07 0×00000 0×10000 Secure rwxl
08 31 0×10000 0×40000 Non-Secure rwxl

Peripheral Domain Status
00 NRF_P0 Non-Secure OK
01 NRF_CLOCK Non-Secure OK
02 NRF_RTC0 Non-Secure OK
03 NRF_RTC1 Non-Secure OK
04 NRF_NVMC Non-Secure OK
05 NRF_UARTE1 Non-Secure OK
06 NRF_UARTE2 Secure SKIP
07 NRF_TWIM2 Non-Secure OK
08 NRF_SPIM3 Non-Secure OK
09 NRF_TIMER0 Non-Secure OK
10 NRF_TIMER1 Non-Secure OK
11 NRF_TIMER2 Non-Secure OK
12 NRF_SAADC Non-Secure OK
13 NRF_PWM0 Non-Secure OK
14 NRF_PWM1 Non-Secure OK
15 NRF_PWM2 Non-Secure OK
16 NRF_PWM3 Non-Secure OK
17 NRF_WDT Non-Secure OK
18 NRF_IPC Non-Secure OK
19 NRF_VMC Non-Secure OK
20 NRF_FPU Non-Secure OK
21 NRF_EGU1 Non-Secure OK
22 NRF_EGU2 Non-Secure OK
23 NRF_DPPIC Non-Secure OK
24 NRF_REGULATORS Non-Secure OK
25 NRF_GPIOTE1 Non-Secure OK

SPM: NS image at 0×1c200
SPM: NS MSP at 0×2001e6d0
SPM: NS reset vector at 0×2217d
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.4.99-ncs1 ***
MQTT AWS Jobs FOTA Sample, version: v2.0.1
Initializing modem library
ret: 0
Initialized modem library
************************* WARNING *************************
provision_certificates called do not use this in production!
This will store the certificates in readable flash and leave
them exposed on modem_traces. Only use this once for
provisioning certificates for development to reduce flash tear.
************************* WARNING *************************
modem_key_mgmt_delete(20210101, 0) => result=0
modem_key_mgmt_delete(20210101, 1) => result=0
modem_key_mgmt_delete(20210101, 2) => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
LTE Link Connecting …
LTE Link Connected!
IPv4 Address 3.96.79.126
client_id: nrf9160
[mqtt_evt_handler:182] MQTT client connected!
[mqtt_evt_handler:235] PUBACK packet id: 12725
[mqtt_evt_handler:235] PUBACK packet id: 36486
[mqtt_evt_handler:235] PUBACK packet id: 47677
I: Configuring socket timeout (30 s)
I: Connecting to nrf-fota-new.s3.ca-central-1.amazonaws.com
I: Downloading: app_update.bin [0]
AWS_FOTA_EVT_START, job id = fota
I: Downloaded 2048/181024 bytes (1%)
I: 2 Sectors of 4096 bytes
I: alloc wra: 0, a10
I: data wra: 0, 2f0
I: Downloaded 4096/181024 bytes (2%)
I: Downloaded 6144/181024 bytes (3%)
I: Downloaded 8192/181024 bytes (4%)
I: Downloaded 10240/181024 bytes (5%)
I: Downloaded 12288/181024 bytes (6%)
I: Downloaded 14336/181024 bytes (7%)
I: Downloaded 16384/181024 bytes (9%)
I: Downloaded 18432/181024 bytes (10%)
I: Downloaded 20480/181024 bytes (11%)
I: Downloaded 22528/181024 bytes (12%)
I: Downloaded 24576/181024 bytes (13%)
I: Downloaded 26624/181024 bytes (14%)
I: Downloaded 28672/181024 bytes (15%)
I: Downloaded 30720/181024 bytes (16%)
I: Downloaded 32768/181024 bytes (18%)
I: Downloaded 34816/181024 bytes (19%)
I: Downloaded 36864/181024 bytes (20%)
I: Downloaded 38912/181024 bytes (21%)
I: Downloaded 40960/181024 bytes (22%)
I: Downloaded 43008/181024 bytes (23%)
I: Downloaded 45056/181024 bytes (24%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 47104/181024 bytes (26%)
I: Downloaded 49152/181024 bytes (27%)
I: Downloaded 51200/181024 bytes (28%)
I: Downloaded 53248/181024 bytes (29%)
I: Downloaded 55296/181024 bytes (30%)
I: Downloaded 57344/181024 bytes (31%)
I: Downloaded 59392/181024 bytes (32%)
I: Downloaded 61440/181024 bytes (33%)
I: Downloaded 63488/181024 bytes (35%)
I: Downloaded 65536/181024 bytes (36%)
I: Downloaded 67584/181024 bytes (37%)
I: Downloaded 69632/181024 bytes (38%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 71680/181024 bytes (39%)
I: Downloaded 73728/181024 bytes (40%)
I: Downloaded 75776/181024 bytes (41%)
I: Downloaded 77824/181024 bytes (42%)
I: Downloaded 79872/181024 bytes (44%)
I: Downloaded 81920/181024 bytes (45%)
I: Downloaded 83968/181024 bytes (46%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 86016/181024 bytes (47%)
I: Downloaded 88064/181024 bytes (48%)
I: Downloaded 90112/181024 bytes (49%)
I: Downloaded 92160/181024 bytes (50%)
I: Downloaded 94208/181024 bytes (52%)
I: Downloaded 96256/181024 bytes (53%)
W: Peer closed connection, will re-connect
[mqtt_evt_handler:250] default: 9
I: Downloaded 98304/181024 bytes (54%)
I: Reconnecting..
I: Configuring socket timeout (30 s)
I: Connecting to nrf-fota-new.s3.ca-central-1.amazonaws.com
I: Downloaded 100352/181024 bytes (55%)
I: Downloaded 102400/181024 bytes (56%)
I: Downloaded 104448/181024 bytes (57%)
I: Downloaded 106496/181024 bytes (58%)
I: Downloaded 108544/181024 bytes (59%)
I: Downloaded 110592/181024 bytes (61%)
I: Downloaded 112640/181024 bytes (62%)
I: Downloaded 114688/181024 bytes (63%)
I: Downloaded 116736/181024 bytes (64%)
I: Downloaded 118784/181024 bytes (65%)
I: Downloaded 120832/181024 bytes (66%)
I: Downloaded 122880/181024 bytes (67%)
I: Downloaded 124928/181024 bytes (69%)
I: Downloaded 126976/181024 bytes (70%)
I: Downloaded 129024/181024 bytes (71%)
I: Downloaded 131072/181024 bytes (72%)
I: Downloaded 133120/181024 bytes (73%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 135168/181024 bytes (74%)
I: Downloaded 137216/181024 bytes (75%)
I: Downloaded 139264/181024 bytes (76%)
I: Downloaded 141312/181024 bytes (78%)
I: Downloaded 143360/181024 bytes (79%)
I: Downloaded 145408/181024 bytes (80%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 147456/181024 bytes (81%)
E: Failed to send HTTP request, errno 128
W: Download socket error. 2 retries left…
I: Reconnecting..
I: Configuring socket timeout (30 s)
I: Connecting to nrf-fota-new.s3.ca-central-1.amazonaws.com
I: Downloaded 149504/181024 bytes (82%)
I: Downloaded 151552/181024 bytes (83%)
I: Downloaded 153600/181024 bytes (84%)
I: Downloaded 155648/181024 bytes (85%)
I: Downloaded 157696/181024 bytes (87%)
I: Downloaded 159744/181024 bytes (88%)
I: Downloaded 161792/181024 bytes (89%)
I: Downloaded 163840/181024 bytes (90%)
I: Downloaded 165888/181024 bytes (91%)
I: Downloaded 167936/181024 bytes (92%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 169984/181024 bytes (93%)
I: Downloaded 172032/181024 bytes (95%)
I: Downloaded 174080/181024 bytes (96%)
[mqtt_evt_handler:250] default: 9
I: Downloaded 176128/181024 bytes (97%)
I: Downloaded 178176/181024 bytes (98%)
I: Downloaded 180224/181024 bytes (99%)
E: Failed to send HTTP request, errno 128
W: Download socket error. 1 retries left…
I: Reconnecting..
[mqtt_evt_handler:250] default: 9
I: Configuring socket timeout (30 s)
I: Connecting to nrf-fota-new.s3.ca-central-1.amazonaws.com
I: Downloaded 181024/181024 bytes (100%)
I: Download complete
I: MCUBoot image upgrade scheduled. Reset device to apply
AWS_FOTA_EVT_DL_PROGRESS, 100% downloaded
[mqtt_evt_handler:235] PUBACK packet id: 2556
[mqtt_evt_handler:235] PUBACK packet id: 45978
AWS_FOTA_EVT_DONE, rebooting to apply update
?** Booting Zephyr OS build v2.4.99-ncs1 ***d 0
Flash regions Domain Permissions
00 02 0×00000 0×18000 Secure rwxl
03 31 0×18000 0×100000 Non-Secure rwxl

Non-secure callable region 0 placed in flash region 2 with size 32.

SRAM region Domain Permissions
00 07 0×00000 0×10000 Secure rwxl
08 31 0×10000 0×40000 Non-Secure rwxl

Peripheral Domain Status
00 NRF_P0 Non-Secure OK
01 NRF_CLOCK Non-Secure OK
02 NRF_RTC0 Non-Secure OK
03 NRF_RTC1 Non-Secure OK
04 NRF_NVMC Non-Secure OK
05 NRF_UARTE1 Non-Secure OK
06 NRF_UARTE2 Secure SKIP
07 NRF_TWIM2 Non-Secure OK
08 NRF_SPIM3 Non-Secure OK
09 NRF_TIMER0 Non-Secure OK
10 NRF_TIMER1 Non-Secure OK
11 NRF_TIMER2 Non-Secure OK
12 NRF_SAADC Non-Secure OK
13 NRF_PWM0 Non-Secure OK
14 NRF_PWM1 Non-Secure OK
15 NRF_PWM2 Non-Secure OK
16 NRF_PWM3 Non-Secure OK
17 NRF_WDT Non-Secure OK
18 NRF_IPC Non-Secure OK
19 NRF_VMC Non-Secure OK
20 NRF_FPU Non-Secure OK
21 NRF_EGU1 Non-Secure OK
22 NRF_EGU2 Non-Secure OK
23 NRF_DPPIC Non-Secure OK
24 NRF_REGULATORS Non-Secure OK
25 NRF_GPIOTE1 Non-Secure OK

SPM: NS image at 0×1c200
SPM: NS MSP at 0×2001e6d0
SPM: NS reset vector at 0×2217d
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.4.99-ncs1 ***
MQTT AWS Jobs FOTA Sample, version: v2.0.1
Initializing modem library
ret: 0
Initialized modem library
************************* WARNING *************************
provision_certificates called do not use this in production!
This will store the certificates in readable flash and leave
them exposed on modem_traces. Only use this once for
provisioning certificates for development to reduce flash tear.
************************* WARNING *************************
modem_key_mgmt_delete(20210101, 0) => result=0
modem_key_mgmt_delete(20210101, 1) => result=0
modem_key_mgmt_delete(20210101, 2) => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
modem_key_mgmt_write => result=0
LTE Link Connecting …

Thanks in advanced!

Zirun

jaredwolff · Apr 14, 2021

zirunhong you’re talking about the line here?

MQTT AWS Jobs FOTA Sample, version: v2.0.1

Depending on how that version is included in your project, you may need to use the -p flag. I often need to use the -p flag for my OTA updates since I grab my version # from git describe.

Additionally, it also may be rejecting your OTA package. Are you able to update the image using newtmgr and see the output change?

zirunhong · Apr 14, 2021

jaredwolff

Thanks for reply.

you’re talking about the line here?

MQTT AWS Jobs FOTA Sample, version: v2.0.1

Yes, I defined it in prj.conf. And I changed to a different tag before build the new binary and upload to S3, just as a indicator for different firmware.

Depending on how that version is included in your project, you may need to use the -p flag. I often need to use the -p flag for my OTA updates since I grab my version # from git describe.

Can you explain a little bit on the -p flag, I am new to it.

Additionally, it also may be rejecting your OTA package. Are you able to update the image using newtmgr and see the output change?

I guess it may be poor signal, I try with another successful download, but still same result.
I do uploaded through newtmgr, with the v2 bootloader. In case I do something wrong, I use the programmer in nRF Connect to write the v2 bootloader before I erase all, then use newtmgr to load flash the bin file.

Thanks,
Zirun

jaredwolff · Apr 14, 2021

zirunhong Can you explain a little bit on the -p flag, I am new to it.

zirunhong when building you can run west build -b circuitojo_feather_nrf9160ns -p This will clean your build directory. This is useful to make sure that your changes to the version have propagated correctly.

zirunhong · Apr 14, 2021

jaredwolff
I see. I try with your suggestion, but it still with the same result.

I noticed from the instruction, the ota download has the erasing process, like this:

[00:02:03.748,931] <inf> download_client: Downloaded 135168/201232 bytes (67%)
[00:02:03.794,494] <inf> fota_flash_block: Erasing sector at offset 0×00021000
[00:02:04.893,188] <inf> download_client: Downloaded 139264/201232 bytes (69%)
[00:02:04.938,720] <inf> fota_flash_block: Erasing sector at offset 0×00022000
[00:02:05.933,013] <inf> download_client: Downloaded 143360/201232 bytes (71%)
[00:02:05.978,546] <inf> fota_flash_block: Erasing sector at offset 0×00023000

While mine only show the download percentage:

I: Downloaded 155648/181118 bytes (85%)
I: Downloaded 157696/181118 bytes (87%)
I: Downloaded 159744/181118 bytes (88%)
I: Downloaded 161792/181118 bytes (89%)
I: Downloaded 163840/181118 bytes (90%)
I: Downloaded 165888/181118 bytes (91%)
I: Downloaded 167936/181118 bytes (92%)
I: Downloaded 169984/181118 bytes (93%)
I: Downloaded 172032/181118 bytes (94%)
I: Downloaded 174080/181118 bytes (96%)

It might be the SDK or library is different. Do you have any suggestions?

Zirun

jaredwolff · Apr 14, 2021

zirunhong I do uploaded through newtmgr, with the v2 bootloader. In case I do something wrong, I use the programmer in nRF Connect to write the v2 bootloader before I erase all, then use newtmgr to load flash the bin file.

To be clear here, were you able to update your application using newtmgr where the version number changes? i.e. the only place where you don’t see change is using FOTA?

jaredwolff · Apr 14, 2021

@zirunhong also are you using an example with LittleFS enabled? You may have to comment out lines 3-7 in pm.yml.file_system (located in nrf/subsys/partition_manager/pm.yml.file_system)

#include <autoconf.h>

##ifdef CONFIG_FILE_SYSTEM_LITTLEFS
#littlefs_storage:
#  placement: {before: [end]}
#  size: CONFIG_PM_PARTITION_SIZE_LITTLEFS
##endif

I originally had to comment this out to give as much space to the application firmware as possible. Since there is an external flash, this is not necessary.

zirunhong · Apr 14, 2021

jaredwolff

To be clear here, were you able to update your application using newtmgr where the version number changes? i.e. the only place where you don’t see change is using FOTA?

Correct, using newtmgr to update my application with different hardcoded version number works, but not on FOTA

also are you using an example with LittleFS enabled?

Yes, I are using the LittleFS, but we use the external W25Q32JV on the feather board, will that be affected?

Zirun

jaredwolff · Apr 15, 2021

zirunhong it’s possible since CONFIG_FILE_SYSTEM_LITTLEFS will be activated. Can you try commenting out those bits and trying again?

zirunhong · Apr 15, 2021

jaredwolff I comment out but still doesn’t work

jaredwolff · Apr 15, 2021

I would attempt to confirm that the image you’re uploading on AWS is the image you expect. I recommend you take a second look at how you’re uploading the images. I’ve made the mistake previously, after building on a clean project, uploading the wrong image all along.

jaredwolff · Apr 15, 2021

Also if you can copy the outputs from upgrading newtmgr here. That way we can confirm you are indeed getting the updated code. Also you may have to post the code that you’re using to generate the version #s.

zirunhong · Apr 15, 2021

jaredwolff
Hi,

Here is the output from newtmgr:
PS C:\Users\Zirun\ncs\v1.5.0\reliance_app\nrf9160_aws_fota> newtmgr -c serial image upload build/zephyr/app_update.bin
176.88 KiB / 176.88 KiB [========================================================================================================================================================================================================================================================================] 100.00% 13.74 KiB/s 12s
Done

Here is how I defined version #:
prj.conf:

CONFIG_APP_VERSION="v2.0.1"
CONFIG_USE_CLOUD_CLIENT_ID=y
CONFIG_CLOUD_CLIENT_ID="nrf9160"
CONFIG_MQTT_BROKER_HOSTNAME="a2v611xt1xntep-ats.iot.ca-central-1.amazonaws.com"
CONFIG_PROVISION_CERTIFICATES=y
CONFIG_CLOUD_CERT_SEC_TAG=20210101

The main.c is exact the same as the aws_fota sample_.

I have confirmed that I have changed the CONFIG_APP_VERSIONto “v2.0.2” and build with -p tag and upload the app_update.bin to s3 bucket.

jaredwolff · Apr 16, 2021

zirunhong I have confirmed that I have changed the CONFIG_APP_VERSIONto “v2.0.2” and build with -p tag and upload the app_update.bin to s3 bucket.

Can you see that you’re updating your device using the bootloader over USB that it’s also updating to v2.0.2? It’s important to confirm this locally before you upload.

Also, how does your device know which image to pull? How you know if it’s pulling the correct one?

zirunhong · Apr 21, 2021

jaredwolff

Hey, sorry for reply late.

Can you see that you’re updating your device using the bootloader over USB that it’s also updating to v2.0.2? It’s important to confirm this locally before you upload.

Yes, I try changing the hard-coded version to “v2.0.2” and using printk() to verify it.

Also, how does your device know which image to pull? How you know if it’s pulling the correct one?

Hmm..that’s something I don’t fully understand, on the instruction of aws__fota sample, it says:

The sample then retrieves the firmware image over HTTP and replaces the current firmware with the downloaded firmware.

I guest it may have a way in bootloader to distinguish it?

Also, I tried in this way: instead of using v2 bootloader and newtmgr, I use the debugger to program the device (west flash --runner nrfjprog). Then I changed the hardcoded version, build with -p tag, upload the app_update.bin to S3 bucket, following the same process of FOTA, and in next reboot after fota download complete, it loaded with the new firmware from S3 (according to the hard-coded version using printk()). I wonder if it is because of the v2 bootloader? Do you have any comment/suggestion on it?

Thanks!
Zirun

jaredwolff · Apr 24, 2021

zirunhong I don’t have any further suggestions. Looks like it’s working now though? If not, I would carefully read the instructions related to FOTA. It’s important you indicate the correct image in. S3 for it to work. (I recommend removing the old images when you’re done with them.)

zirunhong · Apr 28, 2021

jaredwolff
It is working now when uploading the hex file. However, another problem came out, I cant use newtmgr to upload the firmware. When I try to enter DFU mode by holding MODE button and tap RESET button, it didn’t looks like entering the DFU mode, and newtmgr -c serial image upload build/zephyr/app_update.bin doesn’t work. I have the CONFIG_BOOTLOADER_MCUBOOT=y in my prj.conf

jaredwolff · Apr 29, 2021

zirunhong make sure with v2 that you’re holding the MODE button and tapping the RST. Let go of the MODE once you have a blue LED. I added a delay in the bootloader so the device wouldn’t accidentally go into bootloader mode in some operating edge cases.

zirunhong · Apr 29, 2021

jaredwolff

Hi, thanks for reply.

I have try with the instruction and still the same. I have no problem with uploading the bin file with your v2 bootloader.

My concern is when I use my our bootloader, it doesn’t go to DFU mode. First, my understanding is, when compiling the application with CONFIG_BOOTLOADER_MCUBOOT=y in prj.conf, there will be a bootloader firmware/image being created, this should be the hex file in /build/mcuboot/zephyr/zephyr.hex . Please correct me if I am wrong.

Then for application, we can use newtmgr to flash the app_update.bin to the board. The bottle neck I have here is, my bootloader firmware doesn’t allow the board into DFU mode, which I cant flash the binary file for the application.

Thanks,
Zirun

jaredwolff · Apr 30, 2021

zirunhong My concern is when I use my our bootloader, it doesn’t go to DFU mode. First, my understanding is, when compiling the application with CONFIG_BOOTLOADER_MCUBOOT=y in prj.conf, there will be a bootloader firmware/image being created, this should be the hex file in /build/mcuboot/zephyr/zephyr.hex . Please correct me if I am wrong.

That’s slightly correct. When CONFIG_BOOTLOADER_MCUBOOT is selected it will generate a file call merged.hex which will contain the bootloader. zephyr.hex alone on the nRF9160 will not work. You’ll always need a merged.hex. (I believe it’s generate whether or not the bootloader is enabled.)

zirunhong these two statements seem to contradict each other:

I have no problem with uploading the bin file with your v2 bootloader.

The bottle neck I have here is, my bootloader firmware doesn’t allow the board into DFU mode, which I cant flash the binary file for the application.

Can you explain what you mean? You’re able to use newtmgr but not load via AWS FOTA? If I were you I’d compile and run the provided SDK examples and follow Nordic’s instructions explicitly. Make sure you can do it with unmodified code an then work on getting your own firmware working with it as well.