Connecting Feather to Mosquitto: Error mqtt_connect -45

Tonyweil

I followed the instructions in the blog article "How to Connect nRF9160 Feather to Self-Hosted Mosquitto" as closely as possible (amazing instructions!). I got almost to the last step where mqtt_simple was loaded on the nrf9160, tried to connect to the mosquitto server and received the following error in the LTE Link Monitor:

ERROR: mqtt_connect -45

I verifed the mosquitto service was runnning on the server and that the server was accessible by host name and IP address. I double checked all the config files and certificates on the server and nrf9160.

2020-10-27T22:58:54.922Z DEBUG modem << *** Booting Zephyr OS build v2.3.0-rc1-ncs3  ***
2020-10-27T22:58:54.960Z DEBUG modem << The MQTT simple sample started
2020-10-27T22:58:54.962Z DEBUG modem << LTE Link Connecting ...
2020-10-27T23:00:25.556Z DEBUG modem << +CEREG: 2,"4C11","0AD9D110",7,0,0,"11100000","11100000"
2020-10-27T23:00:25.668Z DEBUG modem << +CSCON: 1
2020-10-27T23:00:27.251Z DEBUG modem << +CEREG: 5,"4C11","0AD9D110",7,,,"11100000","11100000"
2020-10-27T23:00:27.293Z DEBUG modem >> AT+COPS=3,2
2020-10-27T23:00:27.298Z DEBUG modem << LTE Link Connected!
2020-10-27T23:00:27.302Z DEBUG modem << OK
2020-10-27T23:00:27.306Z DEBUG modem >> AT+COPS?
2020-10-27T23:00:27.322Z DEBUG modem << +COPS: 0,2,"310410",7
2020-10-27T23:00:27.324Z DEBUG modem << OK
2020-10-27T23:00:27.329Z DEBUG modem >> AT%XCBAND
2020-10-27T23:00:27.336Z DEBUG modem << %XCBAND: 17
2020-10-27T23:00:27.338Z DEBUG modem << OK
2020-10-27T23:00:27.354Z DEBUG modem >> AT+CGDCONT?
2020-10-27T23:00:27.371Z DEBUG modem << +CGDCONT: 0,"IP","hologram","10.69.24.70",0,0
2020-10-27T23:00:27.373Z DEBUG modem << OK
2020-10-27T23:00:27.390Z DEBUG modem >> AT+CGACT?
2020-10-27T23:00:27.398Z DEBUG modem << +CGACT: 0,1
2020-10-27T23:00:27.399Z DEBUG modem << OK
2020-10-27T23:00:27.553Z DEBUG modem << IPv4 Address found 157.230.215.243
2020-10-27T23:00:29.149Z DEBUG modem << ERROR: mqtt_connect -45
2020-10-27T23:00:34.868Z DEBUG modem << +CSCON: 0

jaredwolff

Tonyweil (amazing instructions!).

Very long instructions 😅

Looking at errno.h -45 seems to be correlated to EOPNOTSUPP. (/* Operation not supported on socket */)
So likely it has to do with the certs on the server or client side.

I have a bunch of questions:

Are you using TLS? Or plain text?
Is the port open on that server?
Did you load the certs onto your nRF910 Feather?
Have you been able to connect using the mosquitto cli? That would be one of the first things I would try.

Tonyweil

jaredwolff Did you load the certs onto your nRF910 Feather?

So, the following Nordic post mentioned in your article says that error 45 is usually related to bad client certs and explains how to create and install the certs. I think I did this correctly according to your instructions which are a different method than Nordic's:

I used the current version of LTE Link Monitor v1.1.8 to upload the following cut and pasted from the mosquitto server and entered tag 1234

ca.crt
nrf9160.crt
nrf9160.key

2020-10-27T21:27:42.894Z DEBUG modem << OK
2020-10-27T21:33:02.801Z INFO Updating CA certificate...
2020-10-27T21:33:02.803Z DEBUG modem >> AT%CMNG=0,1234,0,"-----BEGIN CERTIFICATE-----
2020-10-27T21:33:02.817Z DEBUG modem >> .....
2020-10-27T21:33:03.071Z DEBUG modem >> -----END CERTIFICATE-----"
2020-10-27T21:33:03.239Z DEBUG modem << OK
2020-10-27T21:33:03.241Z INFO Updating client certificate...
2020-10-27T21:33:03.244Z DEBUG modem >> AT%CMNG=0,1234,1,"-----BEGIN CERTIFICATE-----
2020-10-27T21:33:03.257Z DEBUG modem >> .....
2020-10-27T21:33:03.511Z DEBUG modem >> -----END CERTIFICATE-----"
2020-10-27T21:33:03.545Z DEBUG modem << OK
2020-10-27T21:33:03.546Z INFO Updating private key...
2020-10-27T21:33:03.547Z DEBUG modem >> AT%CMNG=0,1234,2,"-----BEGIN PRIVATE KEY-----
2020-10-27T21:33:03.560Z DEBUG modem >> .....
2020-10-27T21:33:03.936Z DEBUG modem >> -----END PRIVATE KEY-----"
2020-10-27T21:33:03.991Z DEBUG modem << OK
2020-10-27T21:33:03.992Z INFO Certificate update completed
2020-10-27T21:34:58.095Z DEBUG modem >> AT%CMNG=1
2020-10-27T21:34:58.122Z DEBUG modem << %CMNG: 0,6,"0606060606060606060606060606060606060606060606060606060606060606"
2020-10-27T21:34:58.127Z DEBUG modem << %CMNG: 1234,0,"0000000000000000000000000000000000000000000000000000000000000000"
2020-10-27T21:34:58.153Z DEBUG modem << %CMNG: 1234,1,"0101010101010101010101010101010101010101010101010101010101010101"
2020-10-27T21:34:58.169Z DEBUG modem << %CMNG: 1234,2,"0202020202020202020202020202020202020202020202020202020202020202"
2020-10-27T21:34:58.176Z DEBUG modem << %CMNG: 16842753,0,"0000000000000000000000000000000000000000000000000000000000000000"
2020-10-27T21:34:58.196Z DEBUG modem << %CMNG: 16842753,1,"0101010101010101010101010101010101010101010101010101010101010101"
2020-10-27T21:34:58.209Z DEBUG modem << %CMNG: 16842753,2,"0202020202020202020202020202020202020202020202020202020202020202"

jaredwolff Have you been able to connect using the mosquitto cli? That would be one of the first things I would try.

Some progress! I loaded the mosquitto clients on my Windows laptop.

Mosquitto client works on Windows Laptop to port 8885 TLS on BSD mosquitto server with nrf9160 client certs.

CMD Window 1
c:\Program Files\mosquitto>mosquitto_sub -h mosquitto1.my_domain.org -p 8885 -v -t 'test/topic' --cafile ca.crt --cert nrf9160.crt --key nrf9160.key

CMD Window 2
c:\Program Files\mosquitto>mosquitto_pub -h mosquitto1.my_domain.org -p 8885 -t 'test/topic' -m 'hello_there_again' -d --cafile ca.crt --cert nrf9160.crt --key nrf9160.key
Client mosq-IfHH00JFjJJcjw3HiC sending CONNECT
Client mosq-IfHH00JFjJJcjw3HiC received CONNACK (0)
Client mosq-IfHH00JFjJJcjw3HiC sending PUBLISH (d0, q0, r0, m1, ''test/topic'', ... (19 bytes))
Client mosq-IfHH00JFjJJcjw3HiC sending DISCONNECT

Back to CMD Window 1, hello_world_ message appears
c:\Program Files\mosquitto>mosquitto_sub -h mosquitto1.my_domain.org -p 8885 -v -t 'test/topic' --cafile ca.crt --cert nrf9160.crt --key nrf9160.key
'test/topic' 'hello_there_again'

jaredwolff Are you using TLS? Or plain text?

TLS. I followed all the instructions for TLS. See config file snippets at bottom of this reply:

jaredwolff Is the port open on that server?

Yes.
On the Mosquitto Server:

root@mosquitto1:/usr/local/etc/mosquitto # sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
nobody   mosquitto  1790  3  tcp4   *:*                   *:*
nobody   mosquitto  1790  6  tcp4   *:1883                *:*
nobody   mosquitto  1790  8  tcp4   *:8885                *:*
root     sshd       1235  3  tcp4   157.xxx.xxx.xxx:22  xx.xx.xx.xx:xxxxx
root     sendmail   1099  3  tcp4   127.0.0.1:25          *:*
root     sshd       1096  4  tcp4   *:22                  *:*
root     syslogd    622   7  udp4   *:514                 *:*

On a Laptop Client:

tony@tony-ubuntu:~$ telnet mosquitto1.my_domain.org 8885
Trying 157.xxx.xxx.xxx...
Connected to mosquitto1._my__domain_.org.
tony@tony-ubuntu:~$ telnet mosquitto1.my_domain.org 1883
Trying 157.xxx.xxx.xxx...
Connected to mosquitto1.my_domain.org.

Misc Configuration info:

Droplet server: mosquitto.conf snippets

# Port to use for the default listener.
port 8885

# listener port-number [ip address/host name]
listener 1883
protocol mqtt

I had to change the 3 cert lines from /root/pki/... to /pki/... or the mosquitto service would not start. For example:
cafile /root/pki/ca.crt > cafile /pki/ca.crt

mqtt_simple prj.conf snippets

# MQTT
CONFIG_MQTT_LIB=y
CONFIG_MQTT_LIB_TLS=y

# Application
CONFIG_MQTT_PUB_TOPIC="/my/publish/topic"
CONFIG_MQTT_SUB_TOPIC="/my/subscribe/topic"
CONFIG_MQTT_CLIENT_ID="nrf9160-feather"
CONFIG_MQTT_BROKER_HOSTNAME="mosquitto1.my_domain.org"
CONFIG_MQTT_BROKER_PORT=8885
CONFIG_SEC_TAG=1234

# Set the PDP context
CONFIG_LTE_PDP_CMD=y
CONFIG_LTE_PDP_CONTEXT="0,\"IP\",\"hologram\""

Kconfig snippets

config SEC_TAG
	int "Security tag to use for the connection"
	default 1234

config PEER_VERIFY
	int "Peer verify parameter for mqtt_client"
	default 1
	help
		Set to 0 for VERIFY_NONE, 1 for VERIFY_OPTIONAL, and 2 for VERIFY_REQUIRED.

main.c

Just under the includes:

#if defined(CONFIG_MQTT_LIB_TLS)
static sec_tag_t sec_tag_list[] = { CONFIG_SEC_TAG };
#endif /* defined(CONFIG_MQTT_LIB_TLS) */

client_init section

/* MQTT transport configuration */
#if defined(CONFIG_MQTT_LIB_TLS)
       struct mqtt_sec_config *tls_config = &client->transport.tls.config;

	client->transport.type = MQTT_TRANSPORT_SECURE;

	tls_config->peer_verify = CONFIG_PEER_VERIFY;
	tls_config->cipher_count = 0;
	tls_config->cipher_list = NULL;
	tls_config->sec_tag_count = ARRAY_SIZE(sec_tag_list);
	tls_config->sec_tag_list = sec_tag_list;
	tls_config->hostname = CONFIG_MQTT_BROKER_HOSTNAME;
	client->transport.type = MQTT_TRANSPORT_SECURE;
#else
	client->transport.type = MQTT_TRANSPORT_NON_SECURE;
#endif

Tonyweil

Well, lots of progress. I learned a lot about MQTT along the way.

I disabled TLS and was able to get the MQTT client connected on the nrf9160.

I went back and triple checked all the TLS settings and discovered that the TLS changes to main.c didn't stick. After fixing that, the nrf9160 MQTT client connected to the mosquitto server over TLS, . With all the changes I made, how can I tell if the nrf9160 is actually using TLS or not?

*** Booting Zephyr OS build v2.3.0-rc1-ncs3  ***
The MQTT simple sample started
LTE Link Connecting ...
+CEREG: 2,"4C11","0AD9D110",7,0,0,"11100000","11100000"
+CSCON: 1
+CEREG: 5,"4C11","0AD9D110",7,,,"11100000","11100000"
LTE Link Connected!
IPv4 Address found 157.xxx.xxx.xxx
[mqtt_evt_handler:198] MQTT client connected!
Subscribing to: /my/subscribe/topic len 19
[mqtt_evt_handler:248] SUBACK packet id: 1234

Testing with Mosquito client tools and works as advertised!

I'm not quite understanding the purpose of why it is helpful to send info that is received by the nrf9160 on one topic and then the nrf9160 forwards it to different topic. In practice, how is this useful?

CMD Window 1 - Subscribe to /my/public/topic

c:\Program Files\mosquitto>mosquitto_sub -h mosquitto1.my_domain.org -p 8885 -v -t /my/publish/topic --cafile ca.crt --cert test.crt --key test.key

CMD Window 2 - Send message to /my/subscribe/topic

c:\Program Files\mosquitto>mosquitto_pub -h mosquitto1.my_domain.org -p 8885 -t /my/subscribe/topic -m hello_nrf9160 --cafile ca.crt --cert test.crt --key test.key

nrf9160 serial console results: nrf9160 receives info at my/subscribe/topic and forwards to /my/publish/topic

LTE Link Connected!
IPv4 Address found 157.xxx.xxx.xxx
[mqtt_evt_handler:198] MQTT client connected!
Subscribing to: /my/subscribe/topic len 19
[mqtt_evt_handler:248] SUBACK packet id: 1234
[mqtt_evt_handler:213] MQTT PUBLISH result=0 len=13
Received: hello_nrf9160
Publishing: hello_nrf9160
to topic: /my/publish/topic len: 17

CMD Window 1 - Receives message from nef9160 to /my/publish/topic

c:\Program Files\mosquitto>mosquitto_sub -h mosquitto1.my_domain.org -p 8885 -v -t /my/publish/topic --cafile ca.crt --cert test.crt --key test.key
/my/publish/topic hello_nrf9160

jaredwolff

Tonyweil Well, lots of progress. I learned a lot about MQTT along the way.

Nice. Congrats this is not the easiest thing to do!

Tonyweil With all the changes I made, how can I tell if the nrf9160 is actually using TLS or not?

The best way to tell is to make sure that your non-TLS endpoint is disabled or that your firewall is blocking that port. Also within the MQTT related code you'll see that the socket setup involves referencing your SSL certs. Here are the lines from the MQTT config:

    /* MQTT transport configuration */
    struct mqtt_sec_config *tls_config = &client->transport.tls.config;

    client->transport.type = MQTT_TRANSPORT_SECURE;
    tls_config->peer_verify = CONFIG_PYRINAS_CLOUD_PEER_VERIFY;
    tls_config->cipher_count = 0;
    tls_config->cipher_list = NULL;
    tls_config->sec_tag_count = ARRAY_SIZE(sec_tag_list);
    tls_config->sec_tag_list = sec_tag_list;
    tls_config->hostname = CONFIG_PYRINAS_CLOUD_MQTT_BROKER_HOSTNAME;

Here's how I did it for one of the projects I've been working on.

jaredwolff

Tonyweil I'm not quite understanding the purpose of why it is helpful to send info that is received by the nrf9160 on one topic and then the nrf9160 forwards it to different topic. In practice, how is this useful?

I like differentiating between the RX and TX channels of communicating. Its easier to determine what commands are supposed to go where. Plus if you send to a topic you're not sending any data back to yourself as, in theory, any messages sent to a topic to your subscribed to will bounce right back to the device.

Tonyweil

jaredwolff Here’s how I did it for one of the projects I’ve been working on.

Thanks for all the help. Can you explain at a very high level what this application does?

Tonyweil

jaredwolff I like differentiating between the RX and TX channels of communicating.

Is it safe to say that in some simple cases you might just want to simply Publish some sensor data from the nrf9160 (TX) to a topic(1) that is Subscribed to by some external mqtt client?

You might also choose to use an external mqtt client to Publish to a different topic(2) that the nrf9160 (RX) Subscribes to, say in order to send instructions of some kind to it. In this case, Publishing an echo from the nrf9160 (TX again) to yet another topic(3) that the external mqtt client to Subscribes to might be useful to confirm the command was received.

The mqtt_simple example seems like it is also a more or less a clever way to easily demonstrate subscribe and publish capabilities of the nrf9160, although a little convoluted to understand at first.

jaredwolff

Tonyweil What I was describing was the typical way of using a pub/sub protocol like MQTT. I'm not the MQTT police though. You can do whatever you want with it. 😀

jaredwolff

Tonyweil its my version of the nRF Cloud client with built-in OTA etc. It’s supposed to make adding my self-hosted cloud stuff easier to deploy. Very much so a work in progress. 🙂