Introducing Lion, a self-hostable IoT server for Zephyr

jaredwolff

Hey all,

Some of you caught the Updates! livestream where I talked about Lion — the self-hostable IoT server I’ve been building for Zephyr devices. The companion blog post is now up, and it’s where I dig into the engineering decisions behind it rather than the feature list.

👉 Read it here: Introducing Lion: A Self-Hostable IoT Server for Zephyr Devices

Topics covered:

Device shadow as the source of truth — including why commands live inside the shadow with a generation counter (no separate queue)
Config polling instead of CoAP observe — a concession to NAT timeouts on sleepy nRF9151s
Dictionary-based log streaming over cellular — same trick defmt uses on SWD
Block2 CoAP OTA with MCUboot swap
A firmware versioning scheme (<branch>-<count>-<hash>) that’s actually useful for OTA tracking
Bootstrap PSK with dual-key rotation so you can rotate keys without bricking your fleet

If you want to follow along with the project, sign up for the preview at lioniot.dev. I’ll be sending updates as things move forward — more device targets, CoAP-over-Thread, and onboarding for early users.

Two questions I’d really like input on:

The shadow-as-command-queue choice — putting commands inside the device shadow with a generation counter, instead of a separate queue. Do you think this holds up at scale, or are there failure modes I’m missing?
PSK rotation flow — request new bootstrap PSK, ship firmware, revoke old. Is this the workflow you’d want, or do you have a better pattern in mind?

Bonus question: what’s missing from existing IoT servers that you’d want Lion to do? This is the time to influence the roadmap.

Thanks 🦁

— Jared

test

@cbrake should be good now!

cbrake

Looks good!

AchimKraus

FMPOV, some more details on DTLS would be great:
Will it support ECDHE_PSK cipher suites as well?
Will it support DTLS 1.2 CID (RFC 9146)?
Which implementation is used on the servers side? piondtls/GO, mbedtls/C, californium/Java, or?
Will it support a “DTLS graceful shutdown” (sometimes useful for non breaking server updates).

AchimKraus

Polling:
Does it mean, when the device sends sensor data in e.g. a POST, it will also need to send a separate GET for the Shadow?

jaredwolff

AchimKraus FMPOV, some more details on DTLS would be great:
Will it support ECDHE_PSK cipher suites as well?

Right now only AES128_CCM_8. Since it’s in the CoAP RFC it made sense to start there. The main developer on dimpl was wary, rightfully so, of adding many ciphers since that creates mess to support.

Will it support DTLS 1.2 CID (RFC 9146)?

Draft CID is in use right now. PR in progress.

Which implementation is used on the servers side? piondtls/GO, mbedtls/C, californium/Java, or?

dimpl/Rust

Will it support a “DTLS graceful shutdown” (sometimes useful for non breaking server updates).

I’ve add this to the roadmap because that makes sense! The FOTA side is fairly robust and will simply continue where it leaves off if the connection is interrupted during download.

Does it mean, when the device sends sensor data in e.g. a POST, it will also need to send a separate GET for the Shadow?

As you know keeping a connection open with push capabilities is expensive power and data wise. For devices that are sleepy, lion allows for devices to still receive commands in an atomic manor. All data is stored in CBOR format that way it can be translated easy to JSON to consuming services and stay CBOR for the embedded side.

Device push does work but requires frequent ping/keep alive to prevent NAT from closing out the connection (at least for cellular). It basically sends the same shadow with updates if there’s a diff on the server side.

Thanks for your questions @AchimKraus!

AchimKraus

About the supported cipher suites:
RFC 7252 is from 2014. The TLS group seems to prefer in the meantime the PSK-ECDHE variants (even if there are also arguments to still use PSK or ECDHE-ECDSA with RFC 7250).

jaredwolff

@AchimKraus is there any technical guidance/standard/info on those recommendations? Or is it more of a general “this is the current standard”. Edit: looks like RFC 9325 / BCP 195 is the latest guidance that I can find.

AchimKraus

I remember some discussions in the IETF TLS mailing list. One was
The future of external PSK in TLS 1.3.

AchimKraus

RFC9325 - 7.3 Forward Secrecy

The main pain-point of using PSK on its own is, that someone recording traffic may decrypt also past traffic, once the secret gets compromised. For quite a lot of system I don’t see that “pays-off”, so I would accept that “risk”.
For other systems that could be mitigated by using a HSM.
But in general users should be aware of the resulting security level.

jaredwolff

Understood. I think having PSK-ECDHE for more sensitive applications is definitely warranted. I’ll add it to the list of things to do. 😃