Menu
Home Videos BlogTech ProjectsAiHardwareSoftware Shop Digital Designs Physical Art Layered Maps Chinese Paper Cuttings About Contact Cart (0)

The third radio personality is an OpenThread Radio Co-Processor (RCP) image, with the Thread Border Router (OTBR) stack hosted externally. This chapter covers migrating that external hosting from Docker to the router itself, commissioning a real Matter-over-Thread device, and an extended crash investigation that used live SWD/J-Link debugging to rule out several plausible causes and narrow in on one specific, still-unconfirmed hypothesis.

Why the Docker + TCP-bridge architecture was replaced

The earlier architecture ran OTBR as w1700k-otbr and w1700k-otbr-rcp-proxy Docker containers on a separate Docker host, with the MG21’s UART bridged over TCP from the router (a serial-bridge-arm64 process listening on 10.24.1.237:6638) and reaching the OTBR container through a socat PTY proxy. In live use, otbr-agent crashed repeatedly with:

[W] P-RadioSpinel-: radio tx timeout
[C] P-RadioSpinel-: Failed to communicate with RCP - no response from RCP during initialization
[C] Platform------: HandleRcpTimeout() at radio_spinel.cpp:2035: RadioSpinelNoResponse

which cascaded into the socat proxy seeing “PTY closed,” restarting, and reinstalling itself via apk on every restart — sometimes racing with otbr-agent’s own restart attempts. A shell watchdog deployed to auto-restart otbr-agent every 2 seconds made this measurably worse, racing against the proxy’s own ~2-3 second reinit cycle and producing a faster crash-loop than doing nothing; it was killed. The reliable recovery technique throughout was a hardware reset of the MG21 via AN7581 GPIO536 (active-low RESETn), asserted and released directly through sysfs.

The router’s own package manager, opkg, was found to be completely non-functional on the existing custom ubi2 image (Error loading shared library libubox.so.20240329), ruling out any native opkg-based install. A cross-compiled static otbr-agent via Alpine + QEMU was started but abandoned in favor of a cleaner option: reflashing the router to an official OpenWrt SNAPSHOT build, whose default package manager is Alpine’s apk-tools. The router’s existing custom image turned out to already be built from the same upstream snapshot source tree — the ubi2 label just describes the flash-partition install method, and its busybox-only taint flag explained the missing shared libraries, not a different underlying driver stack. This made reflashing a materially lower-risk decision than it first appeared.

A config-preserving sysupgrade -v (after a sysupgrade -b backup) brought the router up as DISTRIB_RELEASE='SNAPSHOT', r36216-f0d3e332e5, kernel 6.18.44, with apk working cleanly across 11,158 packages.

Native OTBR install and configuration

apk add openthread-br luci-app-openthread luci uhttpd installed cleanly. openthread-br ships only raw binaries and a netifd protocol handler (lib/netifd/proto/openthread.sh) — no init.d service — so it was configured the idiomatic OpenWrt way, as a netifd interface:

config interface 'thread'
    option proto 'openthread'
    option device 'wpan0'
    option backbone_network 'lan'
    option radio_url 'spinel+hdlc+uart:///dev/ttyS1?uart-baudrate=460800'
    option verbose '1'
    option dataset '<hex TLV>'

A brand-new Thread network was formed (channel 24, a fresh PAN ID and network key, since the RCP’s own dataset state does not survive an application reflash). One genuinely reusable finding from this phase: the openthread netifd proto script does not automatically add its interface to a firewall zone. Traffic between br-lan (phones, Home Assistant) and wpan0 (the Thread mesh) was very likely being silently dropped by OpenWrt’s default deny-between-zones policy, which was almost certainly why devices could join the mesh at the radio level while Matter commissioning still failed to complete. Fixed with:

uci add_list firewall.@zone[0].network='thread'
uci commit firewall
/etc/init.d/firewall restart

Commissioning the IKEA ALPSTUGA

An IKEA ALPSTUGA Matter-over-Thread smart plug was paired through Apple Home. The first attempt stuck at “Setting Up” — diagnosis found the underlying Thread/mDNS layer was healthy, but the device had retained stale partition state from a previous stuck attempt (it kept broadcasting MLE Advertisements as if leading its own partition, rather than sending fresh Parent Requests), and this survived multiple power-cycles. A genuine factory reset (not a power-cycle, and not “Remove Accessory” from Apple Home — which is a no-op against an unreachable device, since its removal flow tries to tell the device to leave over the network it can’t reach) cleared the state; the device generated a fresh identity, joined as a Thread child, was promoted to router, and completed Matter commissioning, visible as _matterc._udp and _matter._tcp SRP registrations.

A related transient issue: the border router itself sometimes attached as a Thread child under the ALPSTUGA rather than leading its own partition, which silently broke SRP hosting and Border Routing (a Child cannot publish routes into Thread Network Data). Fixed each time with ot-ctl state router, forcing promotion.

The RadioSpinelNoResponse crash investigation

Even after the full native-OTBR rebuild, otbr-agent continued periodically crashing with the identical radio tx timeoutRadioSpinelNoResponse signature — proving the Docker/TCP/socat layer was never the root cause of this specific fault, even though replacing it was a real improvement for other reasons (no PTY-proxy crash cascades, no remote-host dependency, working package manager, LuCI).

A baud-rate/UART-driver bug, confirmed but unrelated

An attempt to test whether a lower RCP baud rate (115200 instead of 460800) would improve stability required flashing an experimental firmware build, which surfaced a genuine, separate kernel driver bug: asking this OpenWrt kernel’s airoha,en7523-uart driver for a nominal “115200” baud produced garbage, while asking for “230400” produced the exact correct result — because the driver’s clock-divisor logic for this SoC actually halves whatever it’s told, a discrepancy independently corroborated by an in-progress upstream Linux kernel patch series (“serial: 8250: Add AN7581 UART support”) that explicitly describes this chip’s UART needing baud values scaled by a factor of 2.

The 115200-baud firmware itself, once flashed and brought up (with radio_url correspondingly compensated to uart-baudrate=230400), made the crash dramatically worse — a continuous 6-8 second crash loop rather than multi-minute intervals. This was reverted back to the original known-good 460800-baud firmware. A separate check confirmed 460800 itself needs no such compensation (921600 produced an identical result and a driver warning about being unable to achieve the requested rate). The durable, reusable finding from this detour: UART DFU flashing of the MG21 works over the existing project script without any SWD/J-Link hardware, and this specific router/kernel combination needs “230400” requested to reliably talk to the bootloader’s real 115200 baud protocol.

The actual fault: a stack-buffer overflow in otPlatUartSend

Live log correlation established that crashes were not periodic, and every captured crash was immediately preceded by the ALPSTUGA retransmitting the same Matter UDP traffic with Message Reliability Protocol exponential backoff — i.e., it wasn’t getting acknowledged. Routing and RF interference were both checked and ruled out. This pointed at the host-to-RCP transmit direction specifically.

The bug was in the shared Simplicity SDK’s openthread/platform-abstraction/efr32/iostream_uart.c, in this project’s own pre-existing patch to otPlatUartSend() (originally added to work around the W1700K USART path dropping the first byte of a DMA burst, by prepending a sacrificial 0x7e HDLC flag):

{ uint8_t txBuf[256]; txBuf[0] = 0x7e; memcpy(&txBuf[1], aBuf, aBufLength); status = sl_iostream_write(sl_iostream_vcom_handle, txBuf, (uint16_t)(aBufLength + 1)); }

aBufLength is an unbounded uint16_t, and OpenThread’s HDLC transmit chunk size defaults to 2048 bytes — meaning a large enough HDLC-escaped frame could memcpy up to roughly 1.8 KB past the end of a 256-byte stack buffer. Small frames (bare MLE advertisements) never triggered it, consistent with the crash correlating to real application-level Matter traffic rather than idle mesh maintenance. The fix:

static uint8_t txBuf[2049];
uint16_t       len = aBufLength;
if (len > (uint16_t)(sizeof(txBuf) - 1)) { len = (uint16_t)(sizeof(txBuf) - 1); }
txBuf[0] = 0x7e;
memcpy(&txBuf[1], aBuf, len);
status = sl_iostream_write(sl_iostream_vcom_handle, txBuf, (uint16_t)(len + 1));

After this fix, the device survived several button presses that previously crashed it almost immediately — a measurable improvement, explicitly confirmed by the user as “better than before” — but it still eventually crashed again with the identical signature. A kernel serial-driver error-counter check (fe/brk/oe in /proc/tty/driver/serial) taken immediately before and after a live reproduced crash showed zero change, directly exonerating the physical UART link for this remaining fault.

With the fix in place and the crash still occurring, a J-Link/SWD probe already connected to the MG21 (a BRD4002A Rev. A06 Wireless Pro Kit) allowed direct observation of the RCP’s CPU state during a live crash, rather than just host-side log inference.

A reactive approach — poll logs, then connect and halt — proved too slow: by the time the debugger halted the core, otbr-agent had already recovered and resumed normal operation, with all Cortex-M33 fault-status registers reading clean. Proactive breakpoints were then set on every fault vector (HardFault, BusFault, UsageFault, MemManage, NMI) and the reset vector before reproducing the crash live. None of the six breakpoints fired, even though a real crash/restart occurred during the observation window — independently corroborated via lsof on the still-connected GDB client and the J-Link GDB server’s own log, which showed no server-side activity after the breakpoints were armed. This is the most significant confirmed finding of the investigation: the RCP’s ARM Cortex-M33 core never faults and never resets during a live-reproduced crash.

Reading OpenThread’s own source clarified what “radio tx timeout” actually means: not a generic command timeout, but specifically a 5-second wait (OPENTHREAD_SPINEL_CONFIG_RCP_TX_WAIT_TIME_SECS) for a TransmitDone event from the radio peripheral, after the transmit command itself has already been accepted over Spinel. A live SWD read of the HFXO status register (0x4000C058) at idle returned 0x0 — HFXO is fully powered down between transmissions and must cold-relock before every single radio transmit. This connects back to the project’s own pre-existing HFXO_E301 errata workaround (described in the BLE and Zigbee chapters): its own source comments record that removing the unsafe DISONDEMAND transition also removed the “fast path” for subsequent HFXO relocks, meaning every relock after the first one at boot goes through a slower recalibration path. The leading, still-unconfirmed hypothesis is that under sustained rapid transmit activity, this slower relock path occasionally exceeds the 5-second wait window or genuinely stalls, producing exactly the observed symptom without ever faulting the CPU.

The attempt to directly capture the HFXO register’s value during a live stall was cut short by a debugging-tooling incident: a stray, not-cleanly-terminated earlier GDB session, combined with a new polling session, left the MG21 in a state worse than the bug under investigation — every subsequent otbr-agent restart failed its very first Spinel command, even after all host-side GDB and J-Link processes were force-killed. Only a GPIO536 hardware reset pulse recovered it, fully restoring Thread role, mesh reattachment, and Matter/SRP registration. The durable lesson: a GPIO536 reset pulse should be treated as a required cleanup step after any SWD/J-Link session on this board, not merely an emergency-recovery tool — killing the host-side debugger process is not sufficient to guarantee the target chip releases cleanly.

The buffer-overflow fix was made durable by patching the project’s own tools/build-thread-rcp.sh regeneration script directly, so a clean slc generate produces the corrected, bounds-checked version from the start, rather than depending on a by-hand edit to the shared, unversioned Simplicity SDK install path.

Verification and open items

Confirmed: native otbr-agent runs directly against /dev/ttyS1 at 460800 baud via a netifd interface, with mdnsd, LuCI, and uhttpd all working via apk; the old Docker OTBR stack is fully decommissioned; the firewall zone gap is fixed; the ALPSTUGA is commissioned and controllable from Home Assistant; UART DFU flashing works end-to-end on this router/kernel with the 230400-for-bootloader workaround; a real stack-buffer-overflow bug is fixed and made durable in the build script; the physical UART link is exonerated for the remaining crash; and the RCP’s CPU is confirmed, via live SWD breakpoints on every fault vector, to never fault or reset during a live-reproduced crash.

Not confirmed: the root cause of the remaining RadioSpinelNoResponse crash — the HFXO relock hypothesis is well-evidenced but the HFXO status register’s value during an actual live stall was never captured; whether the buffer-overflow fix addressed the same fault partially or a different, now-resolved fault entirely; long-term stability beyond a few minutes of active use; and whether commissioning is stable enough for everyday, unattended household use as opposed to active debugging sessions.