Prior art

Introduction

This is all about the air protocol, i.e. state-of-the-art low-range RF communication protocols, message encapsulation and sending interfaces. We focus on implementations based on the RFM69 library for Arduino by Felix Rusu but looking for LoRa support.

The document investigates the way how values are actually transmitted over the air and how they are encoded, especially when it comes to transmitting multiple values at once leading directly to the question of how much sender and receiver are coupled.

Serialization formats

As a general overview, see https://en.wikipedia.org/wiki/Comparison_of_data_serialization_formats

We will define multiple families of encoding/serialization schemes by enumerating some examples.

Family 1: Binary

Binary format, usually no message encapsulation, bare struct.

It is common to define structs and just send them over the air in their respective binary representation.

This kind of implementation gains a maximum of efficiency:

>>> import struct
>>> payload = struct.pack('!ff', 42.42, 99.99)
>>> payload
'B)\xae\x14B\xc7\xfa\xe1'
>>> len(payload)
8

The drawback is that sender and receiver are tightly coupled by implementing the same payload struct declaration in order to talk to each other.

Family 2: 8-bit clean

ASCII format, arbitrary message encapsulation.

To gain more flexibility, 8-bit clean transport protocols are used. On the pro side, arbitrary convenient and advanced line-based protocols can be designed on top of them. They are also easy to debug, especially when staring at payloads while receiving or forwarding messages through a serial interface.

TODO: Break out into different section. The drawback here is usually payload size, since the maximum packet payload size defined in RFM69.h#L35 is 61 bytes:

// to take advantage of the built in AES/CRC we want to limit the frame size
// to the internal FIFO size (66 bytes - 3 bytes overhead - 2 bytes crc)
#define RF69_MAX_DATA_LEN       61

http://www.airspayce.com/mikem/arduino/RadioHead/RH__RF69_8h_source.html:

// Max number of octets the RH_RF69 Rx and Tx FIFOs can hold
#define RH_RF69_FIFO_SIZE 66

// Maximum encryptable payload length the RF69 can support
#define RH_RF69_MAX_ENCRYPTABLE_PAYLOAD_LEN 64

http://www.airspayce.com/mikem/arduino/RadioHead/RH__RF95_8h_source.html:

// Max number of octets the LORA Rx/Tx FIFO can hold
#define RH_RF95_FIFO_SIZE 255

Current implementations

As far as we can see, there is no standardized way of how to actually talk radio across different open sensor network implementations found in the wild / on github.

RFduino

… has a nice interface, see Temperature.ino:

#include <RFduinoBLE.h>

void setup() {
    // this is the data we want to appear in the advertisement
    // (the deviceName length plus the advertisement length must be <= 18 bytes)
    RFduinoBLE.advertisementData = "temp";

    // start the BLE stack
    RFduinoBLE.begin();
}

void loop() {
    // sample once per second
    RFduino_ULPDelay( SECONDS(1) );

    // get a cpu temperature sample
    // degrees c (-198.00 to +260.00)
    // degrees f (-128.00 to +127.00)
    float temp = RFduino_temperature(CELSIUS);

    // send the sample to the iPhone
    RFduinoBLE.sendFloat(temp);
}

See also

RFduinoBLE.cpp#L94

This is how some RFM69 examples do it

Type: Binary struct

# declare struct
typedef struct {
    int           nodeId; //store this nodeId
    unsigned long uptime; //uptime in ms
    float         temp;   //temperature maybe?
} Payload;
Payload theData;

# fill struct
theData.nodeId = NODEID;
theData.uptime = millis();
theData.temp = 91.23; //it's hot!

# transmit struct
radio.sendWithRetry(GATEWAYID, (const void*)(&theData), sizeof(theData))

# receive struct
if (radio.receiveDone()) {
    theData = *(Payload*)radio.DATA;
    Serial.print(theData.nodeId);
    Serial.print(theData.uptime);
    Serial.print(theData.temp);
}
Here’s how an emon sensor node does it

Type: Binary struct

emonTH_LPL.ino#L27:

// RFM12B RF payload datastructure
typedef struct {
    int temp;
    int temp_external;
    int humidity;
    int battery;
} Payload;
Payload emonth;

emonTxV3_LPL.ino#L28:

typedef struct { int power1, power2, power3, power4, vrms; } PayloadTX;
PayloadTX emontx;

emonTxV3_4_3Phase_Voltage_LPL.ino#L239:

// neat way of packaging data for RF comms
// Include all the variables that are desired,
// ensure the same struct is used to receive.
// The maximum size is 60 Bytes
typedef struct { int power1, power2, power3, va1, va2, va3, Vrms, pnum; } PayloadTX;

// create an instance
PayloadTX emontx;
UniMote of CuPID Controls

Type: 8-bit clean

CuPID Controls: Networked, flexible control and monitoring for any application.

Colin Reese of Interface Innovations designed an overlay protocol over an 8-bit clean data channel for the CuPID Controls system. Over that protocol, he is able to completely augment and control the program running on the target by using user-programmable variables.

The wire message format is simple:

~command;arg1;arg2;arg3

The list of features is really impressive:

  • Create an ATMega sketch that allows read/write of all available Moteino IO in all allowable formats by change of variable values, i.e. no code change required

  • Allow OneWire read operations on all digital IO (only for DS18B20s here)

  • Configure IO for read and report (broadcast) on remotely adjustable schedule

  • Configure channel control, with configurable setpoint and process values and positive and negative feedback

  • Set all key program parameters remotely (via radio) and locally (on serial)

  • Adjustable, metadata-containing, report format for IO, channels, and system parameters

  • Save system configuration to EEPROM and restore upon resume after loss of power

See also

The target can be introspected - a picture says a thousand words:

~listparams
Command character received
NODEID:0,
GATEWAYID:0,
NETWORKID:0,
LOOPPERIOD:2000,
SLEEPMODE:0,
iomode:[0,0,0,0,0,3,0,3,0,0,0,0,0],
ioenabled:[0,0,0,0,0,0,0,0,0,0,0,0,0],
ioreadfreq:[10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000,10000],
ioreportenabled:[0,0,0,0,0,0,0,0,0,0,0,0,0],
ioreportfreq:[0,0,0,0,0,0,0,0,0,0,0,0,0],
chanenabled:[0,0,0,0,0,0,0,0],
chanmode:[0,0,0,0,0,0,0,0],
chanposfdbk:[0,0,0,0,0,0,0,0],
channegfdbk:[-1,0,0,0,0,0,0,0],
chandeadband:[0,0,0,0,0,0,0,0],
chanpvindex:[5,0,0,0,0,0,0,0],
chansv:[15.00,0.00,0.00,0.00,0.00,0.00,0.00,0.00]

The workhorse command is:

~modparam;paramname;index;value

As we are currently interested in sending multiple measurement values (not control commands), here is the place we’ve found:

uni_mote_2p75.ino#L1736:

void sendIOMessage(byte ionum, byte mode, float value) {
    // Initialize send string

    int sendlength = 61;  // default
    if (mode == 0 || mode == 1 ) { // for integer values
        sendlength = 30;
        sprintf(buff, "ionum:%02d,iomode:%02d,ioval:%04d", ionum, mode, value);
        sendWithSerialNotify(GATEWAYID, buff, sendlength, ~serialrfecho);
    }
    else if (mode == 2) { // for float values
        int wholePart = value;
        long fractPart = (value - wholePart) * 10000;
        sendlength = 34;
        sprintf(buff, "ionum:%02d,iomode:%02d,ioval:%03d.%04d", ionum, mode, wholePart, fractPart);
        sendWithSerialNotify(GATEWAYID, buff, sendlength, serialrfecho);
    }
}

serialhandler.py#L276 ff.:

# [...]
# Command responses, including value requests
if 'cmd' in datadict:
    # [...]
    for key in datadict:
        thetime = pilib.gettimestring()
        if key in ['iov', 'iov2', 'iov3', 'pv', 'pv2', 'sv', 'sv2', 'iomd', 'ioen', 'iordf', 'iorpf', 'chen', 'chmd', 'chnf', 'chpf', 'chdb', 'chsv', 'chsv2', 'chpv', 'chpv2']:

# This is for values that are reported by the node
elif 'ioval' in datadict:

elif 'owdev' in datadict:

elif 'chan' in datadict:

elif 'scalevalue' in datadict:

# [...]

Family 3: 8-bit clean, container

Bencode

Jean-Claude Wippler already experimented with Bencode and implemented the fine EmBencode library for Arduino: - http://jeelabs.org/2012/06/22/structured-data/ - http://jeelabs.org/?s=bencode

It’s also mentioned at Serialization for data exchange between micro processor and the web.

Outlook:

You have chosen a nice and simple de facto data encoding protocol. I hope though that you will be
using the extended bencode versions that also allow bool and float to be part of the data… I’m using
them a lot for temperature and humidity for instance!

– http://jeelabs.org/2012/06/22/structured-data/#comment-5591

Discussions

API proposal

How about?

#include <beradio.h>

int NODE_ID = 1;
char * PROFILE_ID = "h1";

void setup() {
    app_initialize(...);
}

void loop() {
    BERadioMessage message(NODE_ID, PROFILE_ID);
    message.temperature(4, 21.63, 19.25, 10.92, 13.54);
    message.voltage(4, 21.63, 19.25, 10.92, 13.54);
    message.send();
    delay(1000);
}

Inspiration

Projects

Jee Labs

computourist

A RFM69 based sensors and MQTT gateway - https://github.com/computourist/RFM69-MQTT-client

Sensor node
Gateway
Communication
Outlook

ULPNode

Serialization

Bencode

Basics
More Bencode

Research

Technology

RFM generations

Node hardware

Bus systems

The gateways

Other serial-to-X forwarders

MySensors

Jet

A dataflow framework and server for multi-node embedded systems - https://github.com/jeelabs/jet

LoRa

TheThingsNetwork

C++ interfaces

Projects

CuPID

UniMote

Command-/channel based overlay protocol over RF

SNAP

SNAP - an embedded network application platform

The Things Network

More

SBMP

Simple Binary Messaging Protocol - USART protocol for microcontrollers - https://github.com/MightyPork/sbmp - https://github.com/MightyPork/sbmp/tree/master/spec - https://github.com/MightyPork/sbmp/blob/master/spec/DATAGRAMS.md - https://github.com/MightyPork/sbmp/blob/master/spec/FRAMING_LAYER.md -