Serialization efficiency

This compares space efficiency between different serialization formats.

Workbench

Setup

Let’s define a standard message payload:

>>> payload = OrderedDict()
>>> payload['#'] = 999
>>> payload['_'] = 'h1'
>>> payload.update({
...            'h1': 488.0,
...            'h2': 572.0,
...            't1': 21.63,
...            't2': 19.25,
...            't3': 10.92,
...            't4': 13.54,
...            'w1': 106.77})

Shootout

Unqualified

When not sending any key/attribute information, we gain a maximum of space efficiency, but lose schema information completely. So the receiver must perfectly know about the values we are sending. We almost can’t leave out or add new values.

Binary

Binary encoding is obviously on top of the list regarding payload size.

>>> import struct
>>> payload_binary = struct.pack(
...     '!Iccfffffff',
...     payload['#'],  payload['_'][0:1].encode(), payload['_'][1:2].encode(),
...     payload['t1'], payload['t2'],   payload['t3'], payload['t4'],
...     payload['h1'], payload['h2'],
...     payload['w1'])
>>> payload_binary
b'\x00\x00\x03\xe7h1A\xad\n=A\x9a\x00\x00A.\xb8RAX\xa3\xd7C\xf4\x00\x00D\x0f\x00\x00B\xd5\x8a='
>>> len(payload_binary)
34

See also

https://docs.python.org/2/library/struct.html

CSVp

The plain version of CSV. Just magic values.

>>> payload_values = [str(value) for value in payload.values()]
>>> payload_csv = ','.join(payload_values)
>>> payload_csv
'999,h1,488.0,572.0,21.63,19.25,10.92,13.54,106.77'
>>> len(payload_csv)
49

Qualified

When sending at least a single-letter identifier describing the sensor (values), we can deduce a lot more information from the message payload.

BERadio

BERadio applies scaling to get rid of float values, single-item compression and encodes the nodeid as integer. It has knowledge about how to apply different scalings and conversions by incorporating profiles to avoid sending unencodable types, e.g. floats.

It is the clear winner of encodings retaining readability through staying ASCII while still including key/attribute information.

Build message:

>>> from beradio.message import BERadioMessage
>>> message = BERadioMessage(999)
>>> message.temperature(21.63, 19.25, 10.92, 13.54)
>>> message.humidity(488.0, 572.0)
>>> message.weight(106.77)

Serialize message:

>>> bytes(message)
b'd1:#i999e1:_2:h11:hli48800ei57200ee1:tli2163ei1925ei1092ei1354ee1:wi10677ee'
>>> len(bytes(message))
75
CSVq

A qualified version of CSV. Prefixes items with shortcut attribute name.

>>> entries = [key + ':' + str(value) for key, value in payload.items()]
>>> payload_csv = ','.join(entries)
>>> payload_csv
'#:999,_:h1,h1:488.0,h2:572.0,t1:21.63,t2:19.25,t3:10.92,t4:13.54,w1:106.77'
>>> len(payload_csv)
74
Bencode

Unfortunately, Bencode is unable to encode float values:

>>> import bencode
>>> bencode.bencode(payload)  
Traceback (most recent call last):
...
KeyError: <class 'float'>

After converting to int values with uniform scaling:

>>> payload_integers = dict([key, int(value * 100) if isinstance(value, float) else value] for key, value in payload.items())
>>> message = bencode.bencode(payload_integers)
>>> message
b'd1:#i999e1:_2:h12:h1i48800e2:h2i57200e2:t1i2163e2:t2i1925e2:t3i1092e2:t4i1354e2:w1i10677ee'
>>> len(message)
90
YAML
>>> import yaml
>>> message = yaml.dump(dict(payload))
>>> message
"'#': 999\n_: h1\nh1: 488.0\nh2: 572.0\nt1: 21.63\nt2: 19.25\nt3: 10.92\nt4: 13.54\nw1: 106.77\n"
>>> len(message)
86
MessagePack

http://msgpack.org/

>>> import umsgpack
>>> message = umsgpack.dumps(payload)
>>> message
b'\x89\xa1#\xcd\x03\xe7\xa1_\xa2h1\xa2h1\xcb@~\x80\x00\x00\x00\x00\x00\xa2h2\xcb@\x81\xe0\x00\x00\x00\x00\x00\xa2t1\xcb@5\xa1G\xae\x14z\xe1\xa2t2\xcb@3@\x00\x00\x00\x00\x00\xa2t3\xcb@%\xd7\n=p\xa3\xd7\xa2t4\xcb@+\x14z\xe1G\xae\x14\xa2w1\xcb@Z\xb1G\xae\x14z\xe1'
>>> len(message)
95
JSON
>>> import json
>>> message = json.dumps(payload)
>>> message
'{"#": 999, "_": "h1", "h1": 488.0, "h2": 572.0, "t1": 21.63, "t2": 19.25, "t3": 10.92, "t4": 13.54, "w1": 106.77}'
>>> len(message)
113

Outlook

Marshallers suitable for embedded use

Classic

Modern

From some discussion about the article RFM69 to MQTT gateway using ESP8266 on Martin Harizanov’s weblog, we should follow to the article Serializing data from IoT nodes by Johan Kanflo. It greatly reflects the zeitgeist and also has pointers to

  • SMILE, a binary serialization of generic JSON data model and

  • UBJSON (UBJSON at Wikipedia), “the universally compatible format specification for binary JSON”.

Other marshallers

Compression

For reducing payload size, there’s also compression, which might come handy.