Skip to main content

Golioth Stream Client

The Golioth platform supports time-series data streaming. The Golioth Firmware SDK includes functions for sending stream data, which will ultimately be routed to a destination based on the pipelines configured in the device's project.

The Golioth Firmware SDK includes functions for working with stream data.

Using Stream​

Stream functions can be used to send any number of time-series data values to the cloud with a single function call. A timestamp is automatically added to the data by the server and stored using the configuration found in the Pipeline for this path. This data will accumulate along with all previously received values.

Stream data uses Pipelines

Pipelines are used to route and optionally transform streaming data on the Golioth Platform. Please see the Pipelines section of the Golioth Docs for more information.

golioth_stream_set() sends data to any path using your choice of these content types: JSON, CBOR, and OCTET_STREAM for arbitrary binary data.

Includes​

#include <golioth/client.h>
#include <golioth/stream.h>

These headers make the Golioth client and Stream API functions available to your program.

Preparing the Data​

The data you want to send must first be prepared as an object. This will include one or more key-values pairs. Choose between JSON or CBOR format.

You Decide the Schema

Golioth doesn't enforce a schema for your data. You may write to the root path using an empty string "", or create "any/path/you/like".

You choose what keys to use with your key-value pairs, and you are free to use nested objects. The only reserved keys are t, ts, time and timestamp; each will be interpreted as a user-supplied timestamp. More on that later.

JSON​

JSON payloads may be formatted using simple snprintf() style string substitution or a library like cJSON. Here is a properly-formatted JSON example:

/* Create a valid JSON string */
char json_buf[] = "{\"name\":\"Golioth\",\"active\":true,\"data\":{\"reading\":1.337,\"calib\":42}}";

JSON is great for prototyping as it's human readable. When sending larger data sets, you may experience bandwidth savings by using CBOR instead.

CBOR​

The same data may be formatted as a serialized CBOR object. On the device this should be done using a library of your choice. The zcbor library is already available as a submodule dependency of the Golioth Firmware SDK.

/* Create serialized CBOR object */
uint8_t cbor_buf[] = { 0xA3, 0x64, 0x6E, 0x61, 0x6D, 0x65, 0x67, 0x47, 0x6F, 0x6C, 0x69, 0x6F,
0x74, 0x68, 0x66, 0x61, 0x63, 0x74, 0x69, 0x76, 0x65, 0xF5, 0x64, 0x64,
0x61, 0x74, 0x61, 0xA2, 0x67, 0x72, 0x65, 0x61, 0x64, 0x69, 0x6E, 0x67,
0xFB, 0x3F, 0xF5, 0x64, 0x5A, 0x1C, 0xAC, 0x08, 0x31, 0x65, 0x63, 0x61,
0x6C, 0x69, 0x62, 0x18, 0x2A };

While CBOR is not human readable, there are online tools like cbor.me that are useful for debugging.

Click to reveal a zcbor encoding example

Here is an example of using zcbor in canonical mode (ZCBOR_CANONICAL or in Zephyr: CONFIG_ZCBOR_CANONICAL) to encode a CBOR payload.

#include <zcbor_encode.h>

uint8_t cbor_payload[128] = {0};
bool my_bool = true;
double my_float = 1.337;
int my_int = 42;
char my_str[] = "Golioth";

ZCBOR_STATE_E(encoding_state, 8, cbor_payload, sizeof(cbor_payload), 0);
zcbor_map_start_encode(encoding_state, 3);
zcbor_tstr_put_lit(encoding_state, "name");
zcbor_tstr_put_term(encoding_state, my_str);
zcbor_tstr_put_lit(encoding_state, "active");
zcbor_bool_put(encoding_state, my_bool);
zcbor_tstr_put_lit(encoding_state, "data");
zcbor_map_start_encode(encoding_state, 2);
zcbor_tstr_put_lit(encoding_state, "reading");
zcbor_float64_put(encoding_state, my_float);
zcbor_tstr_put_lit(encoding_state, "calib");
zcbor_int32_put(encoding_state, my_int);
zcbor_map_end_encode(encoding_state, 2);
zcbor_map_end_encode(encoding_state, 3);

size_t cbor_payload_len = encoding_state->payload - cbor_payload;

// Use `cbor_payload` and `cbor_payload_len` in the Golioth function call
Comparing JSON and CBOR

For this example we see that the CBOR package is 53 bytes while the JSON representation is 68 bytes for the same data. This ratio will change based on how many key-value pairs are in the object, the type of the data, and even their values. In this example, the savings for numerous readings across multiple devices in a fleet over large time periods becomes substantial.

Sending stream data​

/* Send JSON object */
err = golioth_stream_set(client,
"sensor",
GOLIOTH_CONTENT_TYPE_JSON,
(const uint8_t *)json_buf,
strlen(json_buf),
NULL,
NULL);

/* Send CBOR object */
err = golioth_stream_set(client,
"sensor",
GOLIOTH_CONTENT_TYPE_CBOR,
cbor_buf,
sizeof(cbor_buf),
NULL,
NULL);

golioth_stream_set() does not block: the request is enqueued by the Golioth Client and the function returns immediately. Its return value only reports whether the request was queued; pass a callback (see below) to learn the result. The synchronous golioth_stream_set_sync() was removed in Golioth Firmware SDK v0.23.0. For payloads too large to send in one request, use golioth_stream_set_blockwise_sync(), which reads the data block by block from a callback and blocks until the upload completes.

Path Values and Pipelines

Specifying a path allows for custom routing of streaming data to a destination using Pipelines. For more information, please see the Pipelines section of the Golioth Docs.

In the above examples, we tell Golioth to set the "sensor" path using the enum for our desired content type, along with the json_buf/cbor_buf pointer and the length of that array. No callback is registered (NULL). When Golioth receives this data, the server will automatically add a timestamp based on when it was received.

Multiple Data Points and Nested Data

This operation can set multiple values stored in the object. The example also demonstrates data stored as a nested object. After these function calls, the Golioth servers will have a record of the data that looks like this:

{
"sensor": {
"active": true,
"data": {
"calib": 42,
"reading": 1.337
},
"name": "Golioth"
}
}

Using a callback​

When the request completes, an optional callback function may be run to process the result.

Calling the Stream Set Function with a Callback​

enum obj_type {
JSON_ASYNC,
CBOR_ASYNC
};

err = golioth_stream_set(client,
"sensor",
GOLIOTH_CONTENT_TYPE_JSON,
(const uint8_t *)json_buf,
strlen(json_buf),
obj_async_handler,
(void *)JSON_ASYNC);

err = golioth_stream_set(client,
"sensor",
GOLIOTH_CONTENT_TYPE_CBOR,
cbor_buf,
sizeof(cbor_buf),
obj_async_handler,
(void *)CBOR_ASYNC);

golioth_stream_set() sends data to a given path, but the request is enqueued by the Golioth Client so that program execution is not blocked. If a callback function is supplied, it will be called by the Golioth Client when the set operation is completed. An optional callback argument can be specified, this data will be available in the callback function.

In the above examples, we tell Golioth to set the "sensor" path using the enum for our desired content type, along with the pointer to the json_buf/cbor_buf array. The length of that array is supplied, along with with obj_async_handler as the callback function. An enum value is used as the callback argument.

Callback Function​

Callback functions need to follow the type specified by the function that registers them. In this case a golioth_set_cb_fn type callback function must be defined:

static void obj_async_handler(struct golioth_client *client,
enum golioth_status status,
const struct golioth_coap_rsp_code *coap_rsp_code,
const char *path,
void *arg) {
if (status != GOLIOTH_OK) {
// The set operation failed.
return;
}

// The set operation was successful!
switch((enum obj_type)arg) {
case JSON_ASYNC:
printf("JSON async set successful!\n");
break;
case CBOR_ASYNC:
printf("CBOR async set successful!\n");
break;
default:
printf("Async set successful, but arg value unrecognized\n");
}
return;
}

The result is passed to the callback in status. It is recommended that callbacks test for the GOLIOTH_OK status which indicates a successful operation. When status is GOLIOTH_ERR_COAP_RESPONSE, coap_rsp_code holds the non-success CoAP response code sent by the server; for other errors (for example GOLIOTH_ERR_TIMEOUT) no response was received and coap_rsp_code is NULL.

The same callback may be used by multiple requests. In this example, the arg value is used to determine which function led to this callback so that a different message may be printed.

Custom Timestamps​

By default, the server will automatically assign a timestamp to all received streaming data. However, in some cases you may want to assign your own timestamps

/* Create a valid JSON string that includes a timestamp */
char json_buf[] = "{\"name\":\"Golioth\","
"\"active\":true,"
"\"data\":{\"reading\":1.337,\"calib\":83},"
"\"time\":\"2023-08-09T08:27:49-05:00\"}";

Format your custom time as either Unix time or ISO 8601 format, and add it to your payload using any of the special key names: t, ts, time, or timestamp. When a timestamp key is provided, it will be used as the timestamp for the event itself, and will be removed from the object structure.

Timestamps for single or batch data

In both cases (automatic or custom timestamps), one root-level timestamp will be used for all of the data received in the same object. But in cases where you have more than one data payload to send to the same endpoint, batch data may be sent by including each reading as a sibling object, with a different timestamp as a root-member of that sibling object. For example, let's send three temperature readings taken one minute apart:

char json_batch[] = "[{\"temp\":21.5,\"time\":\"2023-08-09T08:27:49-05:00\"},"
"{\"temp\":22.0,\"time\":\"2023-08-09T08:28:49-05:00\"},"
"{\"temp\":22.5,\"time\":\"2023-08-09T08:29:49-05:00\"}]";
Configure timestamps in Pipelines

Pipelines may be used to instruct how and when the Golioth servers should use timestamps received in the Stream data payload. For more information, please see the Pipelines section of the Golioth Docs.

Resources​

Further documentation of the device SDK is available in the Golioth Firmware SDK Reference (Doxygen).