Binary Data

While sending in a semi-structured format, whether text-based with JSON or binary with CBOR, can allow for parsing and transformation without any pre-communicated structure, it also incurs additional overhead. Structured data formats, such as Protocol Buffers, packed C structures, or custom binary encoding, can be useful in constrained environments and can lead to significant bandwidth savings, especially in high volume scenarios.

Many destinations support arbitrary data content types, making it possible to deliver raw structured data from devices to the external service. This example inlcudes a minimal pipeline that ingests binary data (application/octet-stream) from devices and delivers it to a custom Webhook data destination that requires authentication via an API token.

Reminder

Make sure to create a secret named EXAMPLE_API_TOKEN to use this example, or replace with the appropriate authentication mechanism for your webhook target.

Pipeline: Simple Binary Webhook

filter:
  path: "*"
  content_type: application/octet-stream
steps:
  - name: step-0
    destination:
      type: webhook
      version: v1
      parameters:
        url: https://webhook.example.com
        headers:
          x-api-token: $EXAMPLE_API_TOKEN