Legacy LightDB Stream

This example demonstrates using two pipelines to support legacy LightDB Stream behavior. These pipelines match on all CBOR and JSON data respectively. Both utilize the batch destination to support splitting a single message into multiple. For the CBOR pipeline, if data is split in step-1, the resulting messages will be delivered to the JSON pipeline as their content_type is set to application/json via the cbor-to-json transformer in step-0. In the JSON pipeline, if data is split in step-0, the resulting messages will be delivered to the same JSON pipeline.

The extract-timestamp and inject-path transformers are used in both pipelines to alter the data message prior to delivery to LightDB Stream.

Pipeline: Legacy LightDB Stream CBOR

filter:
  path: "*"
  content_type: application/cbor
steps:
  - name: step-0
    transformer:
      type: cbor-to-json
      version: v1
  - name: step-1
    destination:
      type: batch
      version: v1
  - name: step-2
    transformer:
      type: extract-timestamp
      version: v1
  - name: step-3
    transformer:
      type: inject-path
      version: v1
    destination:
      type: lightdb-stream
      version: v1

Pipeline: Legacy LightDB Stream JSON

filter:
  path: "*"
  content_type: application/json
steps:
  - name: step-0
    destination:
      type: batch
      version: v1
  - name: step-1
    transformer:
      type: extract-timestamp
      version: v1
  - name: step-2
    transformer:
      type: inject-path
      version: v1
    destination:
      type: lightdb-stream
      version: v1