OpenAI

OpenAI is a platform that enables developers to build and scale AI experiences powered by industry-leading models and tools. OpenAI's suite of proprietary models support text generation, speech-to-text, vision, and more.

The following example demonstrates leveraging OpenAI's GPT-4o mini model via the Pipelines webhook transformer to describe an image streamed to Golioth. The image is embedded as a Data URL alongside the chosen model and a prompt. The resulting description is passed to a Slack webhook via the webhook destination.

Reminder

Make sure to create a secret named OPENAI_TOKEN in the format Bearer <token>. In this example a SLACK_WEBHOOK secret is also required.

Pipeline: OpenAI Describe Image

filter:
  path: "/image"
steps:
  - name: jpeg
    transformer:
      type: change-content-type
      version: v1
      parameters:
        content_type: image/jpeg
  - name: url
    transformer:
      type: data-url
      version: v1
  - name: embed
    transformer:
      type: embed-in-json
      version: v1
      parameters:
        key: image
  - name: create-payload
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {
              "op": "add",
              "path": "/model",
              "value": "gpt-4o-mini"
            },
            {
              "op": "add",
              "path": "/messages",
              "value": [
                {
                  "role": "user",
                  "content": [
                    {
                      "type": "text",
                      "text": "What's in this image?"
                    },
                    {
                      "type": "image_url",
                      "image_url": {
                        "url": ""
                      }
                    }
                  ]
                }
              ]
            },
            {
              "op": "move",
              "from": "/image",
              "path": "/messages/0/content/1/image_url/url"
            },
            {
              "op": "remove",
              "path": "/image"
            }
          ]
  - name: explain
    transformer:
      type: webhook
      version: v1
      parameters:
        url: https://api.openai.com/v1/chat/completions
        headers:
          Authorization: $OPENAI_TOKEN
  - name: parse-payload
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "add", "path": "/text", "value": ""},
            {"op": "move", "from": "/choices/0/message/content", "path": "/text"}
          ]  
  - name: send-webhook
    destination:
      type: webhook
      version: v1
      parameters:
        url: $SLACK_WEBHOOK