Replicate

Replicate is an AI platform that makes machine learning accessible to every software developer by allowing for running and fine-tuning open source and custom models at scale. Existing models on Replicate suppport use cases that range from video generation to optical character recognition (OCR).

The following example demonstrates captioning images streamed to Golioth using the Salesforce BLIP model. The PNG image is submitted in the webhook destination request by encoding as a Data URL, then embedding in a JSON object.

Predictions on Replicate are asynchronous, meaning that the result is not accessible directly in Pipelines. Results can be viewed in the Replicate console, or a webhook URL can be included in the destination body to instruct webhook to deliver the result to the specified URL after the prediction has completed.

Reminder

Make sure to create a secret named REPLICATE_TOKEN in the format Bearer <token>.

Pipeline: Replicate Caption

filter:
  path: "/images"
steps:
  - name: png
    transformer:
      type: change-content-type
      version: v1
      parameters:
        content_type: image/png
  - name: url
    transformer:
      type: data-url
      version: v1
  - name: embed
    transformer:
      type: embed-in-json
      version: v1
      parameters:
        key: image
  - name: create-payload
    transformer:
      type: json-patch
      version: v1
      parameters:
        patch: |
          [
            {"op": "add", "path": "/version", "value": "2e1dddc8621f72155f24cf2e0adbde548458d3cab9f00c0139eea840d0ac4746"},
            {"op": "add", "path": "/input", "value": {"image": ""}},
            {"op": "move", "from": "/image", "path": "/input/image"}
          ]
  - name: predict
    destination:
      type: webhook
      version: v1
      parameters:
        url: https://api.replicate.com/v1/predictions
        headers:
          Authorization: $REPLICATE_TOKEN