Volga webhook integration

Volga messages can be forwarded to webhook URLs by configuring webhooks in the volga configuration. Each webhook configured connects a set of Volga topics to a webhook URL. Please see the Volga reference for a list of topics and their locations. Typical topics to forward would be alerts and audit logs.

This feature provides a convenient way of forwarding topics with minimal configuration. For more complicated scenarios, consider using the volga-webhooks application, which can be deployed to edge sites and offers more flexibility in terms of configuration.

Configuration

Example 1 - Forwarding system:alerts from the Control Tower

A typical scenario might be to forward messages on system:alerts to a Slack channel. After setting up Slack to receive webhooks (refer to the Slack documentation for more information) you will be given a webhook URL on the form https://hooks.slack.com/services/x/y/z. Data posted to this URL needs to come in the form of a JSON message with a field called text:

{
  "text": "Hello!"
}

Setting this up using Volga webhooks requires just a few fields to be configured. However, since requests to the Slack URL require no authentication, the URL itself should be considered secret, and the first step is to store it in a Strongbox vault:

supctl create strongbox vaults <<EOF
name: webhooks
EOF

supctl create strongbox vaults webhooks secrets <<EOF
name: urls
data:
  slack: 'https://hooks.slack.com/services/x/y/z'
EOF

See the Secrets Management How-to for further information on managing secrets.

With the secret in place, the Volga webhook can be created:

supctl create volga webhooks <<EOF
name: slack-alerts
site-placement:
  sites:
    - control-tower
topics:
  - system:alerts
url: '\${url}'
data: '{"text": "\${MSG_PAYLOAD}"}'
data-format: json
variables:
  - name: url
    value-from-vault-secret:
      vault: webhooks
      secret: urls
      key: slack
EOF

Here is a breakdown of the configuration:

• name - An identifier for this Volga webhook.
• site-placement - Where to run the webhook. In this example, it is going to run at the Control Tower only.
• topics - The Volga topics to forward.
• url - The URL to forward messages to. In this example, the URL is set to the variable url which will be defined below to refer to a Strongbox secret.
• data - The POST data to be sent to the webhook URL. The variable MSG_PAYLOAD is used to place the payload of the Volga message in the text field of the JSON object.
• data-format- Setting this to json ensures that when the data field is evaluated, all variable expansions are escaped in such a way that they can be placed within JSON strings. This is important since system:alerts is a JSON topic and its payload will contain JSON that would otherwise break the JSON format of the POST data.
• variables - The variables needed by the above fields. In this example, the only variable needed is url, representing the Slack URL stored in Strongbox. Note that there is no need to define the MSG_PAYLOAD variable. It is a built-in variable referencing the Volga message payload.

Message forwarding will begin as soon as the configuration is submitted.

Example 2 - Forwarding application logs from edge sites

Using the site-placement/application-deployments configuration, a webhook definition can be distributed to whichever sites are currently running a particular set of applications, essentially having the webhook configuration follow an application around. Combining this with a topic-patterns entry that matches application log topics it is possible to get all application logs forwarded to a webhook URL.

Assuming there is an application called my-app, deployed by the application-deployment my-dep, and the same Slack URL as in the example above, the first step is to update the Strongbox vault to make sure it is distributed to all relevant sites:

supctl merge strongbox vaults webhooks <<EOF
distribute:
  deployments:
    - my-dep
EOF

Once this is done, the webhook can be created like this:

supctl create volga webhooks <<EOF
name: my-app-logs
site-placement:
  application-deployments:
    - my-dep
topic-patterns:
  - ^system:container-logs:my-app\..*
url: '\${url}'
data: '{"text": "\${MSG_PAYLOAD}"}'
data-format: json
variables:
  - name: url
    value-from-vault-secret:
      vault: webhooks
      secret: urls
      key: slack
EOF

Compared to the previous configuration, there are only two important differences:

• site-placement - Just like the vault, the webhook will follow my-dep.
• topic-patterns - Rather than a hard-coded topic name, you can provide a list of regular expressions to select topics. In this example, a single expression is used to select all topics belonging to my-app, but it is of course possible to be more specific and match service and container names too.

Note that these two parameters are both lists. You can track multiple deployments, and you can use multiple regular expressions to find topics. Another thing they have in common is that they reference data that may change over time. Any changes, such as new sites in the application-deployment or new topics matched by topic-patterns are automatically handled by the system, i.e., the webhook configuration will automatically be sent to new sites as needed, and new matching topics will be subscribed to as soon as they are created.

Headers and authentication

As mentioned, Slack webhooks require no authentication, but other services might, and those will typically require an authentication header. Headers are added using the headers list:

supctl merge volga webhooks my-webhook <<EOF
headers:
  - name: Authorization
    value: 'Bearer \${token}'
EOF

This example assumes that token is a variable referencing a Strongbox secret, just like the url variable in the initial example. Headers can of course also contain simple constant values, and you are free to add as many as you like, such as Content-Type, User-Agent, etc.

JSON Pointer variables

Variables can also be defined as JSON Pointers (RFC 6901) that extract fields from the Volga message. When your topic format is JSON, this can be useful to access fields inside the payload of the message. The predefined MSG_PAYLOAD variable used above is just a JSON pointer with the value /payload.

A slightly more complex example could look like this:

supctl merge volga webhooks my-webhook <<EOF
variables:
  - name: x
    value-from-message:
      json-pointer: /payload/a-list/1/a-value
      default: No value!
EOF

The JSON Pointer above would pick out the value hello from a message with the following json payload:

{
  "a-list": [
    "The first element",
    {
      "a-value": "hello"
    }
  ]
}

A pointer does not necessarily need to traverse the entire JSON tree and reach a scalar value. It is also acceptable to pick out pieces of the JSON object. For example, the pointer /payload/a-list/1 would result in a variable containing the following string:

{
  "a-value": "hello"
}

Managing webhooks

A running webhook can be restarted using the restart action. There is normally no need to restart the webhook manually, it will be automatically restarted when necessary.

supctl do volga webhooks slack-alerts restart

A webhook is automatically restarted if its configuration is changed in any way.

You can show the operational state of any configured webhook. Depending on its current state, a few different fields will show up:

• status - The overall status of the webhook.
• secret-status - If the webhook is unable to access a strongbox secret, this field will contain a plain text explanation of what it is looking for.
• consumed-topics - A list of topics currently subscribed to by the webhook.

supctl show volga webhooks slack-alerts
[...]
status: watiting-for-seret
secret-status: Variable url waiting for vault webhooks

supctl show volga webhooks my-webhook
[...]
status: ready
consumed-topics:
  - my-topic
  - another-topic

Further configuration

For more details on webhook configuration, refer to the REST API reference guide.