Volga OpenTelemetry exporters

The OpenTelemetry exporters allow metrics and logs to be exported to an OpenTelemetry collector. The two types of metrics that can be exported are application metrics and (for site-provider tenants) host metrics. You can export built-in Volga topics as well as your own custom topics as logs.

Metric exporters

Setting up a metric exporter is mostly a matter of pointing to the collector. In this example, the collector is actually a Grafana Cloud instance:

supctl create volga otel-metric-exporters <<EOF
name: my-metric-exporter
site-placement:
  sites:
    - stockholm
    - london
sources:
  - host-metrics
  - application-metrics
protocol: http-json
url: "https://otlp-gateway-prod-eu-north-0.grafana.net:443/otlp/v1/metrics"
headers:
  - name: Authorization
    value: 'Basic \${token}'
variables:
  - name: token
    value-from-vault-secret:
      vault: tokens
      secret: grafana
      key: token
EOF

This creates a metric exporter that runs on the two sites stockholm and london, exporting both host and application metrics. The authentication with Grafana is handled using an HTTP header token, which is stored in Strongbox. For further details on how to set up the variable and the Strongbox secret, refer to Volga Webhook integration, where the same scheme is used.

Note that url explicitly specifies the port, 443, which would otherwise default to 4318 for the http-json protocol.

The protocol is set to http-json (rather than the default grpc) because this is what Grafana expects.

To check the current status of a created exporter, query it at a specific site:

supctl show -s stockholm volga otel-metric-exporters my-metric-exporter

name: my-metric-exporter
[...]
status: ready
start-time: 2026-09-11T12:27:10.842Z
consumed-topics:
  - system:host-metrics
  - system:application-metrics

Log exporters

Log exporters allow messages from any Volga topic to be exported as logs to an OpenTelemetry collector. The configuration shares many parameters with metric exporters but is in some ways more similar to the webhook exporter.

Example 1 - Exporting alerts

supctl create volga otel-log-exporters <<EOF
name: alerts-exporter
site-placement:
  all-sites: true
topics:
  - system:alerts
protocol: http-json
url: https://otlp-gateway-prod-eu-north-0.grafana.net:443/otlp/v1/logs
headers:
  - name: Authorization
    value: 'Basic \${token}'
severity-text: '\${severity}'
variables:
  - name: token
    value-from-vault-secret:
      vault: tokens
      secret: grafana
      key: token
  - name: severity
    value-from-message:
      json-pointer: /payload/severity
EOF

Compared to the metric exporter, the interesting differences are:

• topics - Log exporters can choose any topic to export.
• severity-text - An optional field describing the severity of each exported log record. In this case, it is set up using the variable severity, see below.
• variables - Just like webhooks, log exporters can define variables that extract values from Volga messages using JSON pointers. system:alerts is a JSON topic, making it easy to extract its severity field.

Not set in this example is the field body, which selects the actual log message to export. It defaults to ${MSG_PAYLOAD}, which is a variable pointing to the payload of the Volga message, i.e. the entire alert JSON.

Example 2 - Exporting application logs

Application logs live in Volga topics prefixed with system:container-logs. This example sets up a single exporter that exports logs from all applications.

supctl create volga otel-log-exporters <<EOF
name: my-log-exporter
site-placement:
  sites:
    - stockholm
    - london
topic-patterns:
  - ^system:container-logs:.*
start-position: beginning
protocol: http-json
url: https://otlp-gateway-prod-eu-north-0.grafana.net:443/otlp/v1/logs
headers:
  - name: Authorization
    value: 'Basic \${token}'
trim-payload-newline: true
severity-text: '\${severity}'
resource-attributes:
  - name: region
    string-value: emea
variables:
  - name: token
    value-from-vault-secret:
      vault: tokens
      secret: grafana
      key: token
  - name: severity
    value-from-message:
      json-pointer: /payload
      pattern: "^(?<p>[A-Z]*):"
EOF

Compared to the exporters above, there are a few differences here:

• topic-patterns - Used to select topics. This particular pattern matches all container log topics.
• start-position - By default, topics are consumed from the end, meaning that any old logs already on the topic when the exporter is created are ignored. By setting start-position to beginning, all logs are exported.
• trim-payload-newline - Container logs often include a trailing line terminator. By setting this to true, these are removed from the exported payload. Only applies to string topics.
• severity-text - The severity of the log message. See below.
• resource-attributes - Additional resource attributes to include. See further below.

Just as in the previous example, body is not set because the default value of ${MSG_PAYLOAD} works well. For a container log message, the payload of the Volga message is the entire log line.

Some notable default values not configured in either example

• resume - When true (the default), the exporter acknowledges messages on the log topic whenever a batch is exported and will resume from where it left off if it gets interrupted.
• tries - How many times to attempt exporting a metric or a log batch before dropping it and moving on if the collector is unresponsive. Available for both metric and log exporters. For metric exporters, the default value is 1. For logs, it is 0, which means infinite retries (with some caveats, see the reference documentation).

Extracting strings using regular expressions

This example assumes that the applications are using standard Python logging, where each line begins with an all-caps severity string followed by a colon. A typical debug line might be DEBUG:__main__:Hello world!. In order to extract the severity string from the message, the variable severity is configured. Its json-pointer is simply /payload, but it also supplies a regular expression that is used to select a substring from within the payload.

A simple regular expression that matches the severity for Python logs like this one would be just ^[A-Z]*. However, this is somewhat weak as it will match the initial capital letters of any string. The pattern in the example above requires the colon separator too. But the colon is not part of the severity text, so a capture group (?<p>[A-Z]*) is used to extract only the severity text. The name p is significant here. When a regular expression contains a capture group with the name p, only the value of the p group is returned. If there is no group named p, the entire matching string is returned.

It is perfectly possible to do something similar for the body field, for example, it might be tempting to strip the severity text from the body now that it is already sent separately. This could be done using another variable with the pattern ^[A-Z]*:(?<p>.*). However, keep in mind that if the pattern doesn't match at all you will fall back to the default value of the variable, or an empty string if there is no default.

Additional resource attributes

OpenTelemetry exports (metrics as well as logs) automatically include certain metadata fields (such as application and service names for container logs) in the form of OpenTelemetry resource attributes. You can add additional resource attributes in the resource-attributes list. These are just constant values that will be included in every call to the collector. For example, if you have multiple Avassa environments exporting to the same log collector, you may want to add an attribute uniquely identifying the exporting environment.

A configured resource attribute will override an automatically generated attribute with the same name.

Managing OpenTelemetry exporters

A running exporter can be restarted using the restart action. There is normally no need to restart the exporter manually, it will be automatically restarted when necessary.

supctl do -s london volga otel-metric-exporters my-metric-exporter restart

An exporter is automatically restarted if its configuration is changed in any way.

You can show the operational state of any configured exporter. Depending on its current state, a few different fields will show up:

• status - The overall status of the exporter.
• secret-status - If the exporter is unable to access a Strongbox secret, this field will contain a plain-text explanation of what it is looking for.
• consumed-topics - A list of topics currently subscribed to by the exporter.

supctl show -s stockholm volga otel-metric-exporters my-metric-exporter
[...]
status: waiting-for-secret
secret-status: Variable token waiting for vault tokens

supctl show -s london volga otel-metric-exporters my-metric-exporter
[...]
status: ready
consumed-topics:
  - system:host-metrics
  - system:application-metrics

Further configuration

For more details on OpenTelemetry exporter configuration, refer to the REST API reference guide for metric exporters and log exporters.