volga-websocket-api

---

description: ​The Avassa Volga WebSocket API enables real-time, bidirectional communication for producing and consuming messages using websockets.

Volga Web Socket API

Volga has a websocket API, detailed instructions on how to work with volga and websockets can be found in the Volga fundamentals documentation at https://docs.avassa.io/fundamentals/volga

The websocket API covers three different types of Volga operations

• Normal Volga topic producer and consumer operations.
• Volga infra operations.
• The query API, which runs on top of Volga infra.

A number of operations are described, each operation is a JSON object with one field called op. To invoke an operation, send the described JSON object on the authenticated websocket. See the fundamentals text on how to setup the websocket and for example code.

Operations

Normal Volga operations

• open-producer
• produce
• sync
• open-consumer
• ack
• more

The pattern for a producer is to open a websocket to a host in a site, send an open-producer JSON object with the required parameters, and then produce messages to the Volga topic by sending produce messages on the websocket. It doesn't matter which host in the site you connect to.

The pattern for a consumer is to open a websocket to a Volga site, invoke open-consumer with the required parameters, invoke more and then start to receive Volga messages on the websocket.

To close a consumer or producer, just close the websocket. If the topic is deleted, the server will close the websocket connection with status code 4100.

Infra-specific operations

• open-infra-producer
• infra-produce
• infra-sync
• open-infra-consumer

Volga infra is a means to distribute messages to all sites in a large system of sites. It is built on top of standard Volga topics that are stitched together by Volga itself.

To use a Volga infra, a named tenant infra must first be created by configuration.

$ supctl create volga infras << EOF
name: myinfra
format: json
EOF

The pattern for a down producer to Volga infra is to

1. Establish a websocket to the Control Tower.
2. Open an infra producer by sending a open-infra-producer JSON object on the websocket.
3. Send infra-produce messages on the websocket with direction down, this will then produce the message on all sites in the system
4. Code running at the edge sites, opens an infra consumer by establishing a websocket to the local site where the code runs.
5. The edge tenant code sends a open-infra-consumer JSON object on the websocket.
6. The edge tenant code sends a more JSON object on the websocket, at that point the edge code will receive Volga messages on the websocket.

Ping pong

The websocket server will respond to ping frames.

By default, the websocket server will also ping the client, but there is one exception: server-initiated pings are disabled for any client connecting to an Edge Enforcer on its local host (as they typically serve no purpose in this scenario).

Server-initiated pings can be explicitly enabled or disabled by including the query parameter ping=true or ping=false when connecting to the server. This will always override the default logic of the server.

When server-initiated pings are enabled, one ping is sent out every 45 seconds. A non-responsive client will be disconnected after three consecutive ping attempts.

open-producer

The JSON object to open a producer on a websocket. The caller will receive a response JSON object. Example:

{
    "op": "open-producer",
    "location": "local",
    "topic": "mytopic",
    "on-no-exists": "create",
    "create-options":
    {
        "replication-factor": 1,
        "persistence": "disk",
        "format": "string",
    },
    "name": "myprod"
}

Name	Type	Description
op	enumeration open-producer	This field is required.
topic	topic-name	This field is required. The name of the topic at the site. Topic names must be unique for each site for each tenant. A topic resides at a site.
name	string	This field is required. The name of the producer. The name is not required to be unique. The name will be part of the message the consumers receive. It will also show up in various supctl show volga outputs. Choose a readable name.
location	enumeration local parent child-site	This field is required. Where the topic is located. Volga can access topics not only at the site handling the websocket connection, but also at a parent or child site. local Access topic at the local site. parent Access topic at the parent site. child-site Access topic at a child site. The name of the child site must be provided by the child-site parameter.
child-site	string	Name of the site to connect to when location is child-site.
on-no-exists	enumeration fail wait create	This field is required. What to do if the topic does not exist. fail Return an error message wait Return a working producer/consumer handle and wait for the topic to be created. This is typically a good idea when connecting to a remote site where you do not know how many hosts there are, and therefore wish to defer the creation of the topic to code running there. create Create the topic with the parameters specified in create-options
create-options	Object see ws-create-options
send-control-message	boolean	Send a control message once a topic is properly connected and also when it is disconnected. The message will indicate weather the producer/consumer is properly connected. If for example majority is lost, the producer/consumer will still appear to be functioning to the client. If this option is set, the client get notified when the volga producer/consumer is reconnecting behind the scenes The default value is false.
on-overload	enumeration drop-oldest block-producer	What to do when a producer om a ram topic is producing faster than the consumers can consume and the RAM buffer fills up. Note that this applies also to producers that produce in async mode drop-oldest Drop the oldest message in the buffer block-producer Block the producer The default value is block-producer.
on-no-connection	enumeration block drop buffer buffer_and_drop	When a topic is disconnected - and this can be for several reasons. - No RAFT majority in the cluster. - No majority for the topic. - No connection from the edge to the Control Tower. This flag controls the behaviour of a disconnected producer block If the producer uses the sync flag, the producer is blocked until the topic is reconnected. If the producer uses the async flag, the producer will not be blocked until 1000 messages have been buffered drop All messages are dropped if the topic is disconnected. Produce requests will return an error buffer Messages are buffered in RAM indefinitely until the topic is connected again buffer_and_drop Up to 4000 messages can be buffered in RAM. Once 4000 is reached, Volga will start to drop messages. At that point, produce requests will return an error The default value is block.

produce

The JSON object to produce to a topic. A response JSON object is returned on the websocket. Example:

{
    "op": "produce",
    "sync": "sync",
    "payload": "Hello Avassa"
}

Name	Type	Description
op	enumeration produce	This field is required.
sync	enumeration sync async	Messages can be delivered synchronously or asynchronously to Volga. Asynchronous delivery has higher throughput. The default value is sync.
payload	Object	The payload of the message
topic-tag	string	If set, the message will only reach consumers listening for this specific topic-tag.

sync

Data produced to a topic in async mode has not necessarily been written to disk when the produce call returns. Calling sync ensures that all previously produced data is written to disk. Data produced in sync mode is always written to disk before the call returns, but calling sync ensures that a file system sync call is issued. The caller will receive a response JSON object. Example:

{
    "op": "sync"
}

Name	Type	Description
op	enumeration sync	This field is required.

open-consumer

The JSON object to open a consumer on a websocket. The caller will receive a response JSON object. No Volga messages will be received until a more operation has been sent on the websocket. Example:

{
    "op": "open-consumer",
    "location": "local",
    "topic": "mytopic",
    "position": "beginning",
    "on-no-exists": "wait",
    "name": "mycons"
}

Name	Type	Description
op	enumeration open-consumer	This field is required.
topic	topic-name	This field is required. The name of the topic at the site. Topic names must be unique for each site for each tenant. A topic resides at a site.
location	enumeration local parent child-site	This field is required. Where the topic is located. Volga can access topics not only at the site handling the websocket connection, but also at a parent or child site. local Access topic at the local site. parent Access topic at the parent site. child-site Access topic at a child site. The name of the child site must be provided by the child-site parameter.
child-site	string	Name of the site to connect to when location is child-site.
on-no-exists	enumeration fail wait create	This field is required. What to do if the topic does not exist. fail Return an error message wait Return a working producer/consumer handle and wait for the topic to be created. This is typically a good idea when connecting to a remote site where you do not know how many hosts there are, and therefore wish to defer the creation of the topic to code running there. create Create the topic with the parameters specified in create-options
create-options	Object see ws-create-options
send-control-message	boolean	Send a control message once a topic is properly connected and also when it is disconnected. The message will indicate weather the producer/consumer is properly connected. If for example majority is lost, the producer/consumer will still appear to be functioning to the client. If this option is set, the client get notified when the volga producer/consumer is reconnecting behind the scenes The default value is false.
name	string	This field is required. The name of the consumer. There can be arbitrarily many named consumers. If mode is exclusive, the name must be unique. If a second consumer is opened towards a topic with the same name, instead of returning an error, the first consumer is closed and the second overtakes the topic consumption
mode	enumeration exclusive shared standby	The mode for the consumer. exclusive In exclusive mode there can be only one consumer per consumer name at a site shared In shared mode, if there are multiple consumers with the same name, they will receive Volga messages in a round robin order standby In standby mode, there can be multiple consumers at a site with the same name, one of them will be designated as receiver of Volga messages, if that consumer disconnects, one of the remaining standby consumers will be chosen The default value is exclusive.
position OR position-since OR position-sequence-number OR position-timestamp	enumeration beginning end unread OR duration OR int64 OR union	Indicates where to start consuming in the topic stream beginning Start consuming from the beginning of the stream. If the topic has wrapped, i.e., started to drop chunks, the first message will be the oldest one remaining on the topic. end Start consuming from the end of the stream unread Start consuming from last acked message plus one A duration in years, days, hours, minutes and seconds. Format is [<digits>y][<digits>d][<digits>m][<digits>s]. Examples: 1y2d5h, 5h or 10m30s Start consuming from a relative timestamp, eg "1h"Start consuming from a specific sequence number. Negative numbers are relative to the end of the topic.Start consuming from an absolute timestamp expressed either as milliseconds since the epoch or an RFC 3339 timestamp (corresponding to the mtime and time fields received by a consumer respectively).
topic-tag	string	If topic-tag is set, only messages produced with the same topic-tag will be received by this consumer. This can be useful when multiple applications need to share the same topic of infra.
re-match	string	Consume only messages that match this Perl RE pattern, and just skip all other messages.
invert-match	empty	Invert the match provided by the re-match pattern
fields	select-fields-expression	A string that matches: expr = term *( ',' term ) term = path *( '/' '[' expr ']' ) path = field-name [ '/' path ] field-name = identifier / identifier '=' identifier For example: - fields=foo,bar - selects the fields foo and bar - fields=foo/bar - selects the fields bar in foo - fields=foo/[bar,baz] - selects bar and baz in foo - fields=foo=x - selects foo, and renames foo to x in the output Volga can trim incoming messages, delivering only the fields given here.
end-marker	boolean	If true, a special message is sent when there are no more messages to consume, i.e when a consumer would hang. This looks like a regular Volga message but without payload and sequence number, but with an extra JSON field "end-marker": true. The default value is false.
compact-output	boolean	This produces newline delimited JSON messages, just like requesting the content-type application/jsonlines on the REST API. The default value is false.

more

When a consumer has been opened on a websocket, the client must send the more JSON object in order to start receive Volga messages. Once the number of outstanding messages reaches zero, no more Volga messages messages will be received until a new more request is sent. No response is received for this request. Example:

{
    "op": "more",
    "n": 10
}

Name	Type	Description
op	enumeration more	This field is required.
n	uint32	This field is required. The number of messages the client will receive until a new more request must be sent

ack

When topics are opened with position unread, messages are consumed from sequence number where the named consumer last acked plus 1. There can be an arbitrary number of named consumers for a topic at a site, the acknowledgment is specific to the name of the consumer. Consumer that are opened in other modes than unread need not ack messages. Note that there is no need to acknowledge every single message. If you have received and handled 10 messages it is enough to send an ack for the last one. The call does not return a response. Example:

{
    "op": "ack",
    "seqno": 1
}

Name	Type	Description
op	enumeration ack	This field is required.
seqno	uint64	This field is required. Acknowledge the reception of a recently received message. Acks are specific to the named consumer. No response is received for this request.

delete-topic

Administrative call to remove a topic from a site. Returns response. Example:

{
    "op": "delete-topic",
    "location": "local",
    "topic": "mytopic"
}

Name	Type	Description
op	enumeration delete-topic	This field is required.
location	enumeration local parent child-site	Where the topic is located. Volga can access topics not only at the site handling the websocket connection, but also at a parent or child site. local Access topic at the local site. parent Access topic at the parent site. child-site Access topic at a child site. The name of the child site must be provided by the child-site parameter. The default value is local.
child-site	string	Name of the site to connect to when location is child-site.
topic	topic-name	This field is required. Name of the topic to delete

change-options

Change the options of a running topic. Returns response. Example:

{
    "op": "change-options",
    "location": "local",
    "topic": "mytopic",
    "max-size": "50 MB"
}

Name	Type	Description
op	enumeration change-options	This field is required.
location	enumeration local parent child-site	This field is required. Where the topic is located. Volga can access topics not only at the site handling the websocket connection, but also at a parent or child site. local Access topic at the local site. parent Access topic at the parent site. child-site Access topic at a child site. The name of the child site must be provided by the child-site parameter.
child-site	string	Name of the site to connect to when location is child-site.
topic	topic-name	This field is required. Name of the topic for which to change options
num-chunks OR max-size	uint32 OR size	Number of chunks to store on disk for the topic. This decides the total size of the topic. Each chunk is 1 MB. When chunk num-chunks + 1 is needed, chunk 1 is discarded, etc.Number of bytes with SI prefixes (kB, MB, GB, TB) (powers of 1000) or ISO/IEC prefixes (KiB, MiB, GiB, TiB) (powers of 1024). For example 12GB, 22.2 MiB Maximum size on disk for the topic. The topic is divided into chunks of approximately 1MB each. When max-size is reached, the oldest chunk is deleted.
max-days	uint32	Number of days to keep messages on this topic. If not set, messages will be kept until the topic is full.

open-infra-producer

Volga infra is a means to produce Volga messages to all sites for a tenant. Send this JSON object on a websocket to open a system wide producer. If the producer resides at an edge site, it can produce messages to the Control Tower, or to other edge sites. If the producer resides at the Control Tower, it can produce messages to all edge sites. The call returns a response. The tenant infra must be created by means of configuration. Once created, the Volga topics associated to the Infra will reside on all sites in the system. To close the producer, just close the websocket. Example:

{
    "op": "open-infra-producer",
    "infra": "myinfra"
}

Name	Type	Description
op	enumeration open-infra-producer	This field is required.
infra	topic-name	This field is required. The name of the Volga infra. Infra names must be unique across all sites for each tenant.
on-overload	enumeration drop-oldest block-producer	What to do when a producer om a ram topic is producing faster than the consumers can consume and the RAM buffer fills up. Note that this applies also to producers that produce in async mode drop-oldest Drop the oldest message in the buffer block-producer Block the producer The default value is block-producer.

infra-produce

Produce a message on the Infra. If synchronous, the call returns response. Example:

{
    "op": "infra-produce",
    "direction": "down",
    "payload": {"hello": "world"}
}

Name	Type	Description
op	enumeration infra-produce	This field is required.
direction	enumeration up down stitch	This field is required. up down stitch Topic stitching between different sites. It's possible to send data through infra from one site to another, although they're both behind NAT and have no IP connectivity
stitch-site	string	If we use stitching, data will go from one edge site to the Control Tower, turn down there and go 'down' to the indicated edge site
local-site-deliver	boolean	Deliver on infra on the producing site too The default value is false.
destination-sites	array of Object see destination-site	Deliver only to these sites. If unset, all sites where the current tenant resides will receive the message.
sync	enumeration sync async	Messages can be delivered synchronously or asynchronously to Volga. Asynchronous delivery has higher throughput. The default value is sync.
payload	Object	The payload of the message
topic-tag	string	If set, the message will only reach consumers listening for this specific topic-tag.

open-infra-consumer

This call opens an Infra consumer. An Infra consumer behaves like a a normal topic consumer, the client must issue more, ack, etc. The call returns response. To close the consumer, just close the websocket. Example:

{
    "op": "open-infra-consumer",
    "infra": "myinfra",
    "position": "beginning",
    "name": "myinfracons"
}

Name	Type	Description
op	enumeration open-infra-consumer	This field is required.
infra	topic-name	This field is required. The name of the Volga infra. Infra names must be unique across all sites for each tenant.
name	string	This field is required. The name of the consumer. There can be arbitrarily many named consumers. If mode is exclusive, the name must be unique. If a second consumer is opened towards a topic with the same name, instead of returning an error, the first consumer is closed and the second overtakes the topic consumption
mode	enumeration exclusive shared standby	The mode for the consumer. exclusive In exclusive mode there can be only one consumer per consumer name at a site shared In shared mode, if there are multiple consumers with the same name, they will receive Volga messages in a round robin order standby In standby mode, there can be multiple consumers at a site with the same name, one of them will be designated as receiver of Volga messages, if that consumer disconnects, one of the remaining standby consumers will be chosen The default value is exclusive.
position OR position-since OR position-sequence-number OR position-timestamp	enumeration beginning end unread OR duration OR int64 OR union	Indicates where to start consuming in the topic stream beginning Start consuming from the beginning of the stream. If the topic has wrapped, i.e., started to drop chunks, the first message will be the oldest one remaining on the topic. end Start consuming from the end of the stream unread Start consuming from last acked message plus one A duration in years, days, hours, minutes and seconds. Format is [<digits>y][<digits>d][<digits>m][<digits>s]. Examples: 1y2d5h, 5h or 10m30s Start consuming from a relative timestamp, eg "1h"Start consuming from a specific sequence number. Negative numbers are relative to the end of the topic.Start consuming from an absolute timestamp expressed either as milliseconds since the epoch or an RFC 3339 timestamp (corresponding to the mtime and time fields received by a consumer respectively).
topic-tag	string	If topic-tag is set, only messages produced with the same topic-tag will be received by this consumer. This can be useful when multiple applications need to share the same topic of infra.
re-match	string	Consume only messages that match this Perl RE pattern, and just skip all other messages.
invert-match	empty	Invert the match provided by the re-match pattern
fields	select-fields-expression	A string that matches: expr = term *( ',' term ) term = path *( '/' '[' expr ']' ) path = field-name [ '/' path ] field-name = identifier / identifier '=' identifier For example: - fields=foo,bar - selects the fields foo and bar - fields=foo/bar - selects the fields bar in foo - fields=foo/[bar,baz] - selects bar and baz in foo - fields=foo=x - selects foo, and renames foo to x in the output Volga can trim incoming messages, delivering only the fields given here.
end-marker	boolean	If true, a special message is sent when there are no more messages to consume, i.e when a consumer would hang. This looks like a regular Volga message but without payload and sequence number, but with an extra JSON field "end-marker": true. The default value is false.
compact-output	boolean	This produces newline delimited JSON messages, just like requesting the content-type application/jsonlines on the REST API. The default value is false.

infra-sync

Equivalent to the regular sync operation. Ensures that previously sent messages are written to disk and that a file system sync call is issued. The call returns response. Example:

{
    "op": "infra-sync"
}

Name	Type	Description
op	enumeration infra-sync	This field is required.

volga-message

Volga message object received by a consumer.

Name	Type	Description
time	date-and-time	Message timestamp in RFC 3339 format.
mtime	uint64	Message timestamp in milliseconds since the epoch.
seqno	uint64	Sequence number of the message. This is the number that can be used to ack the message.
remain	uint64	Number of messages remaining until another more call is required.
producer-name	string	The name of the producer of the message.
host	string	The name of the host that produced the message.
origin	string	The site where the message originated (only present for infra consumers).
payload OR end-marker	Object OR boolean	The dataSet if the consumer has requested end-marker and the end of the stream has been reached

query-topics-message

JSON object which is the result of a query-topics request.

Name	Type	Description
time	date-and-time	This field is always present. Message timestamp in RFC 3339 format.
mtime	uint64	This field is always present. Message timestamp in milliseconds since the epoch.
host	string	This field is always present. The name of the host that produced the message
site	string	This field is always present. The name of the site that produced the message
application	string	Present if the message was related to a specific application.
service-name	string	Present if the message was related to a specific service.
service-instance	string	Present if the message was related to a specific service instance.
container-name	string	Present if the message was related to a specific container.
payload	Object

query-topics-dry-run

Result of a query-topics dry run.

Name	Type	Description
specified-sites	array of string	Sites targeted by the query.
non-responding-sites	array of string	Sites that did not respond.
site-reports	array of Object see site-report	For each responding site, the list of topics that are going to be searched.

volga-control-message

If control messages are requested for a consumer or a producer, messages on this format will be received on the websocket.

Name	Type	Description
connected	boolean	Boolean indicating the new connection status
kind	enumeration control-message

response

Common response object for a number of operations.

Name	Type	Description
result	enumeration ok error	This field is always present.
errors	array of Object see response-errors

query-topics

This is the websocket API to query-topics. The data passed in is a full query-topics JSON object, with the additional leaf op set to query-topics. The response varies depending on the input:

• If dry-run is true, the response is a query-topics-dry-run object.
• If count-matches is true, the response is a simple JSON object containing the number of matches, eg {"count": 36}.
• If an error occurs, the response is a response object.
• Otherwise, the result will be a stream of matching volga messages. If auto-more is false, the socket will function as a regular volga consumer, initially returning a response object and requiring the client to issue more requests in order to receive messages. If auto-more is true, no response object is sent and no more messages are required. The server will continuously deliver matching messages.

If payload-only is false, the messages will come in the form of query-topics-message objects, but if payload-only is true, they will be raw volga messages, which can be JSON objects or plain strings. Note that it is possible for a single query to contain multiple sets of topics with different values for payload-only. If an error occurs, the response will be a response object.

Refer to the REST API documentation and the Volga fundamentals document for a more detailed description of the query-topics functionality.

Example: Search a topic name called mytext on all sites and just print the data from that topic.

{
    "op": "query-topics",
    "follow": true,
    "all-sites": true,
    "topics":
    [
        {
            "topic-names": ["mytext"],
            "output": {"payload-only": true}
        }
    ]
}

Name	Type	Description
op	enumeration query-topics	This field is required.
auto-more	boolean	Send results continuously, with no need for the client to issue more requests. Currently on by default. The default value is true.
sort	boolean	Sort all messages by timestamp. When true, the control tower will wait for all sites to respond before returning anything to the caller, ensuring that messages are delivered in timestamp order. Note that when follow is also set, it is not possible to wait for all sites to respond as that would entail waiting indefinitely for possible future messages. Instead, messages are buffered and sorted for up to three seconds, first at the site level and then at the Control Tower, to account for network lag and system clock discrepancies, but under poor conditions it is still possible that some messages will be delivered out of order. The default value is false.
follow	boolean	If follow is true, the output from the search will continue indefinitely. If follow is false, all sites that participate in the query will utilize the end-marker feature in Volga for the chosen topics and report to the Control Tower when there is no more data. If one ore more sites chosen by the query fails to respond, the unfollow-timeout will trigger and the query is aborted The default value is false.
site-timeout	duration	A duration in years, days, hours, minutes and seconds. Format is [<digits>y][<digits>d][<digits>m][<digits>s]. Examples: 1y2d5h, 5h or 10m30s Timeout controlling how long to wait for unresponsive sites to deliver log data The default value is 10s.
compact-output	boolean	Produce newline-delimited JSON messages. This can make life easier for a parser which consumes the output data. The default value is false.
dry-run	boolean	This call will report which sites, and topics are chosen by the query. Furthermore, when issuing a query towards a set of sites, if one or several sites are down or unresponsive, this flag will report all sites that never responded, using the site-timeout value The default value is false.
since OR start-time OR position-sequence-number	duration OR date-and-time OR int64	A duration in years, days, hours, minutes and seconds. Format is [<digits>y][<digits>d][<digits>m][<digits>s]. Examples: 1y2d5h, 5h or 10m30s By default, queries will start at the beginning of all selected topics. This parameter will make the query start at the specified time relative to current time. For example $ supctl do volga query-topics --since 1h ..... will search all topics starting with messages from one hour ago.Similar to the since parameter, an explicit start time can be specified, eg 2022-09-02T18:23:17.153313Z.Begin search at this sequence number in each topic. When searching in multiple topics at once, this is mostly useful with negative negative numbers, for example -100 to search through the last 100 messages of each topic.
duration OR stop-time	duration OR date-and-time	A duration in years, days, hours, minutes and seconds. Format is [<digits>y][<digits>d][<digits>m][<digits>s]. Examples: 1y2d5h, 5h or 10m30s This parameter limits the timespan within which messages are searched for. $ supctl do volga query-topics --since 1h --duration 10m .... Will search all topics for messages that are between 50 and 60 minutes old.Similar to the start-time parameter, an explicit stop time can be specified, eg 2022-09-02T18:23:17.153313Z
drop-until-n-remain	int32	At the Control Tower, collect all data according to the provided filters, and deliver the last n items gathered from all the specified sites. This implies follow false
count-matches	boolean	Count number of occurrences that match the provided filters. Counting occurs at the edge, and starts after start-time settings, and drop settings have been applied. No data is returned, only the count of matches as a single JSON object, for example: {"count": 36} The default value is false.
site-names OR re-match-sites OR sites-from-application-deployment OR match-site-labels OR all-sites	array of string OR regexp OR string OR label-match-expression OR empty	The default value is all-sites.A list of site namesPCRE regular expression. Query all sites that match the given regular expression. To check which sites match the regular expression, use the dry-run flagRun the query only on the sites that have the specified application deploymentA string that matches: expr = ( expr 'and' expr ) / ( expr 'or' expr ) / ( 'not' expr ) / ( label '=' value ) / '{}' / label The {} expression matches the empty label set. For example: (foo and not(bar = xx)) or (baz = red) or {} Use a site label expression to specify which sites to run the query on, for example: - system/type = edge - (city = stockholm) or (city = gothenburg) Query all sites that the tenant has access to. This is the default.
topics	array of Object see query-topics

Common Objects

The ws-create-options Object

Name	Type	Description
replication-factor	uint32	Integer deciding how many replicas to create for a topic. Always use an odd number. A majority of replicas is required for a topic to be operational. The default value is 1.
persistence	enumeration disk ram	disk means the topic is persistent on the host where it was originally placed at creation time. If ram is used, replication-factor is implicitly 1. The default value is disk.
local-placement	boolean	Include the host handling the request in the replication set. For topics with replication-factor 1, this means that the host handling the request is guaranteed to become the topic handler and the topic can never be moved to another host. For topics with higher replication factors, the host handling the request will be part of the initial replication set as long as its maintenance-mode is not set to out-of-service, but the topic can be moved to other hosts later as needed. The default value is false.
num-chunks OR max-size	uint32 OR size	Number of chunks to store on disk for the topic. This decides the total size of the topic. Each chunk is 1 MB. When chunk num-chunks + 1 is needed, chunk 1 is discarded, etc.Number of bytes with SI prefixes (kB, MB, GB, TB) (powers of 1000) or ISO/IEC prefixes (KiB, MiB, GiB, TiB) (powers of 1024). For example 12GB, 22.2 MiB Maximum size on disk for the topic. The topic is divided into chunks of approximately 1MB each. When max-size is reached, the oldest chunk is deleted.
format	enumeration string json	The format of messages on this topic. string The message payload is a string. json The message payload is an arbitrary json object.
match-host-labels	label-match-expression	A string that matches: expr = ( expr 'and' expr ) / ( expr 'or' expr ) / ( 'not' expr ) / ( label '=' value ) / '{}' / label The {} expression matches the empty label set. For example: (foo and not(bar = xx)) or (baz = red) or {} Run the topic on hosts whose labels match the given expression. If not provided, volga will choose from all available hosts on the site. The available host labels in a site can be read from /assigned-sites/SITE as host-labels.
max-days	uint32	Number of days to keep messages on this topic. If not set, messages will be kept until the topic is full.
encryption	enumeration signature full	Messages can be encrypted or cryptograhically signed when written to disk.
transit-key	name	Transit key to use for signing or encryption. When creating topics in the Control Tower itself, the key will automatically be created if it does not already exist. For topics on edge sites, the key must be created beforehand.
ephemeral	boolean	If set to true, the topic will be deleted immediately when there are no connected producers or consumers. The default value is false.

The destination-site Object

Name	Type	Description
site	string

The query-topics Object

A topic query consists of site selection, and then a list of topic selectors, combined with with filters and output modifiers. Thus, multiple topics can be searched with the same query, applying different filters and output modifiers to each set of of topics in the topic list

Name	Type	Description
topic-names OR re-match-topic-name OR match-topic-labels	array of topic-name OR regexp OR label-match-expression	A list of topic namesPCRE regular expression. A regular expression that matches one or more topicsA string that matches: expr = ( expr 'and' expr ) / ( expr 'or' expr ) / ( 'not' expr ) / ( label '=' value ) / '{}' / label The {} expression matches the empty label set. For example: (foo and not(bar = xx)) or (baz = red) or {} Choose topics based on the topic labels
filter	Object see query-topics-filter
output	Object see query-topics-output

The query-topics-filter Object

Name	Type	Description
re-opts	array of enumeration caseless anchored ungreedy	Regular expression flags to apply to expressions in this filter. The re-opts applies to all the various filter expressions, but not to the site selection
merged-drop-until-re-match OR merged-drop-until-string-match	regexp OR string	PCRE regular expression. On the merged output from all topics, and all sites, drop all data until the provided regular expression matches. Stream from that point forward. This is only useful when you have applications at different sites that interact. Otherwise it is better to use the drop-until-re-match flagOn the merged output from all topics and all sites, drop all data until the provided string matches. Stream from that point forward. This is only useful when you have applications at different sites that interact. Otherwise it is better to use the drop-until-string-match flag
merged-re-match	regexp	PCRE regular expression. Run regular expression match on the merged output from all sites. Similar to merge-drop-until-re-match, this is only meaningful when there is interaction between applications at multiple sites.
drop-until-re-match OR drop-until-string-match OR drop-until-last-re-match OR drop-until-last-string-match	regexp OR regexp OR regexp OR string	PCRE regular expression. On the selected topics, search for the first entry matching the provided regular expression and stream from that point forwardPCRE regular expression. On the selected topics, search for the provided string and stream from that point forward.PCRE regular expression. On the selected topics, search for the last (newest) entry that matches the provided regular expression, and stream from that point on.On the selected topics, search for the last (newest) entry that matches the provided string, and stream from that point on.
re OR string OR re-match OR match-json	Object see query-topics-filter-re OR Object see query-topics-filter-string OR regexp OR Object	PCRE regular expression. On the selected topics, return only entries that match the provided regular expressionA JSON object that that is matched against the payload. This is best explained by an example. Assume that an item in the payload data in a chosen topic looks like: { "struct": { "xxx": 44, "foo": "bar" }, "num": 10, "mylist2": [ { "item": 55 } ], "mylist": [ 1, 2, 3 ] } The following JSON objects will match the entry above, thus ensuring that the log item is chosen. First example is an AND match, both entries must match "filter": { "match-json" : { "struct" : { "xxx" : 44 }, "num" : 10 } }, The JSON value null is used as a wildcard, thus resulting in a structure match only "filter": { "match-json" : { "struct" : null, "num" : 10 } }, Next example is an OR based list match, if we provide multiple entries in a list, at least one of the entries in the provided list must match "filter": { "match-json" : { "mylist" : [3, "qqqq"], } }, And finally a combination of AND and OR based match with a list of structs that will match "filter": { "match-json" : { "mylist" : [3, "qqqq"], "mylist2" : [13, 5, {"item" : 55}, {"foo" : 4433}], "num" : 7 } },

The query-topics-filter-re Object

Name	Type	Description
match	array of regexp	PCRE regular expression. On all sites where the query runs, search for entries in the chosen topics that match all the provided regular expressions.
no-match	array of regexp	PCRE regular expression. On all sites where the query runs, search for entries in the chosen topics that do not match any of the provided regular expressions.

The query-topics-filter-string Object

Name	Type	Description
match	array of string	On all sites where the query runs, search for entries in the chosen topics that match all the provided strings.
no-match	array of string	On all sites where the query runs, search for entries in the chosen topics that do not match any of the provided strings.

The query-topics-output Object

Name	Type	Description
payload-only	boolean	Deliver payload only. By default, Volga will embed the log entry in a JSON object with additional data about the entry, such as the time when the message was produced etc The default value is false.
format	string	With payload-only data for text logs, Volga can be instructed to prepend each line with a header, thus making it possible to distinguish between log entries from different sites/hosts/topics. The following modifiers are available: - %h The host that produced the message - %s The site that produced the message - %p The payload of the message - %t The time when the log entry was originally produced - %T The name of the topic the message originates from The default value is %h : %p.
fields	select-fields-expression	A string that matches: expr = term *( ',' term ) term = path *( '/' '[' expr ']' ) path = field-name [ '/' path ] field-name = identifier / identifier '=' identifier For example: - fields=foo,bar - selects the fields foo and bar - fields=foo/bar - selects the fields bar in foo - fields=foo/[bar,baz] - selects bar and baz in foo - fields=foo=x - selects foo, and renames foo to x in the output If output is JSON, Volga can trim the messages, displaying only certain fields. If payload-only is true, the trimming will occur at the edge site, otherwise it will occur on the JSON message constructed by Volga

The response-errors Object

Name	Type	Description
error-message	string
error-info	Object see volga-api-error

The volga-api-error Object

Name	Type	Description
reason	enumeration no-such-site no-such-topic mode-error log-corrupt not-enough-nodes bad-name no-exists policy-violation bad-pattern no-such-infra infra-topic system-topic bad-topic-name bad-format bad-options bad-direction bad-json bad-request internal-error too-big-message drop archived-topic bad-transit-key	This is a list of errors that can occur over websocket and the REST interface to volga no-such-site When we open topics using location child-site, and we ask for a nonexistent site, this error is returned no-such-topic A nonexistent topic was requested mode-error When we open topics in shared or standby mode and not all consumers with the same name have the same mode log-corrupt This is a catastrophic error, it means that the files for the topic are corrupt, and cannot be opened. This error will require manual intervention not-enough-nodes The requested replication-factor for a topic cannot be fulfilled at a site bad-name Topic names can only contain the characters 0-9, A-Z, a-z, and +_=:@.- no-exists Over the websocket API, topics are automatically created the first time they are opened, this is not the case for the Volga REST API. If a produce or consume operation is attempted towards a nonexistent topic over the REST API, this error is returned policy-violation Strongbox policy made the Volga operation fail bad-pattern When a bad regular expression is provided to the API no-such-infra A nonexistent infra was requested. infra-topic The requested operation is not applicable for an infra topic. system-topic The requested operation is not applicable for a system topic. bad-topic-name The topic name includes invalid characters or is otherwise malformed. bad-format The message does not conform to the format of the topic. bad-options The create-options do not match the already existing ones. bad-direction An unavailable direction was requested for a volga infra message. bad-json Message could not be decoded as JSON. bad-request Bad or unavailable operation requested. internal-error An internal error occurred, check logs for more info too-big-message A message cannot be bigger than the chunk-size for a topic drop A message is dropped from a disconnected topic archived-topic The requested operation is not applicable for an archived topic bad-transit-key The specified transit key cannot be used for the requested encryption type.

The site-report Object

Name	Type	Description
site	string
topics	array of string