Queues Dashboard

The Queues dashboard lets you monitor the health and performance of various queueing systems in CockroachDB. To view this dashboard, access the DB Console and click Metrics on the left-hand navigation, and then select Dashboard > Queues. Use the Graph menu to display metrics for your entire cluster or for a specific node. To the right of the Graph and Dashboard menus, a time interval selector allows you to filter the view for a predefined or custom time interval. Use the navigation buttons to move to the previous, next, or current time interval. When you select a time interval, the same interval is selected in the SQL Activity pages. However, if you select 10 or 30 minutes, the interval defaults to 1 hour in SQL Activity pages. When viewing graphs, a tooltip will appear at your mouse cursor providing further insight into the data under the mouse cursor. Click anywhere within the graph to pin the tooltip in place, decoupling the tooltip from your mouse movements. Click anywhere within the graph to cause the tooltip to follow your mouse once more.

All timestamps in the DB Console are shown in Coordinated Universal Time (UTC).

The Queues dashboard displays the following time series graphs:

Queue Processing Failures

The Queue Processing Failures graph displays processing failures experienced across various queuing systems. Hovering over the graph displays values for the following metrics:

Metric	Description
GC	The number of replicas which failed processing in the garbage collection queue, as tracked by the `queue.gc.process.failure` metric.
Replica GC	The number of replicas which failed processing in the replica garbage collection queue, as tracked by the `queue.replicagc.process.failure` metric.
Replication	The number of replicas which failed processing in the replica queue, as tracked by the `queue.replicate.process.failure` metric.
Split	The number of replicas which failed processing in the split queue, as tracked by the `queue.split.process.failure` metric.
Consistency	The number of replicas which failed processing in the consistency checker queue, as tracked by the `queue.consistency.process.failure` metric.
Raft Log	The number of replicas which failed processing in the Raft log queue, as tracked by the `queue.raftlog.process.failure` metric.
Raft Snapshot	The number of replicas which failed processing in the Raft repair queue, as tracked by the `queue.raftsnapshot.process.failure` metric.
Time Series Maintenance	The number of replicas which failed processing in the time series maintenance queue, as tracked by the `queue.tsmaintenance.process.failure` metric.

Queue Processing Times

The Queue Processing Times graph displays the processing rate across various queue systems. Hovering over the graph displays values for the following metrics:

Metric	Description
GC	The processing rate for the garbage collection queue, as tracked by the `queue.gc.processingnanos` metric.
Replica GC	The processing rate for the replica garbage collection queue, as tracked by the `queue.replicagc.processingnanos` metric.
Replication	The processing rate for the replication queue, as tracked by the `queue.replicate.processingnanos` metric.
Split	The processing rate for the split queue, as tracked by the `queue.split.processingnanos` metric.
Consistency	The processing rate for the consistency checker queue, as tracked by the `queue.consistency.processingnanos` metric.
Raft Log	The processing rate for the Raft log queue, as tracked by the `queue.raftlog.processingnanos` metric.
Raft Snapshot	The processing rate for the Raft repair queue, as tracked by the `queue.raftsnapshot.processingnanos` metric.
Time Series Maintenance	The processing rate for the time series maintenance queue, as tracked by the `queue.tsmaintenance.processingnanos` metric.

Replica GC Queue

The Replica GC Queue graph displays various details about the health and performance of the replica garbage collection queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the replica garbage collection queue, as tracked by the `queue.replicagc.process.success` metric.
Pending Actions	The number of pending replicas in the replica garbage collection queue, as tracked by the `queue.replicagc.pending` metric.
Replicas Removed	The number of replica removals attempted in the replica garbage collection queue, as tracked by the `queue.replicagc.removereplica` metric.

Replication Queue

The Replication Queue graph displays various details about the health and performance of the replication queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the replication queue, as tracked by the `queue.replicate.process.success` metric.
Pending Actions	The number of pending replicas in the replication queue, as tracked by the `queue.replicate.pending` metric.
Replicas Added	The number of replica additions attempted by the replication queue, as tracked by the `queue.replicate.addreplica` metric.
Replicas Removed	The number of replica removals attempted by the replication queue, as tracked by the `queue.replicate.removereplica` metric.
Dead Replicas Removed	The number of dead replica removals attempted by the replication queue, as tracked by the `queue.replicate.removedeadreplica` metric.
Learner Replicas Removed	The number of learner replica removals attempted by the replication queue, as tracked by the `queue.replicate.removelearnerreplica` metric.
Replicas Rebalanced	The number of replica rebalancer-initiated additions attempted by the replication queue, as tracked by the `queue.replicate.rebalancereplica` metric.
Leases Transferred	The number of range lease transfers attempted by the replication queue, as tracked by the `queue.replicate.transferlease` metric.
Replicas in Purgatory	The number of replicas in the replication queue’s purgatory currently awaiting allocation options, as tracked by the `queue.replicate.purgatory` metric.

Split Queue

The Split Queue graph displays various details about the health and performance of the split queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the split queue, as tracked by the `queue.split.process.success` metric.
Pending Actions	The number of pending replicas in the split queue, as tracked by the `queue.split.pending` metric.

Merge Queue

The Merge Queue graph displays various details about the health and performance of the merge queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the merge queue, as tracked by the `queue.merge.process.success` metric.
Pending Actions	The number of pending replicas in the merge queue, as tracked by the `queue.merge.pending` metric.

Raft Log Queue

The Raft Log Queue graph displays various details about the health and performance of the Raft log queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the Raft log queue, as tracked by the `queue.raftlog.process.success` metric.
Pending Actions	The number of pending replicas in the Raft log queue, as tracked by the `queue.raftlog.pending` metric.

Raft Snapshot Queue

The Raft Snapshot Queue graph displays various details about the health and performance of the Raft repair queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the Raft repair queue, as tracked by the `queue.raftsnapshot.process.success` metric.
Pending Actions	The number of pending replicas in the Raft repair queue, as tracked by the `queue.raftsnapshot.pending` metric.

Consistency Checker Queue

The Consistency Checker Queue graph displays various details about the health and performance of the consistency checker queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the consistency checker queue, as tracked by the `queue.consistency.process.success` metric.
Pending Actions	The number of pending replicas in the consistency checker queue, as tracked by the `queue.consistency.pending` metric.

Time Series Maintenance Queue

The Time Series Maintenance Queue graph displays various details about the health and performance of the time series maintenance queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the time series maintenance queue, as tracked by the `queue.tsmaintenance.process.success` metric.
Pending Actions	The number of pending replicas in the time series maintenance queue, as tracked by the `queue.tsmaintenance.pending` metric.

MVCC GC Queue

The MVCC GC Queue graph displays various details about the health and performance of the garbage collection queue. Hovering over the graph displays values for the following metrics:

Metric	Description
Successful Actions	The number of replicas successfully processed by the garbage collection queue, as tracked by the `queue.gc.process.success` metric.
Pending Actions	The number of pending replicas in the garbage collection queue, as tracked by the `queue.gc.pending` metric.

Protected Timestamp Records

The Protected Timestamp Records graph displays the number of protected timestamp records (used by backups, changefeeds, etc. to prevent MVCC GC) per node, as tracked by the spanconfig.kvsubscriber.protected_record_count metric.

Summary and events

Summary panel

A Summary panel of key metrics is displayed to the right of the timeseries graphs.

Metric	Description
Total Nodes	The total number of nodes in the cluster. Decommissioned nodes are not included in this count.
Capacity Used	The storage capacity used as a percentage of usable capacity allocated across all nodes.
Unavailable Ranges	The number of unavailable ranges in the cluster. A non-zero number indicates an unstable cluster.
Queries per second	The total number of `SELECT`, `UPDATE`, `INSERT`, and `DELETE` queries executed per second across the cluster.
P99 Latency	The 99th percentile of service latency.

If you are testing your deployment locally with multiple CockroachDB nodes running on a single machine (this is not recommended in production), you must explicitly set the store size per node in order to display the correct capacity. Otherwise, the machine’s actual disk capacity will be counted as a separate store for each node, thus inflating the computed capacity.

Events panel

Underneath the Summary panel, the Events panel lists the 5 most recent events logged for all nodes across the cluster. To list all events, click View all events.

The following types of events are listed:

Database created
Database dropped
Table created
Table dropped
Table altered
Index created
Index dropped
View created
View dropped
Schema change reversed
Schema change finished
Node joined
Node decommissioned
Node restarted
Cluster setting changed

Get Started

CockroachDB and AI

Feature Overview

Connect to an Application

Self-Hosted Deployments

Schema Design

Reads and Writes

Stream Data

Multi-Region Capabilities

Optimize Performance

Integrate

Dashboard navigation

Queue Processing Failures

Queue Processing Times

Replica GC Queue

Replication Queue

Split Queue

Merge Queue

Raft Log Queue

Raft Snapshot Queue

Consistency Checker Queue

Time Series Maintenance Queue

MVCC GC Queue

Protected Timestamp Records

Summary and events

Summary panel

Events panel

See also

​Dashboard navigation

​Queue Processing Failures

​Queue Processing Times

​Replica GC Queue

​Replication Queue

​Split Queue

​Merge Queue

​Raft Log Queue

​Raft Snapshot Queue

​Consistency Checker Queue

​Time Series Maintenance Queue

​MVCC GC Queue

​Protected Timestamp Records

​Summary and events

​Summary panel

​Events panel

​See also

Dashboard navigation

Queue Processing Failures

Queue Processing Times

Replica GC Queue

Replication Queue

Split Queue

Merge Queue

Raft Log Queue

Raft Snapshot Queue

Consistency Checker Queue

Time Series Maintenance Queue

MVCC GC Queue

Protected Timestamp Records

Summary and events

Summary panel

Events panel

See also