> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Backup and Restore Monitoring

CockroachDB includes metrics to monitor [backup](https://docs.cockroachlabs.com/docs/v24.2/backup), [restore](https://docs.cockroachlabs.com/docs/v24.2/restore), and [scheduled backup](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#on-previous-running-option) jobs. You can use monitoring integrations to alert when there are anomalies, such as backups that have failed or restore jobs encountering a retryable error.

You can access the [Prometheus Endpoint](#prometheus-endpoint) to track and alert on backup and restore metrics. We recommend setting up monitoring to [alert](https://docs.cockroachlabs.com/docs/v24.2/monitoring-and-alerting#alerting-tools) when anomalies occur. You can then use the following SQL statements to inspect details relating to schedules, jobs, and backups:

* [`SHOW SCHEDULES`](https://docs.cockroachlabs.com/docs/v24.2/show-schedules)
* [`SHOW JOBS`](https://docs.cockroachlabs.com/docs/v24.2/show-jobs)
* [`SHOW BACKUP`](https://docs.cockroachlabs.com/docs/v24.2/show-backup)

> [!NOTE]
>
> Metrics are reported per node. Therefore, it is necessary to retrieve metrics from every node in the cluster. For example, if you are monitoring whether a backup fails, it is necessary to track `scheduled_backup_failed` on each node.

## Prometheus endpoint

You can access the [Prometheus endpoint](https://docs.cockroachlabs.com/docs/v24.2/monitoring-and-alerting#prometheus-endpoint) (`http://<host:<http-port/_status/vars`) for backup and restore metrics.

Refer to the [Monitor CockroachDB with Prometheus](https://docs.cockroachlabs.com/docs/v24.2/monitor-cockroachdb-with-prometheus) tutorial for guidance on installing and setting up Prometheus and Alertmanager to track metrics.

### Available metrics

We recommend the following guidelines:

* Use the `schedules.BACKUP.last-completed-time` metric to monitor the specific backup job or jobs you would use to recover from a disaster.
* Configure alerting on the `schedules.BACKUP.last-completed-time` metric to watch for cases where the timestamp has not moved forward as expected.

| Metric | Description |
| - | - |
| `schedules.BACKUP.failed` | The number of scheduled backup jobs that have failed. **Note:** A stuck scheduled job will not increment this metric. |
| `schedules.BACKUP.last-completed-time` | The Unix timestamp of the most recently completed scheduled backup specified as maintaining this metric. **Note:** This metric only updates if the schedule was created with the [`updates_cluster_last_backup_time_metric` option](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#schedule-options). |
| `schedules.BACKUP.protected_age_sec` | The age of the oldest [protected timestamp record](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#protected-timestamps-and-scheduled-backups) protected by backup schedules. |
| `schedules.BACKUP.protected_record_count` | The number of [protected timestamp records](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#protected-timestamps-and-scheduled-backups) held by backup schedules. |
| `schedules.BACKUP.started` | The number of scheduled backup jobs that have started. |
| `schedules.BACKUP.succeeded` | The number of scheduled backup jobs that have succeeded. |
| `schedules.round.reschedule_skip` | The number of schedules that were skipped due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the [`on_previous_running=skip`](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#on-previous-running-option) schedule option. |
| `schedules.round.reschedule_wait` | The number of schedules that were rescheduled due to a currently running job. A value greater than 0 indicates that a previous backup was still running when a new scheduled backup was supposed to start. This corresponds to the [`on_previous_running=wait`](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#on-previous-running-option) schedule option. |
| `jobs.backup.currently_paused` | The number of backup jobs currently considered [paused](https://docs.cockroachlabs.com/docs/v24.2/pause-job). |
| `jobs.backup.currently_running` | The number of backup jobs currently running in `Resume` or `OnFailOrCancel` state. |
| `jobs.backup.fail_or_cancel_retry_error` | The number of backup jobs that failed with a retryable error on their failure or cancelation process. |
| `jobs.backup.fail_or_cancel_completed` | The number of backup jobs that successfully completed their failure or cancelation process. |
| `jobs.backup.fail_or_cancel_failed` | The number of backup jobs that failed with a non-retryable error on their failure or cancelation process. |
| `jobs.backup.protected_age_sec` | The age of the oldest [protected timestamp record](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#protected-timestamps-and-scheduled-backups) protected by backup jobs. |
| `jobs.backup.protected_record_count` | The number of [protected timestamp records](https://docs.cockroachlabs.com/docs/v24.2/create-schedule-for-backup#protected-timestamps-and-scheduled-backups) held by backup jobs. |
| `jobs.backup.resume_failed` | The number of backup jobs that failed with a non-retryable error. |
| `jobs.backup.resume_retry_error` | The number of backup jobs that failed with a retryable error. |
| `jobs.restore.currently_paused` | The number of restore jobs currently considered [paused](https://docs.cockroachlabs.com/docs/v24.2/pause-job). |
| `jobs.restore.currently_running` | The number of restore jobs currently running in `Resume` or `OnFailOrCancel` state. |
| `jobs.restore.fail_or_cancel_failed` | The number of restore jobs that failed with a non-retriable error on their failure or cancelation process. |
| `jobs.restore.fail_or_cancel_retry_error` | The number of restore jobs that failed with a retryable error on their failure or cancelation process. |
| `jobs.restore.protected_age_sec` | The age of the oldest [protected timestamp record](https://docs.cockroachlabs.com/docs/v24.2/architecture/storage-layer#protected-timestamps) protected by restore jobs. |
| `jobs.restore.protected_record_count` | The number of [protected timestamp records](https://docs.cockroachlabs.com/docs/v24.2/architecture/storage-layer#protected-timestamps) held by restore jobs. |
| `jobs.restore.resume_completed` | The number of restore jobs that successfully resumed to completion. |
| `jobs.restore.resume_failed` | The number of restore jobs that failed with a non-retryable error. |
| `jobs.restore.resume_retry_error` | The number of restore jobs that failed with a retryable error. |

## See also

* [Third-Party Monitoring Integrations](https://docs.cockroachlabs.com/docs/v24.2/third-party-monitoring-tools)
* [Manage a Backup Schedule](https://docs.cockroachlabs.com/docs/v24.2/manage-a-backup-schedule)
* [Backup and Restore Overview](https://docs.cockroachlabs.com/docs/v24.2/backup-and-restore-overview)
* [Backup Validation](https://docs.cockroachlabs.com/docs/v24.2/backup-validation)